Draft Requirements Document FSI Training Evaluation System.pdf

Unclassified

United States Department of State Foreign Service Institute George P. Shultz National Foreign Affairs

Training Center 4000 Arlington Boulevard Arlington, Virginia 22204

Centralized Evaluation System Requirements Document

Last updated February 2020 Ver. 2.00

Prepared by: FSI’s Office of the Director FSI’s Office of the Executive Director for Management FSI’s Schools and Centers.

FSI Centralized Evaluation System Scope of Objectives Page ii

Table of Contents

1. Purpose

2. Background

3. Scope

3.1. General

3.2. Key Personnel

3.3. Period and Location of Performance

4. Current Environment

4.1. Existing/Legacy Environment Components

4.2. Other Development Tools to be encountered

5. New Centralized Evaluation System

5.1. Centralized Data Repository

5.2. Account Management

5.3. Instrument Design and Development

5.4. Data Collection Capability

5.5. Data Analytics and Reporting

5.6. Instrument Management and Administration

5.7. Integration with Other Systems

5.8. Technical and Security Requirements

5.9. Contractor Support

6. Constraints:

6.1. Access Control

6.2. Authentication:

6.3. Security Requirements: Personnel Security Clearances

6.4. Non-Disclosure Agreement

6.5. Data Ownership

Appendix A. Centralized Evaluation System Detailed Requirements Appendix B Centralized Evaluation System Data Elements Appendix C Centralized Evaluation System User Roles Appendix D Acronyms and Glossary Appendix E Business Logic Diagrams Appendix F FSI Policies and Standards Appendix G Centralized Evaluation System Functionality Priority List

TECHNICAL EXHIBIT 1

TECHNICAL EXHIBIT 2

FSI Centralized Evaluation System: Requirements Doc.

1. Purpose The Foreign Service Institute (FSI) is seeking an enterprise-wide Centralized Evaluation System (CES) and a contractor to plan, deploy, tailor and integrate the CES based on the specifications and requirements of the Statement of Objectives and in compliance with Department of State and FSI Policies and Standards. The contractor will assist the Institute in identifying, configuring, customizing and deploying a new CES. The CES must be comprised of a FedRamp authorized solution rated ‘FedRamp Authorized’ with a FISMA moderate approval or the option for an on-premises deployment. Details about requirements are provided in Appendix A FSI requires the capacity to evaluate its training programs. FSI uses training evaluation data to understand the extent to which FSI’s training meets the learning and development needs of all its user populations. FSI requires that training evaluation data be gathered and stored in a central repository so that FSI administrators can run queries using data from a variety of FSI courses and user feedback from post-training evaluation instruments. This effort requires a new Centralized Evaluation System to manage training evaluation processes. This effort is neither a Learning Management System (LMS) nor an electronic survey tool.

The CES will support management of FSI’s training evaluation and needs analysis processes. The objective is to evaluate comprehensively by assessing the effectiveness, relevance, impact, and reach of training to the Department achieved through its training and professional development programs and to support curriculum development and instruction. FSI will also evaluate systematically by collecting, analyzing, and reporting evaluation data to support evidence-based decision making. The CES will make this possible by serving as a repository for evaluation data, managing accounts, supporting the design and development of data collection instruments, collecting data, storing data collected with a variety of ways (e.g., survey or telephone interview), analyzing data, reporting results, and managing and administering instruments.

2. Background The Foreign Service Institute (FSI) is the federal government's primary training institution for officers and support personnel of the U.S. foreign affairs community. FSI’s mission is to develop the men and women our nation requires to fulfill our leadership role in world affairs and to advance and defend U.S. interests.

At the George P. Shultz National Foreign Affairs Training Center in Arlington, Virginia, the FSI provides more than 800 courses—including some 70 foreign languages—to more than 170,000 enrollees a year (79,500 distinct students) from the U.S. Department of State (DOS) and more than 40 other government agencies and the military service branches.

The Institute's programs include training for the professional development of Foreign Service administrative, consular, economic/commercial, political, and public diplomacy officers; for specialists in the fields of information management, office management, security, and medical practitioners and nurses; for Foreign Service Nationals who work at U.S. posts around the world;

and for Civil Service employees of the State Department and other agencies. Ranging in length from one day to two years, courses are designed to promote successful performance in each professional assignment, to ease the adjustment to other countries and cultures, and to enhance the leadership and management capabilities of the U.S. foreign affairs community.

Other courses and services help family members prepare for the demands of a mobile lifestyle and living abroad and provide employees and their families with important information about such critical and timely topics as emergency preparedness and cyber security awareness, among others.

3. Scope

FSI is seeking a contractor to provide all computing and support services as well as expert knowledge in the evaluation area required to validate, plan, deploy, integrate, configure, train and provide operations and maintenance services for the system. The CES must be deployed into a target environment that is compliant with DOS policies including Cloud solutions approved by the DOS Cloud Computing Governance Board (CCGB). Upon deployment of the CES, the contractor will provide ongoing services required to operate, maintain, and safeguard the CES, train FSI staff on the use of the CES.

3.1. General

The Contractor is responsible for all planning, scheduling, staffing, management, administration and training requirements for the contract and/or task order. These activities include, but are not limited to:

1. All activities associated with preparing proposals in response to the Department’s SOO

2. All activities associated with recruiting and hiring staff, such as advertising, screening applicants, interviewing, reference checking, etc.

3. Maintaining “in-house” skills, teaming and/or subcontract arrangements to ensure staff with the requisite experience.

4. Screening and processing prospective contract staff to ensure all Contractor employees used under this contract meet personnel hiring and security clearance requirements.

5. Planning for and making all necessary arrangements to ensure that Contractor personnel performing field work have all necessary supplies and equipment.

6. All activities associated with managing subcontractors/team members, such as identifying and qualifying personnel, negotiating subcontracts, reviewing invoices, ensuring compliance with the security and other requirements of this contract, etc.

The above items are not separately priced or billable under the contract or any task order.

3.2. Key Personnel

The following personnel are considered key personnel by the Government:

1. Project Manager. The contractor will provide a project manager who shall be responsible for the performance of the work. The name of this person and an alternate who shall act for the contractor when the manager is absent shall be designated in writing to the Contracting Officer. The contract manager or alternate shall have full authority to act for the contractor on all contract matters relating to daily operation of this contract. The contract manager or alternate shall be available between, 7:00 a.m. to 5:00 p.m. and as needed during system upgrades and updates, Monday thru Friday except Federal holidays or when the

Government facility is closed for administrative reasons.

2. Evaluation Subject Matter Expert. The contractor will provide an evaluation expert who shall ensure that the proposed system capabilities adhere to evaluation best practices.

3. System Architect Subject Matter Expert: The contractor will be an expert at the platform/application and network environment proposed with proven experience at a similar position in a project of similar magnitude. The contractor will also be responsible for the system design, integration, and configuration the solution to meet FSI’s requirements as well as to technical configuration and documentation.

No key personnel identified in the contractor’s bid/proposal may be replaced or switched without prior approval by FSI. Replacements must have equal or greater professional qualifications as the originally proposed personnel.

Qualifications for key personnel are listed below:

Project Manager

Manage contractor staff in order to provide support a team of government Program Managers (PM). Manage Contractor staff and mitigate any issue that may arise. Prepare program documentation, communicate frequently with performers, leadership, PMs and government staff.

Support the PM by interfacing with government contracting officers with various program requests, outreach strategies, and solutions leveraging core capabilities. Prepare program documentation, briefings, and participate in proposal concepts, evaluation assistance, relationship building, and facilitate the transition of developed technology to clients and deployment. Assist with the establishment of technology project metrics, publish technical analysis in support of the government needs, and generate and present comprehensive technical briefings, papers, and recommendations. Provide support to the PMs in planning process for advancing technologies by combining strategic planning along with detailed technical assessment and analysis of alternatives. Conducting background research to discuss, discover, and evaluate the feasibility of new concepts and shaping research program goals. Developing and researching ideas for the formulation of programs, including supporting in the assessment of proposed ideas and determining the technical merit, methodologies, innovation level and risk, and benefit. Assist with the management of R&D programs and performers in formal design reviews and informal status meetings.

Minimum Education Required:

• BS/BA degree in Computer Science, Information Sciences, or related IT discipline.

o Allowable Substitution: Additional ten (10) years of related professional experience can be substituted for a BS/BA degree.

• Project Management Professional Certification (PMP)

Minimum Experience Required:

• A minimum of 10 (ten) years of professional experience in IT project management is required.

• A minimum of 15 (fifteen) years related in the software development field.

• Experience as a Team Lead on large projects or significant segment of large complex projects.

• Knowledge of project planning, monitoring, and financial activities related to program management.

• Knowledge of managing multiple project teams and meetings.

• Knowledge of projects and programs with well-defined metrics.

• Knowledge of program management and technical monitoring performed by a contracting officer’s technical representative for the US government.

• Experience planning, scheduling, executing and monitoring projects of varying scope, sizes and durations; overseeing the day-to-day activities of one or more cross-functional project teams; overseeing all aspects of projects.

• Experience delivering projects on-time, on budget, and meeting functional and quality specifications.

• Experience managing issues, risks, and change using appropriate and agreed upon processes and tools established within the POC.

• Experience implementing and managing continuous process improvement (CPI), quality assurance (including, but not limited to, tracking performance, cost outcomes, CPI awareness training).

• Able to fully utilize Microsoft Project Manager.

• Experience overseeing all projects phases, including designing, planning, managing and executing SDLC lifecycle.

• Experience creating and enforcing release plans and change management.

• Experience prioritizing work to ensure alignment with business goals and objectives.

• Experience proactively managing risks and issues; developing and executing risk response plans, running point on project escalations and monitoring issues to resolution.

• Experience measuring, monitoring and reporting work progress to ensure high-quality deliverable are produced on-time, within budget, and meet or exceed commitments and expectations.

• Experience documenting and supporting the implementation of project management process improvements and completing all documentation required for the project.

• Experience analyzing new and complex projects and related problems; creating innovative solutions involving finance, scheduling, technology, methodology, tools, and solution components.

• Experience preparing long and short-range plans for application selection, systems development, systems maintenance, and production activities and for necessary support resources.

• Experience as a subject-matter expert for multiple complex work assignments

Evaluation Subject Matter Expert The Evaluation Subject Matter Expert (SME) will apply their in-depth knowledge of evaluation to advise the contractor’s team as they develop and implement the CES. Support the contractor team in integrating best practices into the evaluation functionality of the CES and ensure that the functionality is technically correct. Maintain communication with the Government to ensure

Government evaluation specifications and requirements (as stated in the SOO and the FSI Policies and Standards) are understood by the contractor and accurately translated to the contractor team. Ensure that the CES complies with FSI Policies and Standards (see Appendix G). Advise on the implementation plan and deployment (including training) of the CES at FSI to ensure evaluation functionality is at the forefront of deployment considerations. Advise FSI users as they learn to work with the CES. Coordinate with each FSI School and Center to align and configure the CES to specific School and Center Standard Operating Procedures.

Minimum Education Required:

• MA/MS in Program Evaluation and Measurement, Program Evaluation, or related evaluation discipline.

Minimum Experience Required:

• A minimum of 10 (ten) years’ experience in applying the theories, principles, methods, and practices of evaluation for the purpose of measuring effectiveness of training programs.

• Knowledge and experience in the design of education and/or training courses.

• Experience developing and using reliable and valid learner assessments.

• Experience developing and using a variety of evaluation data collection instruments.

• Experience evaluating the reliability and validity of data collection instruments.

• Knowledgeable of measurement properties and psychometric testing of evaluation instruments

• Experience using a wide variety of quantitative and qualitative research design approaches.

• Experience analyzing quantitative and qualitative data.

• Knowledge and experience developing and designing evaluations for programs of varying scales and durations.

• Experience identifying the most effective approaches, tools, instruments and techniques for evaluating training effectiveness for a wide variety of audiences and topics.

• Experience coordinating the development of policies and processes to support the appropriate use of data-collection methods and approaches in order to evaluate program and course effectiveness.

• Experience writing evaluation plans and reports

• Familiarity with the Kirkpatrick training evaluation model

• Skilled at communicating and reporting technical issues to non-technical audiences

System Architect Subject Matter Expert:

This position requires a security clearance of secret level.

A System Architect design and develops systems; identifies major components; determines impact on workflow processes; and develops testing strategies, test cases and testing plans. This position will also design databases and developing methods used in these databases.

Minimum Education Required

• MS degree in Computer Science, Information Sciences, Cyber Security or related IT discipline. Allowable Substitution: BS or BA degree and an additional ten (10) years of professional experience related to the specific position can be substituted for an MS or MA degree.

Minimum Experience Required:

• The most recent 10 years’ experience must demonstrate work as System Architect;

designing and implementing information systems involving hardware, software, database storage, and network platforms including cloud.

• A minimum of 15 (fifteen) years related in the Software Development field.

• Evaluate applications already in place. Create diagrams representing current application designs, data architecture and flows. Evaluate cloud applications, platforms, infrastructures and standards in order to align with current and future business strategies. Evaluate and recommend COTS, Cloud or GOTS, application software packages.

• Must be able to envision solutions to complex problems and be able to explain and demo them to technical and non-technical audiences

• Implement different authentication methods, such as Single Sign-On via Active Directory, two-factor authentication, ADFS, SAML and OAuth 2.0

• Expert knowledge on monitoring and improving system performance including tuning, application profiling, code scanning tools and able to facilitate code reviews

• Expert knowledge of Relational Databases and creating stored procedures. Able to produce ERD with normalized data structures. Ability to develop, troubleshoot, and deploy database T-SQL scripts.

• Develop technical roadmaps for future implementations, integrations and redesigns.

Prepare and provide software-related technical documentation.

• Identify problems and recommend solutions using evidence-based, data driven analysis.

Produce estimates and lead the execution of approved work streams and tasks.

• Document systems, SOPs and best practices for software developers, QA, DB Admins and other engineers.

Key Personnel

(a) The Department reserves the right to review the qualifications of all staff identified as “Key Personnel” that are selected to work on any task order before assignment, including the individuals proposed (in the Contractor's proposal submitted) and any replacements for these individuals, and to reject individuals who do not meet the qualifications set forth in this contract. Any exceptions to the labor category qualifications require written approval from the COR.

(b) The Department reserves the right to require the Contractor to remove or reassign from this Contract any “Key Personnel” Contractor employee(s) who is/are deemed incompetent, careless, unsuitable, or whose continued employment under the contract is deemed contrary to the best interests of the Government. Failure of “Key Personnel” Contractor employee(s) to comply with the Department’s standards of conduct may also be grounds for the Department or the client agency (thru the Department) to require removal or reassignment of offending Contractor employee(s).

The Contractor shall remove or reassign any “Key Personnel” employee from performance of contract tasks within five (5) working days of receiving a written notice from the Contracting Officer that the employee's performance or conduct is unsatisfactory or unacceptable. Further, the Contractor shall immediately remove any “Key Personnel” Contractor employee found to represent a real or anticipated threat to the safety of government records, Government employees, or other Contractor employees. The Contractor shall reimburse the Department for the cost of any Department-provided or funded training given to “Key Personnel” for Contractor employees who cease to perform contract duties within six (6) months after receiving that training.

(c) At a minimum, the following positions/labor (Project Manager and Evaluation Subject Matter Expert) categories are considered “Key Personnel” positions for this Contract at the contract level.

(d) Any individual performing work under the labor categories listed in paragraph (d) above is subject to the following:

(1) Replacement of any Key Personnel is subject to the prior written consent of the COR or his/her designee.

(2) Requests for replacement shall include a detailed resume containing a description of position duties and mandatory qualifications, as well as information about the qualifications of the individual(s) proposed. All replacement personnel must provide the same levels of effort as the replaced staff, and must be at similar levels (or better) of education, qualifications and experience of the personnel to be replaced, with no upward adjustment to the prices set forth in the task order.

(3) Contractor proposals to move any Key Personnel off the contract or a task order, or from one task order to another shall be submitted in writing at least thirty (30) days in advance of proposed move, and are subject to the approval of the COR, or his/her designee, including approval of proposed replacement.

(4) In the event of a key personnel resignation, the Contractor shall notify the COR in writing within five (5) days of receiving notice of resignation.

3.3. Period and Location of Performance

The Period of Performance will be comprised of a one (1) year base period with six (6) one- year option periods.

The government requires that implementation of the CES system to production must occur no later than the end of the base year of contract performance, which includes meeting all objectives stated in this document. By not later than the end of base year, the contractor must complete the configuration, customization, and deployment in order to have a fully functional system with an ATO awarded by the Department of State.

The government requires that the vendor provide a complete project management plan within the first 60 days of this effort, to be approved by the COR, identifying positions working onsite vs offsite.

The primary place of performance will be offsite at the Contractor’s site. The Government will require meetings related to this effort to be performed onsite within the location listed below. The post-deployment performance location for ongoing CES product maintenance and development will also be offsite at the Contractor’s site.

U.S. Department of State, Building Annex SA-42

George P. Shultz National Foreign Affairs Training Center (NFATC) 4000 Arlington Boulevard

Arlington, VA 22204

Alternate location: U.S. Department of State, Building Annex SA-15 Pomponio Plaza East 1800 N. Kent Street Arlington, VA 22209

Alternate location: U.S. Department of State, FSI Annex 1200 Wilson Boulevard Rosslyn, VA 22209

For contractor staff working onsite, FSI will provide a maximum of four (4) workspaces including equipment, materials, and information necessary for contractor staff to perform tasks associated with this effort. Transportation between contractor facilities and FSI will not be charged to the government.

For performance being conducted offsite, the contractor will be responsible for having the appropriate workspace, equipment, and material necessary to provide all deliverables requested by the government. The government will provide needed equipment for remote network access if required to perform offsite work. Travel in and/or around the local Washington, D.C., metro area for the purpose of working at the contractor’s place of performance will not be charged to the government.

Occasional attendance at the request of the government for meetings or other events offsite in the local Washington, D.C. metro area may be required. The government will notify contractor personnel of meeting requests in advance of attendance.

4. Current Environment Currently FSI does not have a Centralized Evaluation System in place to manage training evaluations. FSI currently maintains the Student Record System, a custom-developed computer application located inside Department of State (DOS) internal network that enrolls students into FSI classes, provides billing services, tracks courses offered, and produces a variety of training reports. The new CES will not replace this Student Record System, but rather integrate with it. Requirements pertaining to the integration of the Centralized Evaluation System and the Student Record System.

Existing/Legacy Environment Components

• FSI’s standard desktop environment consists of the following items:

• ffice 2016 Professional Plus

• Microsoft Officrosoft Edge

• verlight 5.1

• Adobe Reader v11

4.1. Adobe Shockwave Player 12.2tiveOther Devells to be encountered

• Micrt .NET Frameworker 2012 Managemenundation Server 2015

4.2. Adobe ShockFlasher 12.2AActiveX 23 Java 8 64-bitoe2016365Microsoft Office 2016 Professional PlusGoogle ChromeMicrosoft Windows 10FSI’s standard desktop environment con:

5. New Centralized Evaluation System FSI requires an integrated, modern, customer centric CES that will manage the training evaluations required to educate and prepare a globally deployed student audience performing the Department’s mission critical activities. The CES must provide a centralized data repository; manage accounts; enable instrument design and development; collect data; analyze and report data; manage and administer instruments; and integrate with other systems while meeting technical and security requirements with the flexibility to adapt to changes in U.S. Federal policy and data governance while maintaining alignment with DOS strategic goals and FSI policies and standards. The CES features must be easily integrated with State departments systems.

The new Centralized Evaluation System (CES) and the target environment sought by the Institute must enable it to meet all objectives expressed in this section and fulfill all “mandatory” requirements expressed in Appendix A of this document.

In addition, the CES and the target environment will provide FSI leadership and policy administrator visibility into business operations and allow for changes to be made over time in as flexible a manner as possible. The CES solution must continue to provide upgrades and releases to the purchased COTS system that keep pace with changing technology over the life of the system.

The CES, the target environment and associated support services technologies sought must be compatible with FSI’s standard desktop operating environment listed under the section entitled “Existing/Legacy Environment Components” above.

5.1. Centralized Data Repository

The Centralized Evaluation System must be able to store large amounts of quantitative and qualitative data collected through surveys and other instruments over multiple years. The system must also allow storage and retrieval of data collection items and instruments. Metadata must be used for tagging purposes and to allow for advanced queries of specific areas of interest. See Appendix A for specific data repository requirements.

5.2. Account Management

A centralized user account repository must be used to create accounts, reset user passwords, delete and disable user accounts, enforce access through groups, roles, and permissions. See Appendix A for specific account management requirements.

5.3. Instrument Design and Development

Authorized users must be able to create instruments of various types to be used for evaluation and assessment of training. See Appendix A for specific instrument design and development requirements.

5.4. Data Collection Capability

The Centralized Evaluation System must provide a variety of means for collecting data. One method might be a survey in which students are asked to report on the perceived usefulness of the class directly after a class ends. Students might be asked to provide feedback and complete data-collection instruments 6 months, 1 year, and 2 years after the class ends. These data would be tied to one another in some way, so analyses can be processed using the data, and so users with appropriate permissions could see that the data are related. Another data collection method might be a telephone interview with the student’s supervisor in which the interviewer enters comments into the system on behalf of the supervisor. The system must also be able to manage a 360o evaluation. See Appendix A for specific data collection requirements.

5.5. Data Analytics and Reporting

The CES must provide predefined reports and dashboards will be necessary to report and summarize the data. Additionally, custom queries and ad-hoc queries must be able to be created to gather information to answer specific questions or to improve the quality of training. Third party data visualization and data analytics tools (such as Tableau Desktop and Power BI) must be able to successfully connect to the data storage. See Appendix A for specific analysis and reporting requirements.

5.6. Instrument Management and Administration

Authorized users must be able to manage and administer the use of instruments of various types for evaluation of training. See Appendix A for specific instrument management and administration requirements.

5.7. Integration with Other Systems

The Centralized Evaluation System must interface with the existing Student Record System and its databases containing information about courses and students—including their names, job titles, pay grades, etc.. The Centralized Evaluation System must support an integrated list of instructors for courses that have a large number of instructors. In the future, the CES must be capable of connecting with FSI’s cloud-based learning management system via API. See Appendix A for specific integration requirements.

5.8. Technical and Security Requirements

The selected contractor must comply with technical and security requirements stated in this document as well as Department of State modifications required to obtain all the necessary and appropriate assessment and authorization to operate certification. See Appendix A for specific technical and security-related requirements.

5.9. Contractor Support

The selected contractor must provide several levels of customer and technical support (L2 and L3 for server administrators/side). See Appendix A for specific support requirements.

6. Constraints:

This section lists laws, rules, regulations, standards, technology limitations and other constraints that the service and/or service provider must adhere to or work under.

6.1. Access Control

In accordance with Foreign Affairs Manual reference 12 FAM 643.2-1 https://fam.state.gov/, the contractor must be responsible for the following access control safeguards:

6.1.1. System managers will restrict access to data, programs, etc., based on the identity of the users and/or the functional role for each group to which they belong.

6.1.2. System managers will restrict access to data, programs, etc., based on the sensitivity (as represented by a label) of the information contained in the data, program, etc. and the formal authorization (i.e., clearance) of users to access information of such sensitivity.

6.1.3. System managers must limit access to system software, utilities, and system services that could be used to gain unauthorized access to system data and software through network services.

System managers will prohibit users from accessing functions to which they have no defined need.

6.1.4. Allow System Administrators to create and manage system permissions and assign those permissions to roles, so that those permissions can be assigned to Users based on their job role and division.

6.1.5. Allow System Authorized Users to assign roles to Users based on the needs of their job role and division, so that they can protect data confidentiality and ensure separation of duties.

6.1.6. Allow Users to request to be granted explicit system permissions, so that Users can perform job duties that are outside the normal scope of their role.

6.1.7. Allow the System and Users to routinely certify User information in order to maintain data and acknowledge user access.

6.1.8. Allow System Authorized Users to remove system permissions and roles from a User, so that they can ensure that the User can no longer access related systems, subsystems, processes, and data.

6.1.9. The System needs to remove all permissions and roles from any Users who are inactive for a certain period of time, so that only Users with a current business need can access the system.

6.1.10. Allow System Authorized Users to view all Users who have system permissions and roles, so that they can determine who may or may not need to have those permissions.

6.1.11. Where possible, system managers must ensure that all systems’ access permissions default to no access for all access types definable for the connection, until a specific need has been established and approved. (Access types include read, write, modify, execute, delete, append, etc.) Once a need has been established, the principle of least privilege must be employed allowing only the level of access necessary for a user’s job duties.

6.1.12. The system manager must configure service windows for the SIS and associated support technologies for only those hours operationally required. After-hours access must not be configured unless a specific operational need has been identified and documented by the system manager and ISSO.

6.1.13. System managers must restrict node-hopping programs to those users who have a legitimate need to access remote systems. Where a need for intermediate logons by users to gain access to distant systems has been established, system managers must limit the functionality of these node-hopping programs in order to promote controlled access. These programs can be limited using system-level security controls, locally developed procedures, or other mechanisms that control the selection of input parameter values.

6.1.14. System managers must ensure that users are limited to only one remote logon session at a time

6.1.15. System managers must ensure that SIS and associated support systems comply with applicable agency and Institute access control policies.

6.2. Authentication:

In accordance with Foreign Affairs Manual reference 12 FAM 632.1-4 https://fam.state.gov/, the contractor must be responsible for the following identification and authentication controls:

6.2.1. System managers must configure systems to require user identification and authentication.

6.2.2. The System needs to perform routine activities to identify instances that may require a Student User Account to be automatically updated or disabled.

6.2.3. System managers must configure networked systems to require a Smart Card and passphrase for user authentication. This includes networked devices, e.g., multi-function printers and digital senders that require user authentication. Send requests for exceptions to the Smart Card requirement to IRM/IA.

6.2.4. System managers must configure standalone systems to require either:

• Both a Smart Card and passphrase or Access Code/Pin or

• A password for user authentication

6.2.5. Personnel with elevated system privileges must have separate privileged and user accounts, and the privileged account must not be used to perform user activities, e.g., sending email or accessing external classified web sites/applications.

6.2.6. System managers must immediately disable user IDs under the following conditions:

• Whenever notified by a user's supervisor that the user no longer requires AIS access (e.g., user no longer employed with the Department);

• Whenever notified by a proper authority, such as the human resources officer, that the user's employment has been terminated with the Department; or

• When the user ID is no longer needed (e.g., obsolete account)

6.3. Security Requirements: Personnel Security Clearances

The data that will reside in the Centralized Evaluation System will have PII. Tasks related to the Centralized Evaluation system must be performed in a secure facility.

6.4. Non-Disclosure Agreement

Contractor personnel shall have access to sensitive verbal, written and electronic information while performing under this effort. This information may be security and acquisition sensitive information, which may only be utilized for official use only, or other contractor’s proprietary documentation relevant to the project. In order to protect this information from unauthorized release or use, the Contractor must have their employees sign a nondisclosure agreement, provided by the government, and return to the CO prior to commencing work at the site for the assigned onsite representative.

6.5. Data Ownership

All Centralized Evaluation System data and integrated systems are solely owned by the US Government.

The service provider must not have a copy of production data outside the production environment. The contractor must not use the CES system to complete demonstrations unless the CO or COR provides a written approval.

Appendix A. Centralized Evaluation System Detailed Requirements

This appendix contains the set of requirements for an CES that schools and its support organizations within FSI have identified for the CES.

General Requirements

A.1 The system must provide a means to collect data (e.g., surveys, forms, interviews, focus groups, observations, checklists, etc.)

A.2 The contractor must connect the Centralized Evaluation System (data repository, data analytics and reporting, account management) with FSI’s Student Record System.

This interaction will occur inside the State Department’s intranet.

A.3 The contractor must integrate the CES with the Instructor Record System, a system that tracks instructors in one of FSI’s schools.

A.4 The Centralized Evaluation System must be hosted on a FedRamp Authorized environment (FISMA Moderate rated) or an On-Premise environment.

A.5 The contractor must configure the Centralized Evaluation System to allow respondents to fill out instruments on the public internet without signing into a State Department system -for public instruments- or require users to authenticate -based on the instrument creator requirement.

A.6 The Centralized Evaluation System must be able to collect and present two different types of information. The first is related to training evaluation. The second is related to general feedback (for example, customer service surveys).

While these two types of data may reside in one database or in multiple databases, they must always remain distinct.

A.7 The contractor must enable the system to allow users with the appropriate permissions to present a disclaimer and terms of conditions on the login page.

A.8 The contractor must apply FSI branding to the CES.

A.9 The system must provide customizable branding options for training evaluations and for general feedback.

A.10 The system must have a modern, appealing visual interface for administrators and respondents by following current and open IT standards.

A.11 The system must enable respondents to access instruments on mobile devices (such as smart phones and tablets), laptops, or desktops, and the display must automatically optimize for the device.

A.12 The CES must meet the requirements of Section 508 of the Rehabilitation Act of 1973 as amended (29 U.S.C., 794 (d)).

A.13 The CES must allow users to perform actions only if they possess the correct role and permissions. In order for a user requesting to receive access to a higher role, it must be previously approved by the appropriate POC. Roles are listed and defined in C.

A.14 The contractor must configure the CES to log and report on requests from a user with an appropriate role to view PII. Roles are listed and defined in Appendix C.

Centralized Repository

A.15 The system must provide a robust database structure capable of storing and manipulating data from at least 800 courses that train a minimum of 170,000 enrollees each year, over a period of several years. Historical data must always be accessible.

A.16 The contractor must link the CES’s centralized repository with FSI’s internal Student Record System to transfer data into the CES. Refer to the list of integration requirements (Appendix B) to view a complete list of fields to be retrieved from the internal Student Record System.

A.17 The system must provide a repository that stores and retrieves instruments and responses—that is, surveys, focus group protocols, structured interview questions, course rubrics, observations, tests, quizzes, and other data collection instruments used at FSI. Other data collection instruments could include unsolicited comments and the system must be able to record: How the comment arrived (for example, an email, verbally, an online discussion forum, or an observation), the actual comment, and the associated course number and section number (if the comment is associated with one). A complete list of instruments is found in the Glossary in Appendix D.

A.18 The CES’s centralized repository must allow automated data population from evaluation instruments as well as manually entered data collected by other methods, such as focus groups, structured interviews, and unsolicited comments.

A.19 The CES must have an item bank that will be populated by users with appropriate permissions. Data managers must also be able to tag items so that administrators can search the item bank when creating instruments.

Account Management

A.20 The contractor will implement roles listed and defined in Appendix D.

A.21 After the CES sends an instrument via email to a user, CES must not require the user to log in to complete instruments. The user’s submission must not be anonymous— it should be tracked.

A.22 Student Record System enrollments must automatically reconcile with the CES.

Instrument Design and Development

A.23 The system must enable instrument designers to create multiple data collection instruments and features (surveys, questionnaires, forms, interviews, focus groups, observations, checklists, informal feedback transcriptions, 360o assessments).

A.24 The contractor must integrate the CES with FSI’s internal Student Record System so that the system can automatically populate instruments with School, Division, Course Name, Course Short Name, Course Code, Section, Instructor, Course Completion Date, and/or Delivery Method, depending on the needs of the instrument designer.

Refer to Appendix B to view a complete list of the fields to be retrieved from the internal Student Record System.

A.25 The contractor must enable the CES to allow instrument designers to import course objectives from the Student Record System into instrument items or into a descriptive field.

A.26 The CES must have architecture that supports a listing of all instructors (up to 100 for a given course) retrieved from FSI’s Instructor Record System, for all instruments associated with that course, so that respondents are able to identify and select the instructor at the time the instrument is completed.

A.27 The contractor must ensure staff with appropriate roles will have the ability to edit the listing of instructor names associated with courses that have large numbers of instructors; the CES will enable users with appropriate roles to create and maintain the instructor list within the CES.

A.28 The contractor must enable users with certain roles to manually enter information about courses, instructors, and students into instruments.

A.29 The system must allow an instrument to be created by copying an instrument that already exists within the same user account.

A.30 The system must support question branching and skip logic.

A.31 The system must support the Likert scale and the customization of scale values/weighting. For example, if the possible responses to a question are Very Frequently, Frequently, Occasionally, Rarely, and Never, instrument designers can have the Very Frequently answer be the first (left-most) answer or the last (right-most) answer.

A.32 The system must provide for validation of data entries (for example, if a text box requests a year, the input must be numbers and not letters).

A.33 The system must provide dropdowns and other structured data fields wherever possible to reduce data entry errors and facilitate consistent data formats.

A.34 The system must permit instrument designers to insert questions (between existing questions) in design mode.

A.35 The system must permit instrument designers to rearrange questions.

A.36 The system must permit users with an appropriate role to populate an item bank that will be available FSI-wide. Items in the item bank will be tagged to enable searching.

A.37 The system must allow users with an appropriate role to modify a question after it is added to the item bank.

A.38 The contractor must enable the CES to permit users with the appropriate role to assign metadata tags to questions and corresponding responses (data) using custom metadata dictionaries while creating and modifying instruments. This data structure will support analysis and mapping.

A.39 Tags must include Kirkpatrick Levels 1, 2, 3, and 4 and various constructs being measured (effectiveness, relevance, reach, and impact, at a minimum).

A.40 The contractor must enable instrument designers to assign multiple tags to an item.

A.41 When instrument designers add or modify a metadata tag to an existing question, the system must prompt them for a decision if (a) all new data are to be tagged, or (b) all existing data and all new data are to be tagged.

A.42 The system must permit instrument designers to add course-specific questions appended to FSI-wide or School/Center-wide questions. For example, in a 10-question instrument, the first six questions might be School/Center-wide and questions 7 through 10 would be course-specific.

A.43 The system must permit instrument designers to mark individual questions as being required (for example, with an asterisk) and ensure that the respondent will answer these questions before submission.

A.44 The system must accommodate a wide variety of question types and response styles—for example, true/false (binary), multiple choice, multiple correct answer, short answer, embedded answers (Cloze), checklist, dropdowns, drag-and-drop (with proper tagging), slider response, rankings, multiple textboxes, matrix, and radio buttons. Instruments are listed in the Glossary in Appendix D.

A.45 The system must accommodate questions of type “grid.” At FSI, a grid question type permits the respondent to provide an answer by interactively manipulating an image on the computer screen. For example, he or she may click a specific spot on a picture of a map to indicate where something (say, a hospital) should be located.

A.46 The system must permit item “piping,” where answers from items are inserted into stems of follow-on items (for example, respondent’s name, language, or location).

A.47 The system must permit instrument designers to add comments that explain to other designers how/why the questions or format are structured a certain way.

A.48 The system must permit instrument designers to establish templated blocks of questions.

A.49 The system must permit instrument designers to show/hide individual questions or a group of questions—on the same page or future pages—based on prior responses.

A.50 The system must permit instrument designers to change a question’s format after it has been built (for example, convert a multiple choice [check boxes] question to option [radio] buttons).

A.51 The system must permit instrument designers to edit answer choices or rearrange answer choice sequences after they are created (but not yet live or live with no responses).

A.52 The system must permit instrument designers to randomize survey items, randomize pages, and randomize a block of questions.

A.53 The system must permit instrument designers to add an “Other (please specify)” and “Comment” for the same question with the ability to turn off character limits.

A.54 The system must permit instrument designers to add a question with a qualitative response box with the ability to turn off character limits.

A.55 The system must provide instrument designers the ability to use images (for example, pictures of buildings) as answer choices. The pictures might or might not have labels.

A.56 The system must permit authorized users to import instrument templates (of questions) from sources such as Microsoft Word and Microsoft Excel.

A.57 The system must permit instrument designers to create and deploy 360° assessments and store/analyze data from responses.

A.58 The system must permit instrument designers to adjust page layouts, to include vertical and horizontal spacing.

A.59 The system must permit instrument designers to create and save custom layout themes (color, font, etc.) and insert a logo on individual instruments if desired.

A.60 The system must provide instrument designers with predefined navigation buttons.

A.61 The system must support a customizable progress meter (preferably several options, such as bar, pages, and percentage) to show the respondent while he or she completes an assessment.

A.62 The system must support optional instrument titles or headers, page or section titles, and footers so, for example, instructions for completing the instrument or a thank you text can be included.

A.63 The system must support auto-numbering of questions (which will automatically adjust based on logic and branching).

A.64 The system must provide the instrument designer with the ability to save a partially designed instrument and complete it later.

A.65 The system must support sending a test message with a link to a draft instrument to a specified email address prior to releasing the survey for responses.

A.66 The system must provide the capability to purge test data; the “delete test data” option will prompt the designer to confirm the action before deleting the test data.

A.67 The system must support version tracking—that is, enable designers to revert to an earlier version of an unpublished survey.

A.68 The system must…