ACTS SOW - 10-03-2017.pdf

Attachment 1

STATEMENT OF WORK (SOW)

AIRMAN CERTIFICATE TESTING SERVICE (ACTS)

Federal Aviation Administration (FAA)

Flight Standards Service Regulatory Support Division

Airman Testing Standards Branch/AFS-630 Mike Monroney Aeronautical Center (MMAC)

Oklahoma City, OK

October 3, 2017

INTRODUCTION

U.S. domestic airmen include pilots, mechanics, repairmen, dispatchers, parachute riggers, flight navigators, and flight engineers. An individual must be certificated to operate as an airman in the National Airspace System (NAS). To earn an airman certificate, an applicant must successfully complete appropriate knowledge and practical tests to demonstrate mastery of the knowledge and skill needed to safely operate in the NAS.

Successful completion of the Federal Aviation Administration’s (FAA) airman knowledge test meets the airman knowledge demonstration requirement and allows the applicant to take the practical test.

All U.S. airman knowledge tests are currently multiple choice form tests that are developed and maintained by the Regulatory Support Division’s (AFS-600) Airman Testing Standards Branch (AFS-630). The majority of tests are delivered to applicants through an FAA-designated test provider who holds an Organization Designation Authorization (ODA). The ODA provides its own scheduling system and service support (help desk). AFS-630 provides test items and form test blueprints to the ODA via a file transfer protocol (FTP) server. The ODA, at its own expense, groups the questions into tests according to the FAA’s form test blueprints and provides the tests to its test sites via its independent test driver/delivery system. The test sites administer the tests and send the test results and data back to the ODA. The ODA puts the exam results on the FTP server where they are picked up by the FAA and added to a database where a stored procedure creates a table that is sent to the FAA’s certification support system, currently the Integrated Airman Certification and Rating Application (IACRA).

On occasion the FAA delivers tests via Flight Standards District Offices (FSDOs). The FAA, the U.S. Military, and foreign Civil Aviation Authorities (CAA) or Regional Safety Oversight Organizations (RSOO) utilize a bootable Knoppix CD to connect the non-ODA test sites to the FAA servers for test delivery.

There are permanent and temporary test sites in the Airman Testing System (see Appendix 5 for types and locations). In accordance with this Statement of Work (SOW), the winning Vendor is expected to deliver tests to sites in accordance with Section 8.2.4.3 of this SOW.

AFS-630 is responsible for:

1. 14 domestic test banks totaling 14,000 items grouped into 89 test types and 444 form tests (See Appendix 3).

2. 12 international test banks totaling 11,000 questions grouped into 88 test types and 144 form tests (See Appendix 3).

3. An external testing system that delivers an average of 150,000 tests annually (this includes the new Unmanned Aircraft System (UAS) tests).

4. Delivering 3,000 tests annually to U.S. Military, temporary and Suriname test sites.

5. Maintaining the existing test banks, test types, and items.

6. Developing new test banks, test types, and items.

7. Oversight and management of the testing system.

The FAA is changing its approach to airman testing and is seeking expert test development, delivery, and assessment services for several reasons.

1. Testing demand is increasing rapidly.

a. Airman knowledge testing for commercial Unmanned Aircraft System (UAS) operators began on August 29, 2016. This increased airman testing by 25 percent. Unlike traditional airman, 14 CFR

107.65 (b) requires these individuals to retest every two 2 years.

b. Future integration of other UAS categories and types of operations into the NAS will also increase testing activity and

c. The demand for the FAA to support CAAs and RSOOs with airman knowledge test delivery is increasing.

FAA’s current business model for testing cannot support these increasing testing demands.

2. The current FAA airman test development system is very tenuous due to its age and vulnerabilities. Its capabilities are very limited and it cannot support the academic innovations AFS-630 must employ.

3. The two methods FAA currently uses to deliver tests are cumbersome and have limited capabilities. One global on-demand test delivery system for all airman knowledge testing delivered by the FAA is required.

4. The FAA does not possess the expertise to ensure its knowledge test items are academically sound. To ensure that airman are appropriately prepared to enter the NAS, the FAA requires services only psychometric experts can provide.

SCOPE OF WORK

The purpose of this request is to procure a knowledge testing vendor(s) (hereafter, “Vendor”) support the development, delivery, and management of airman knowledge items and tests that will enable airman testing applicants to fulfill the FAA’s airman knowledge test requirement.

Using FAA-provided aviation technical expertise for the substantive content of test questions, the Vendor will provide a complete knowledge testing system and make available all professional, technical, administrative, and management support services needed to accomplish the requirements set forth in this SOW. The Vendor must provide all of the resources and logistical support necessary to administer airman knowledge tests, including, but not limited to, a system for scheduling airman knowledge test applicants, personnel trained in all functions related to administration of airman knowledge tests, computer equipment for applicant testing, other required equipment, and physical facilities.

(Test sites in 8.2.3.3.b, d and e of this SOW provide their own proctors, facilities, and equipment.)

As a Federal Government agency, the FAA is required to provide security and privacy protections necessary to mitigate the risk and magnitude of the harm resulting from the unauthorized access, use, or disclosure of information or information systems used or operated by the FAA or by a Vendor for the FAA. All computer systems are required to be compliant with the Federal Information Security Management Act of 2002 (FISMA) as described by the Vendor using the Memorandum of Understanding (MOU) for System Interface template in Appendix 7.

In summary, AFS-600 is seeking a Vendor to:

1. Support the FAA’s development, assessment, maintenance, and enhancement of test items, tests, and supplementary test materials, with automated tools and academic expertise.

2. Provide web-based delivery of tests to airman applicants to support an estimated:

a. Deliver 150,000 airman tests a year at proctored domestic sites including unincorporated territories of Puerto Rico and Guam that the Vendor provides. In 2016, 109,000 traditional airman exams and 16,000 UAS exams were delivered. UAS exam delivery began October of 2016. The first 4 months of 2017 14,800 UAS exams were delivered (3,000 in January; 3,500 in February, 4,500 in March and 3,800 in April). The 150,000 exam estimate was determined by multiplying the January through April 2017 average (3,700), by 12 (44,300) and adding the annual traditional exams (109,000).

b. 3,000 tests administered a year at COR approved test sites that are provided and proctored by:

i. Foreign CAA and RSOOs, these are permanent sites.

ii. The U.S. Military, these are permanent sites.

iii. The FAA, these are Ad-hoc sites that may only deliver one test.

Note: This number will increase as more CAAs, RSOO, and U.S.

Military sites are added (See Appendix 5).

This SOW includes tasking in the areas of:

1. Data migration, requirements, and security.

2. Test registration and proctored testing.

3. Provision of web-based delivery of airman knowledge tests to:

a. Vendor sites where the Vendor provides proctors.

b. FAA identified international, U.S. Military, and temporary sites (typically FAA offices) where the host provides proctors.

4. Updates to the FAA certification support system (currently IACRA) at the conclusion of each test.

5. Web-based automated workflow for item development, item development training and support, as well as item and exam development tools and services.

6. Item and exam assessment services for FAA airman knowledge form tests, test types, and test banks that employ psychometric techniques widely accepted by the academic community.

7. Blueprint development expertise.

8. Data management.

9. Reporting.

This is a zero (0) dollar contract award. The Vendor assumes the responsibility for the airman knowledge testing requirements in this SOW on a no-cost to the Federal Government basis. The Vendor will not be reimbursed by the Federal Government for fees, costs, or any other charge or expense. The Vendor is expected to cover its costs and any profit by charging reasonable testing fees to applicants. The FAA reserves the right to set a maximum testing fee of $160 per test for this contract and will re-evaluate this fee every 3 years. All test fee increases will be approved in writing by the Division Manager responsible for FAA airmen testing (currently AFS-600).

Target dates for the items required in this contract are included in a deliverables chart in Section 23.0 of this SOW. These dates will be finalized in a mutually agreed Program Management Plan (PMP) which includes a schedule and milestones (see 5.1 below).

Progress will be monitored against the PMP and communicated accurately in accordance with the requirements in the Program Management Status Reports (PMSR). Work performed under this contract will be subject to performance review standards relating to quality and timeliness of products produced in response to the requirements in this SOW, compliance with the Memorandum of Understanding (MOU) for System Interface and adequacy of Vendor project management practices.

TYPE AND TERM OF AGREEMENT

The Federal Aviation Administration (FAA) contemplates award of an Indefinite-Delivery/ Indefinite-Quantity (IDIQ) Task Order contract resulting from this Screening Information Request (SIR). The term of the contract is a three (3) Year Base Period and seven (7) one

(1) Year Option Periods.

CONTRACTING OFFICER’S REPRESENTATIVE (COR)

The FAA will delegate a Contracting Officer’s Representative (COR) who will provide technical direction to the Vendor supporting this effort. They will coordinate with the

Vendor to establish the level of effort required to support this contract during the period of performance and will manage the schedule of events in this SOW. The COR does not have the authority to direct the Vendor to make changes that will affect the contract terms or conditions, and/or change the Statement of Work (SOW). A warranted FAA Contracting Officer (CO) has the only authority to modify the contract. The COR will receive all the deliverables from this contract unless he/she has identified a specific alternate for specific deliverable in writing.

PROGRAM MANAGEMENT

The Vendor must assign a dedicated Program Manager (PM), who will be the main point of contact for the COR on all tasks, to organize, plan, schedule, implement, control, analyze, and report on all elements of the contract. The name of the individual(s) shall be designated in writing to the CO at contract award. The PM shall be responsible for the oversight of work performed under this contract and shall have sufficient corporate authority to direct, execute, and control all elements of the program. The types of tasks include: technical, business and administrative planning, organizing, directing, coordinating, controlling and approval actions (e.g., cost/schedule/performance measurement, contract management, data management, etc.). The PM shall be prepared, at all times, given reasonable notice, to present and discuss the current status of contract requirements.

5.1 Program Management Plan

The Vendor shall develop and maintain a Program Management Plan (PMP) to support its task management and reporting requirements. The PMP should specify the work tasks required to meet the objectives at an overview level and must include a timeline detailing the specific dates for completion and delivery of items specified in the proposal. The timeline must be realistic for carrying out the project.

PMP updates shall be submitted as part of the Program Management Status Report (PMSR). Any proposed changes or delays to the PMP shall be discussed with the COR when the PMSR is delivered. The PMP must be provided to the COR and approved by the AFS-600 Division Manager within 30 days of contract award.

REFERENCES

Applicable documents are attached to this SOW.

ACRONYMS AND DEFINITIONS

Acronyms and definitions are listed in Appendix 1 of this SOW.

REQUIREMENTS

8.1 Phase One: Data Transfer.

In this phase the Vendor must transfer the FAA airman knowledge test banks and associated data to its system within 6 months of contract award. It must:

1. Meet requirements in 10.0, 12.0, 13.0, and 14.0 of this SOW prior to data migration.

2. Transfer existing data, (item banks, form test blueprints, testing data, applicant information, etc.) to the Vendor’s system.

3. Electronically validate all transferred data against the original source for accuracy after the transfer is completed and correct it to 100 percent accuracy.

8.2 Phase Two: Test Delivery

8.2.1 Operational Readiness –Within 12 months of completing Section 8.1 above and prior to initiation of test delivery the Vendor must demonstrate, to the FAA’s satisfaction, its ability to:

1. Provide secure testing sites that adhere to applicant authentication, test proctoring, testing room standards and security measures to protect test integrity in accordance with this SOW. Vendor’s method of applicant identification authentication, testing room standards and test monitoring procedures must be approved by the COR prior to initiation of test delivery.

2. Provide reasonable accommodation for testing applicants with disabilities in compliance with applicable law, including the Rehabilitation Act of 1973 and the Americans with Disabilities Act.

Additional information is available at https://www.section508.gov.

3. Deliver tests to a FAA approved test site in Paramaribo, Suriname that the FAA will identify upon Vendor request and a representative sample of test sites described in 8.2.4.3 below.

4. Produce the reports required in Section 16.0.

5. Establish and deactivate proctor access to FAA’s electronic certification support system (currently IACRA)

6. Exchange data with FAA’s electronic certification support system (currently IACRA) in accordance with the requirements in this contract.

7. Ensure its proctors have satisfactorily completed training on functions being conducted on behalf of the FAA. The FAA function training elements must be approved by the FAA and include:

a. The proper administration of the airman knowledge tests.

b. Security awareness and Rules of Behavior.

c. How to access test administration information during testing.

8. Provide controls in place to ensure its employees adhere to the testing requirements in this SOW.

9. Consistently administer airman knowledge tests in accordance with this contract. As part of this requirement, the Vendor must conduct an airman knowledge testing operational readiness practical demonstration. This demonstration must occur no later than 10 months after migrating the items, form tests, test banks, and associated data from the FAA to the Vendor and prior to initial testing of actual applicants.

8.2.2 Scheduling and Registration - The FAA will not vet, approve, or qualify test applicants. All applicants will schedule testing directly with the Vendor.

Currently over 60% of test applicants “walk in”, they do not schedule in advance. The Vendor may schedule/register applicants prior to their presenting themselves for testing, however, ALL eligibility determining documents MUST be validated immediately prior to the test being delivered.

At a minimum, the Vendor must:

1. Provide a secure web-based test scheduling service for airman knowledge test applicants.

2. Inform the public of the location of test sites and the test administration process.

3. Allow flexibility for applicants to choose the date and location for taking the test.

4. Every time for each applicant who holds an airman certificate:

a. Validate that the applicant’s information in FAA’s electronic certification support system (currently IACRA) matches the identification presented by the test applicant. The Vendor will be able to log into IACRA to compare the applicant identification presented at the time of testing to the information recorded in IACRA to determine if it is a match.

b. Transfer the Applicant ID (FTN), last name, first/middle name and name suffix data fields of each airman (not all applicants are airmen) taking a knowledge test from FAA’s electronic certification support system (currently IACRA) to the Vendor’s system and make that information part of the applicant’s testing information for the test being administered.

5. Ensure each applicant has government-issued identification that includes a photo, signature, date of birth, and permanent physical mailing address. The Vendor must verify the identities of:

a. U.S. citizens and resident alien airman knowledge test applicants with one U.S. or State Government-issued identification that meets the requirements above.

b. Test applicants who are not U.S. citizens or resident aliens with two government-issued identifications, one of which is a passport, that meet the requirements above.

6. Determine the applicant meets the FAA airman knowledge test eligibility requirements in Appendix 4 of this SOW to take that test before it is administered. The Vendor will contact the FAA employee designated by the COR if clarification of eligibility requirements is needed.

7. Be capable of placing and releasing an automatic or manual registration hold on any applicant to prevent testing.

a. Automatic holds must be in accordance with the table located in Appendix 2 of this SOW.

b. Manual holds must be provided for any specified period of time including "indefinitely" as directed by the COR.

8.2.3 Testing. Within 30 days of completing the operational readiness requirements in 8.2.1 the Vendor must:

1. Provide proctored test administration. Proctors must:

a. Not proctor tests administered to their own students.

b. Not be a FAA-designated examiner.

c. Have the following qualifications:

1. Be a citizen or legal resident alien of the United States if the proctor is located in the U.S. or a territory of the U.S.

2. Have no felonies in previous 5 years.

ii. Read, write, speak, and understand the English language. (See Advisory Circular AC 60-28A, English Language Skills Required by 14 CFR parts 61, 63, and 65).

iii. Successfully complete the Vendor’s proctor training on the authorized functions being conducted on behalf of the FAA.

Note: Any deviation from these qualifications requires prior written approval from the FAA employee designated by the COR.

2. Provide a web-based testing system that:

a. Includes secure testing sites that adhere to strict security guidelines, including applicant authentication, test proctoring and measures to protect test integrity.

b. Prevents the applicant from exiting the testing environment and entering any other platform available on a computer connected to an external or internal network while a test is being administered.

c. Monitors every applicant to prevent cheating or other unauthorized conduct addressed in Title 14 of the Code of Federal Aviation Regulations (14 CFR) parts 61.37 and 65.18. The Vendor must ensure no person is left unsupervised in the test room during the testing period.

d. Offers a means for applicants to make notes or perform mathematical calculations during the test and ensures those notes and calculations are destroyed at the conclusion of the test.

e. Accepts input from one or more devices for answering test questions (e.g. keyboard, mouse, touch screen, etc.).

f. Requires the applicant to acknowledge the test type being administered, name, and date of birth prior to the start of the test.

g. Generates a unique test ID number for each test.

h. Automatically changes the sequence of responses for each item prior to test delivery.

i. Automatically changes the sequence of items on each form test, while keeping “topic” groups together, prior to test delivery.

j. Captures the applicant’s answers immediately.

k. Limits data loss to one applicant response during an exam.

l. Provides an average system response time of less than 2 seconds.

m. Enforces the maximum test session length for that test type.

n. Displays test progress information for the applicant.

o. Displays test supplemental material on screen. The FAA is open to working with the Vendor to determine, and take advantage of, the best delivery solutions for supplementary test material. If unable to provide the supplemental material on screen when test delivery begins, a plan to maintain security of the written supplemental material and provide supplemental material electronically must be provided to the COR and approved by the AFS-600 Division Manager prior to the initiation of test delivery. Test supplemental material can be found at:

http://www.faa.gov/training_testing/testing/supplements

p. Offers an on-screen calculator.

q. At the conclusion of each test, generates and prints a test report for the applicant that, at a minimum, includes:

i. Name of applicant.

http://www.faa.gov/training_testing/testing/supplements

ii. Title of test.

iii. Test score.

iv. Pass/Fail, as appropriate.

v. Date the test report expires in accordance with 14 CFR part

61.39. (a) (1) (i) and (ii).

vi. Applicant ID.

vii. Date of test.

viii. Test type code.

ix. Unique test ID (see 8.2.4.2.g).

x. Either the Learning Statement Codes (LSC) or Airman Certification Standards (ACS) Codes and information, as appropriate for the test taken. For tests based on:

1. LSC, print codes in numerical order according to the LSC Reference Guide and include the LSC description of the code. For example:

PLT039; Interpret airport landing indicator

For LSC and associated descriptions go to http://www.faa.gov/training_testing/testing/media/LearningSt atementReferenceGuide.pdf.

2. ACS codes, print codes in numerical order according to the ACS and include the ACS description of the code. For example:

PA.IV.B.K6; Emergency procedures during approach and landing.

For ACS codes and descriptions go to https://www.faa.gov/training_testing/testing/acs/ and select the ACS for the appropriate exam.

http://www.faa.gov/training_testing/testing/media/LearningStatementReferenceGuide.pdf http://www.faa.gov/training_testing/testing/media/LearningStatementReferenceGuide.pdf https://www.faa.gov/training_testing/testing/acs/

The authorized instructor’s statement below (9-point font minimum) when an applicant fails a test. Ensure that this statement appears on the same page as the applicant’s name and test type and that there is adequate space for the instructor to enter the required information.

On _______________ (date) I gave the above named applicant _________ hours of additional instruction, covering each subject area shown to be deficient, and consider the applicant competent to pass the knowledge test.

Name __________________________________

Cert. No. _______________________(print clearly)

Type of instructor certificate_________________________________________________

Signature_________________________________________________

r. Includes a process to capture and effectively address complaints from test takers, to include those dealing with test environment, exam content, test questions, and scoring. The process will include identification of complaints appropriate to elevate to the FAA. The Vendor will summarize the survey information and provide it to the FAA in an agreed upon report format monthly. For elevated complaints the FAA will either provide a direct response to the applicant, or return the complaint to the Vendor for response.

s. Includes a process that deactivates a proctor’s access to FAA’s electronic certification support system (currently IACRA) within 24 hours of the proctor no longer having responsibilities for tasks provided under this contract.

t. Delivers web-based sample airman tests to applicants preparing to take a knowledge exam. Currently sample tests are available from the AFS-630 website at no charge (http://www.faa.gov/training_testing/testing/test_questions/). Test sites may allow applicants to take sample tests at a test site but applicants must NOT be allowed to take sample tests at test sites when airman tests are being delivered to that site. Sample tests may be delivered to home/personal computers. Initially, sample tests will be existing form tests identified by a “W” at the beginning of the test code (See Appendix 3). At the end of the test, the Vendor will allow the sample test taker to view and print the title of the test and Learning Statement Codes or Airman Certification Standards Codes of the questions missed on the sample test.

http://www.faa.gov/training_testing/testing/test_questions/

u. Updates the FAA certification support system (currently IACRA) at the conclusion of each test in accordance with the MOU required in Section 12.0 of this SOW, with:

1. Exam Date

2. Grade

3. Exam Code

4. Score

5. Take

6. Last Name

7. First/Middle Name

8. Applicant ID

9. Exam Title

10. Site ID

11. Expiration Date

12. Learning Statement (string of learning statements missed)

13. Name Suffix

v. When requested by the applicant, issue a replacement test report to any applicant that the Vendor has tested in accordance with this contract.

3. Deliver tests to the following test areas/sites (see Appendix 5):

a. Vendor supplied domestic and international testing locations that will provide the same geographic coverage (within 25 miles) as current ODA test sites. These sites can be found at:

http://www.faa.gov/training_testing/testing/media/test_centers.pdf The Vendor may establish additional domestic test sites and temporary domestic test sites at its discretion.

Deviations to the 25-mile requirement must be approved by the COR on a case by case basis. Approvals will be based on Vendor justification that the number and location of test sites sufficiently maintains adequate coverage for testing within a given area. After considering the Vendor’s written justification and determining that sufficient opportunity to test will be maintained, the COR may approve:

I. eliminating test site location(s) described. above or

II. an alternative plan to the site locations described above.

b. U.S. Military sites which can be found at:

http://www.faa.gov/training_testing/testing/media/faacenters.pdf.

During the last 12 months the FAA delivered 2,800 tests free of charge to 40 domestic, U.S. Territory and internationally located U.S. Military test sites. The Vendor is expected to work with the http://www.faa.gov/training_testing/testing/media/test_centers.pdf http://www.faa.gov/training_testing/testing/media/faacenters.pdf

U.S. Military when making decisions concerning adding or eliminating individual U.S. Military sites. U.S. Military test sites are completely self-supported; the military provides the facilities and staffing required for these test sites and only requires delivery of airmen tests from the Vendor. The FAA understands the Vendor will incur costs “adopting” these third party sites and monitoring their compliance with the Vendor’s requirements and the requirements of this SOW. The Vendor will not use funds collected from other test sites in accordance with this SOW to pay fees incurred at U.S.

Military test sites. However, we anticipate that, since U.S. Military test sites are completely self-sufficient and because the Vendor appreciates the services provided by U.S. Military personnel, the Vendor will provide tests to U.S. Military test sites at a significantly discounted rate.

c. Unincorporated U.S. territory test sites. These sites are permanent and supplied by the Vendor. This includes test sites in Barrigada, Guam; and San Juan, Puerto Rico. We recently received a request for a site at Saipan, Mariana Islands that we expect will be approved. The FAA realizes these remote sites that deliver a limited number of airman tests will be a challenge to maintain economically.

The FAA will consider alternative plans for SOW compliance for these remote sites on a case by case basis. Any alternative method of compliance to the requirements in this SOW, must be approved by the COR.

d. Foreign CAAs and RSOOs approved by COR. CAA and RSOO sites are completely self-supported and require only that the Vendor deliver airman tests to their sites. Payment for tests delivered at these sites or any other service (assessments, item development, etc.) requested by a foreign CAA or RSOO will not be taken from domestic fees collected as a result of complying with this SOW.

Please refer to Section 11.0 of this SOW.

e. FAA temporary test sites. These are usually FAA district offices.

The COR must approve FAA temporary test sites.

All new international test sites must be approved by the COR.

Unless alternate means of compliance are approved, all test sites must meet the requirements of this SOW.

The COR must be given a 30-calendar day advance notice prior to initiation of airman knowledge test delivery at new sites.

8.2.4 Item and Exam Development. No more than 12 months after contract award the Vendor must provide a web-based automated customizable workflow for test development, item development, boarding, editing, and validation that allows test items to be categorized into stages. The workflow must include:

1. A check-in and check-out process, with ability to view but not edit items that are checked-out by someone else.

2. Word processor tools that include spell check.

3. The ability to track all changes and versions of tests and questions, including details of all changes made, date stamp, and the user who made each change.

4. Automatic question numbering.

5. The ability to query, find, and replace.

6. The ability to display graphics (charts, pictures, drawings, graphs) in questions.

7. The ability to capture and display relevant test and item information, e.g.

Angoff rating, usage, difficulty, statistics, reviewer comments, and edits.

8. The ability to create, assign, search, archive and retrieve item indexes and metatags, such as: enemies, conflicting questions, topics, content, specifics, series, LSC and ACS codes, references, regulations and status for individual questions using established codes.

9. The ability to create and update reference tables (lists) in the database.

10. The ability to automatically generate items based on defined criteria, e.g. aircraft performance/limits data, such as weight and balance.

11. An agreed upon validation process that:

a. Limits selection of validation questions to be use on each test so as not to cause unnecessary worry with an overabundance of questions on any one test (currently 5 items per form).

b. Ensures responses to validation questions do not count against the applicant’s final test score.

12. Selection and weighting of items so tests generated for the test type have similar difficulty levels and item exposure is limited.

13. The ability to create a custom test by selecting a custom set of test items that may not be in accordance with a blueprint (e.g., a custom test built to test a specific area or a few specific areas, usually only two or three topics from a blueprint).

14. Within 30 days of the FAA or Foreign CAA and RSOO assigning the Subject Matter Expert (SME) item or exam development duties the Vendor will provide instruction or guidance on use of the tool provided to meet this requirement, and training on effective item writing.

8.2.5 Additional Test Banks – In this phase, and every option year that follows, at the COR’s request the Vendor is required to provide AFS-600 and FAA-approved Foreign CAAs and RSOOs the ability to create:

1. New master test banks with associated test banks and test types (approximately 12 associated banks, 80 test types, and 11,000 questions) as needed, and

2. Test types for existing master banks as needed.

8.2.6 Capability to increase sample tests, types of sample tests, and sample test items.

8.2.7 Role based information sharing. The Vendor must provide an electronic customizable dashboard for each FAA role. For example, the FAA test bank manager role dashboard would include (but not be limited to):

1. Status of assigned areas.

2. Open boarding items.

3. Assigned responsibilities that are performing outside set standards.

8.3 Phase Three: Improve test bank items and exam blueprints.

8.3.1 Assessments - Assess each test type and its associated supplemental material in the domestic test banks in accordance with current Standards for Educational and Psychological Testing, or other legally defensible test development standard. Most test types have never been assessed. In the last 2 years we have assessed three test types and are currently doing assessments on four others (Learning Statement Reference Guide is available at: https://www.faa.gov/training_testing/testing/). Assessments must use FAA provided SMEs when content experts are needed and must:

1. Determine the appropriate test session length for each test type.

2. Include a viability report on the cut score compared to the level of certification for the test type being assessed and if the cut score does not meet the standard for that test type, provide recommendations for:

a. Standard setting to conform to the passing score and

b. A change to the pass/fail cut score.

The FAA will determine the corrective action implemented to meet the standard.

https://www.faa.gov/training_testing/testing/

3. Assess test blueprints in conjunction with the task analysis required in 4.

below, determine the appropriate cognitive level for the test and make recommendations to improve the blueprint. All test types have a blueprint but they are very basic consisting of the total number of questions for that test type, required subject areas and the number of questions required in each area.

4. Using FAA provided SMEs, conduct a task analysis for each test type to determine the capabilities and necessary knowledge for airman applicants, which will serve as a key component of test blueprint development. The FAA will provide SMEs from industry and FAA.

These SMEs must be utilized to ensure an accurate rendering of tasks and performance objectives. The development of an ACS is the FAA’s only task analysis effort. Three ACSs have been completed (Private, instrument and UAS) and four (Commercial, Airline Transport Pilot, Airframe Mechanic and Powerplant Mechanic) are underway. The task analysis required here will validate the ACSs completed and provide the first analysis for the remainder of the test types. It will take several years but eventually the FAA will develop an ACS for each test type.

The FAA reserves the right to require the Vendor to conduct a subsequent task analysis at a later time if the FAA deems it necessary to do so.

5. Provide an academically sound blueprint for each test type for FAA approval.

6. Determine whether random test generation or form test is the appropriate delivery mechanism for each test type.

a. For test types that use form tests, determine the number of form tests appropriate for the test type, the number of items required for that form test blueprint to accurately measure knowledge, and the number of items needed to support the number of form tests needed.

b. For test types that will use random test generation, determine the number of items needed to support random test generation. This will be accomplished in the order specified in Section 8.3.2 of this

SOW.

7. In conjunction with the FAA, establish a process for determining which tests should be reviewed to ensure test accuracy and currency.

8.3.2 Initiate the assessments required in phase 8.3.1. Beginning in Base Year 2 of this contract the FAA is requiring two assessments be accomplished each year until all exams have been assessed. The sequence can be adjusted in writing by the COR but the initial plan is that they be accomplished in the following order:

1. PVT - Private Pilot

2. AMG – Aviation Mechanic General

3. IRA – Instrument

4. AMP – Aviation Mechanic Powerplant

5. COM - Commercial

6. AMA – Aviation Mechanic Airframe

7. CFI - Flight and Ground Instructor and Pilot Examiner

8. IAR – Inspection Authorization

9. ATP – Airline Transport Pilot

10. RIG – Parachute Rigger

11. LSP – Light Sport Pilot

12. FEX – Flight Engineer

13. UAS – Unmanned Aircraft Systems

8.4 Phase Four: Individualized testing. The Vendor must provide:

8.4.1 Automated on-demand, unique test generation (such as LOFT) in Option Year 2 for at least one test type identified for random test generation in item

8.3.1 above and one each following year in the order they were assessed until random test generation is completed for the identified exams. Every test generated must conform to the blueprint for that test type.

8.4.2 The ability to incorporate advanced testing methods, such as video-based, drag-and-drop, flash animation, situational judgment, etc. into items no later than Option Year 5.

QUALIFICATION REQUIREMENTS

9.1 The following are the minimum qualification requirements. Proposals that fail to meet these requirements will be rejected. The Vendor must:

9.1.1 Have delivered a minimum 100,000 web-based exams to applicants worldwide each year of the last 5 years.

9.1.2 Have developed items, managed item banks, developed exam blueprints, assessed test items and assessed test performance each year of the last 5 years.

9.1.3 At the time of submission of the SIR capability statement hold, and must maintain throughout the life of the contract, a membership in at least one of the following associations:

1. Association of Test Publishers (ATP)

2. American Educational Research Association (AERA)

3. National Council on Measurement in Education (NCME)

4. Society for Industrial and Organizational Psychology (SIOP)

5. Council on Licensure, Enforcement and Regulation (CLEAR)

6. Institute for Credentialing Excellence (ICE)

SECURITY

10.1 The term “information security” means protecting information and information systems from unauthorized access, use, disclosure, disruption, modification, or destruction in order to provide confidentially, integrity, and availability. The Vendor shall provide the minimum security controls required to protect Federal information and information systems in accordance with:

1. The Federal Information Security Management Act (FISMA) (as amended)

2. Title III of the E-Government Act of 2002 (as amended)

3. P.L. 107-347 (as amended)

4. NIST 800-53 Special Publication (as amended)

5. OMB Circular A-130, Management of Federal Information Resources (as amended)

6. The Privacy Act of 1974 (as amended)

7. The Department of Transportation Compendium (as amended)

The FAA is purchasing a service from the Vendor. If the Vendor elects to utilize a cloud service provider (CSP) the Vendor must comply with FedRAMP and notify the FAA of its intent.

The Vendor shall demonstrate ability to comply with these requirements prior to contract award.

10.2 As part of the Vendor’s system to protect sensitive information, the Vendor must:

1. Have a comprehensive system security plan that ensures compliance with the MOU for System Interface. (See Section 12.0 of this SOW) prior to data transfer.

2. Utilize configurable permissions and user-defined roles.

10.3 Test Security – Prior to initiation of test delivery and for the duration of the contract the Vendor must:

1. Ensure no unauthorized person has access to the test, test procedures, test results and reports, or the test software code.

2. Ensure applicants are properly checked into and out of the test room.

3. Ensure test security is not compromised. If there is an incident of actual or suspected misconduct, including cheating, by an applicant or Vendor employee, the Vendor must:

a. Immediately notify the geographically responsible Flight Standards District Office (FSDO), which can be identified at http://www.faa.gov/about/office_org/field_offices/fsdo/ and fully support the FAA’s investigation of the incident.

b. Provide the FSDO all the information and evidence related to the misconduct incident.

c. Prevent the applicant from any further testing until FAA concludes its investigation and notifies the Vendor to release the hold.

d. Notify the COR, or person designated in writing by the COR, in accordance with Section 16.1 of this SOW.

4. When test validity is challenged, provide witnesses, copies of records, and copies of reports regarding the test or the Vendor's services in any legal challenges to the test. To summarize, the Vendor must provide witnesses for legal proceedings and all information required by the FAA to answer inquiries about the test or any particular administration of the challenged test.

5. Provide witness testimony in any legal challenges that arise from allegations involving suspected cheating or other incidents that occurred during a testing event.

6. Ensure no unauthorized copies of the test or test content are made.

7. Ensure test access and test scores are strictly controlled at all times and tests are only used for authorized purposes.

8. Ensure no person takes photographs or makes copies of the computer screens displayed by the test. All personal belongings, including cell phones, must be checked in with the test administrator prior to the beginning of a testing session.

9. Ensure access to the FAA testing material is restricted only to individuals authorized by the FAA and Vendor employees who have a "need to know." The Vendor must receive affirmative authorization from the COR to release test material or information to any person.

10. Show that any Vendor or employee of a Vendor associated with the present contract has been notified that they shall not offer test preparation courses or participate in the sale of test preparation or study materials for airman knowledge exams for at least 1 year after the termination of this contract or their employment.

http://www.faa.gov/about/office_org/field_offices/fsdo/

CIVIL AVIATION AUTHROITIES (CAAs) AND REGIONAL SAFETY OVERSIGHT ORGANIZATIONS (RSOOs)

Currently, the FAA delivers tests to Suriname and, over the next 5 years, anticipates delivering tests to additional foreign CAAs and RSOOs (see Appendix 5 for more detail).

No funds collected by the Vendor for delivering U.S. domestic tests will be used to support item development, test development, test delivery, maintenance of test banks, or any other service that supports foreign CAA and RSOO testing activity. When notified by the COR the Vendor will “seed” foreign CAA and RSOO test banks with specified universal questions (such as performance or weather) from the US test banks at no cost to the CAA or RSOO, otherwise all Foreign CAA and RSOO costs will be funded apart from this contract. This contract merely requires that the services provided to the FAA under the terms of this contract be made available to foreign CAAs and RSOOs approved in writing by the FAA when arrangements for those services and associated costs have been negotiated through the FAA.

The sole exception to the “No funds” stipulation above is that these funds may be used to support test delivery, but no other service, to Suriname for up to one year after the Vendor begins delivering tests in compliance with this SOW. At the end of this period the FAA expects that a new international agreement will be in place that will allow this contract to support Suriname’s requirements.

SYSTEM INTERFACE

The Vendor’s system must interface with the FAA certification support system (currently IACRA) that is managed by the Civil Aviation Registry Division (AFS-700). The Vendor must execute the Memorandum of Understanding for System Interface in Appendix 7 with the FAA as a condition of establishing this interface. The FAA and the Vendor will electronically share data in accordance with the MOU in Appendix 7 through this interface.

QUALITY CONTROL PLAN (QCP)

The Vendor shall develop, document, and implement an internal Quality Control Plan (QCP) for work performed under this contract. The Vendor shall provide a written QCP describing the inspection system for the requested services listed in this SOW. The QCP shall include procedures to prevent, identify, correct, and ensure non-recurrence of defective services. The written QCP is required when proposals are submitted. The CO will notify the Vendor of acceptance or required modifications to the QCP. The Vendor shall make appropriate modifications and obtain acceptance of the QCP by the CO within 45 days after contract award. The QCP shall be updated as changes occur then submitted for review and acceptance by the CO.

CONTINUITY OF OPERATIONS PLAN (COOP)

Prior to data migration the Vendor must have a Continuity of Operations Plan (COOP) in place that includes at a minimum:

1. How uninterrupted service levels will be maintained during normal operations at the primary data center facility with recovery point of no more than 15 minutes.

2. Disaster recovery plans that meet a recovery point objective of four (4) hours or less of data loss and a recovery time objective of eight (8) hours or less of down time.

3. A disaster recovery facility at least 30 miles from the primary operational facility (100 miles if the primary operational facility is located on the U.S. Gulf Coast or California earthquake zone).

4. Maintenance and storage of backups at an offsite secured facility at least 30 miles from the primary operational facility (100 miles if the primary operational facility is located on the U.S. Gulf Coast or California earthquake zone).

INSPECTION

15.1 The FAA will conduct both periodic and unscheduled inspections. During these inspections, the Vendor must accommodate the FAA at its home office and testing facilities, including when tests are being administered. The purpose of such inspections are, but not limited to ascertaining:

1. Effectiveness of Vendor’s corrective actions from its QCP findings.

2. Whether feedback collected in accordance with Section 8.2.3.2 f. of this SOW is being appropriately addressed.

3. Compliance with the MOU for System Interface.

4. Compliance with the standards and requirements in this SOW.

15.2 Within 30 days after an FAA site visit the Vendor must get the COR’s approval of its plan to address all FAA concerns identified during the site visit. For each concern the plan should include:

1. root cause analysis,

2. corrective action that corrects the concern,

3. corrective action that eliminates reoccurrence of the concern; and

4. a timeline for completing the plan.

Deviations to the plan, including timeline, must be approved by the COR.

REPORTS

The Vendor is required to provide FAA with standardized and ad-hoc reports, downloads, and/or extracts as needed. These reports must be in a manner acceptable to the FAA and either scheduled or on request.

The Vendor must provide the following reports to the FAA, as directed by the COR, in the format requested and on the schedule indicated below. Any deviation from this schedule must be approved in writing by the COR.

1. Immediately when an event described in 10.3.3 above occurs.

2. Weekly provide test data forensics that identify testing behavior that could indicate cheating, e.g. deviations from the norm, such as short test times, multiple takes with large increase in score, higher than normal scores with a particular proctor, etc.

3. Weekly report of anomalies and status of associated corrective action identified during the QCP inspection process in 13.0 above.

4. Bi-weekly progress reports until test delivery is initiated.

5. Monthly reports of complaint information collected per 8.2.3.2.r above.

6. Monthly PMSR in writing to the COR by the end of the fifth business day of the following month. At a minimum, the PMSR will include:

a. A statement of accomplishments (milestones achieved) during the month.

b. Actual program status versus the Program Management Plan in 5.1 above.

c. Proposed changes to the PMP.

d. Problems that could affect the PMP timeline or task/deliverables.

7. On request generate, export and save both standard and ad hoc reports on any data contained in the ACTS.

8. On request reconstruct any individual test previously administered in the exact question version administered including question number, question stem, response and whether the response was correct or incorrect, and the overall test score

9. Be able to provide, upon request, some or all of the following applicant and testing information:

a. Applicant’s signature.

b. Applicant’s unique ID number from the FAA certification support system (currently IACRA).

c. Location (site ID) where test was administered.

d. Physical address, including country, where test was administered.

e. Time test began and ended.

f. Test take number.

g. Proctor’s name.

h. Citizenship of applicant.

i. Copies of:

i. Exam authorization (see Appendix 4).

ii. Identification presented to take exam.

j. Name of 14 CFR part 141, 142, or 147 school of which the applicant is a graduate, if applicable. The FAA will provide a list of approved schools that the Vendor must use to authenticate certificates.

k. Date of graduation from school in j. above.

PROTECTION AGAINST MISCONDUCT

In performing the services described herein, the Vendor agrees to comply with, and assume responsibility for its employee’s compliance with, the following requirements:

1. All work required by this SOW shall be performed under the supervision of the Vendor or the Vendor's responsible employees.

2. Every Vendor employee involved in the administration of airman knowledge tests must sign a nondisclosure agreement whereby the employee agrees to maintain confidentiality of all airman knowledge test information both during and after employment with the Vendor.

3. Any airman testing information made available in any format shall be used only for the purpose of carrying out the provisions of this contract.

Airman testing information includes all items, test blueprints, test forms and test data.