RFP_HSTS01-15-R-HRM015_Vendor_Clarification_Questions_and_TSA_Responses.pdf

RFP HSTS01-15-R-HRM015 Vendor Clarification Questions and TSA Responses

1Pages 16

Question No.

RFP Document or Attachment Reference Offeror Question TSA Response

How many assessment items does TSA have in its item bank or database that you would like the contractor to include or have access to when developing and validating future assessments? TSA currently has approximately 3500 items, however, TSA anticipates with the award of the TAP contract this number will increase.

Given the large procurement requirements and proposal requirements -- would TSA grant a 3 week extension to the Due Date?

Per Amendment 00001 issued on July 15, 2015, page 129 paragraph 2 GENERAL INFORMATION of the solicitation is amended to extend the proposal submission due date to no later than 4:00 PM local Washington D.C. time on August 7, 2015.

3 SOW 2.3.2.1

To facilitate a distance analysis, please provide zip code data for the airports, Federal Air Marshal Service (FAMS) Field Offices and any other TSA facility that requires a test center within 60 miles of that location.

If zip code data is unavailable, please provide the specific city location.

Attachment 12 has been added to the RFP providing zip code data for airport locations. The list is not exhaustive and does not include FAMS locations or other TSA facilities, however, it will provide a general sense of where test centers are needed. The airport list is subject to change. Full location list will be provided to the awardee after contract award. Proposals shall be in accordance with SOW 2.3.2.1.

4 SOW 2.3.2.1 Will space at the required TSA locations be available for the contractor to provide for testing of candidates?

Currently, we do not anticipate contractor proctors to administer tests at TSA facilities. TSA proctors will be used for tests administered at a TSA facility. In the event that contractor proctors are needed at TSA facilities, TSA will work with the contractor to identify available space.

5 SOW 2.3.2.3.

For assessments requiring the proctored setting, please provide test volume by month by site for the past 12 months.

The number of assessments fluctuates yearly based on hiring needs. TSA anticipates TSO candidate testing will be the highest volume of contractor-proctored testing. Historical data for TSO CBT was provided in the chart on page 5 and 6 of the solicitation.

6 SOW 6

Please clarify what Contractor personnel are subject to TSA’s suitability check and the specific requirements of the suitability check.

See page 90 paragraph 21 - 5200.209.004 Suitability Determination for Contractor Employees (Aug 2013) of the RFP . All contractor employees seeking to provide services to TSA under a TSA contract …

Solicitation PDF, Section II, Topic 9, Deliverable #55, Page

Deliverable #55 indicates that a Quality Control Plan (QCP) is due with the proposal response. However, the RFP proposal submission instructions on pages 131-1335 of the same document do not identify where it is to be included.

The Draft QCP is due ten (10) calendar days after Post Award Meeting. A Final QCP is due within ten (10) business days after receipt of comments from the Government.

Solicitation PDF Section II, Topic 1.2, Page 6

Can TSA provide information about the current process flow for the TSO certification testing program?

Offeror would like to understand the steps the individual goes through to complete the certification program (invitation, testing, results, etc.) and how that might be different from the other programs identified.

Currently, TSOs are assigned tests in the Online Learning Center (OLC) (TSAs Learning Management System). TSOs launch a test via the OLC when proctored by a TSA Test Administrator. The test is completed in TSAs current online testing system, and the results are passed into the OLC. TSA Test Administrators pull feedback reports from TSAs current online testing system. If an online testing environment is not available, TSOs are provided the test via pdf and answers are recorded via a scantron. The results are manually entered into the OLC. The scantrons are shipped to TSAs current testing system vendor for scanning and archiving.

Attachment 1, Contract Pricing File

1.5.A Labor Cats and Rates - For the mock scenario, should the vendor assume that the newly developed biodata assessment has not already been configured for online delivery on the offeror's platform? Yes, vendor can assume the assessment has not been configured for online delivery

Attachment 1, Contract Pricing File

1.5.A Labor Cats and Rates - Should the offeror assume the same mock scenario should be used providing labor rates and hours in Years 2-5 and Award Years 1-3? Or conversely, should the offeror provide rates and hours in those additional years that reflect the maintenance of the biodata assessment? Offerors shall use the same mock scenario when providing labor rates and hours in Years 2-5 and Award Years 1-3.

Attachment 1, Contract Pricing File

Can TSA clarify that the prices associated to Task Order 001 only need to be provided in the Excel Attachment 1, 1.5.C tab, cells C4-C19? Or, does the pricing table provided in Attachment 3 need to be provided in Volume II, Price/Business Proposal?

Both. As stated in Attachment 3, Offerors shall insert Unit Prices and Total Prices consistent with pricing included on Attachment 1 “CATS Contract Pricing”.

Solicitation PDF Section IV, Topic 4, Page 132

Can TSA clarify if the Offeror's Volume I, Factor 1, Section 1 response should be organized according the SOW provided in the Solicitation document or according to Task Order 001 (Attachment 3)?

The offeror shall describe and demonstrate the technical expertise, technical understanding, and detailed approach to accomplish all requirements specified in the SOW and Task Order 1. Organization of the proposal is at the discretion of the offeror.

2Pages 16

Attachment 1, Contract Pricing File

I.5.C CLIN Pricing Schedule - In Attachment 9, how does the evaluation period align with the Award Term(s)? In other words, if a contractor is deemed to have performed above acceptable for years 2 and 3 would they be awarded Award Term 1; Year 4, awarded Award Term 2; and, Year 5, awarded Award Term 3? Is that a correct understanding - if so, do the award terms go into effect after Year 5?

The understanding is correct; however, TSA is not obligated to issue orders beyond the minimum guarantee so it is possible that an award term earned could be exercised prior to year 6.

Attachment 1, Contract Pricing File

I.5.C CLIN Pricing Schedule, Row 16 - Seat Time (Administer assessment at a proctored test center per hour) – Is it our understanding that the government would like to see one price for an hour of testing, and will assume that hourly price extends out to multi-hour seat time? E.g., if one hour is $10 then 5 hours is $50. Yes, that understanding is correct.

Attachment 1, Contract Pricing File

VI.6 Sample Catalog Pricing tab - 3. If a vendor has an assessment listed on the VI.6 Sample Catalog Pricing tab, but also has an equivalent assessment, which one would the government prefer to see and priced?

Follow-on, is it preferred by the government to propose the listed assessment – meaning, will the vendor receive a higher rating for having access to the listed assessment?

Attachment 1 has been amended to delete the column for equivalent assessments for price analysis purposes. Offerors shall only price the listed assessments in its possession and shall not leave out pricing for any of the listed assessments in its possession. Only the assessments common across proposals will be considered for price comparison purposes.

Attachment 1, Contract Pricing File

I.5.C CLIN Pricing Schedule, Row 6 - Certification and Accreditation has one pricing cell. However, SOW Section 9 (Deliverable #39) states that it will need to be recertified/maintained every 2 years. Where should the Offereor include these recertification/maintenance costs?

Recertification/maintenance will not be separately priced. Recertification/maintenance costs are considered part of system O&M.

Attachment 1, Contract Pricing File

I.5.C CLIN Pricing Schedule, Row 18 - Hosting TSA provided assessments (TSA developed or 3rd party custom developed) – this line item has volumes associated with it. Does this just representing hosting these assessments, or does the government see delivery also being part of the line item?

TSA expects all TSA-developed assessments hosted on the TAP system to be available for administration through the TAP system.

Attachment 1, Contract Pricing File

In Section 1.2, page 7 of the Solicitation PDF it is stated that TSA expects 125,000 certification tests annually for the current TSO workforce. However, it is unclear where in the 1.5.C CLIN Pricing Scheule the cost for delivering those tests should be notated. Please provide guidance.

Please reference Row 18 of 1.5.C CLIN Pricing Scheule, "Hosting TSA Provided Assessments".

Attachment 1, Contract Pricing File

In Attachment 1, on the VI.6 Sample Catalog Pricing tab, there are 3 sets of sample assessments. However, rows 11, 20, 27 show a total of each grouping implying that each group is weighted equally. We would like to suggest that the stated rows reflect an average of each group, with row 29 representing an average of the group averages.

The average will be calculated by adding up the total proposed prices for all the assessments common across proposals and dividing that total by the number of assessments common across proposals. Attachment 1 has been amended to delete the formulas for calculating the averages. Also, see response to question #15.

Attachment 1, Contract Pricing File

In Attachment 1, on the VI.6 tab, how are the ‘equivalent assessments’ being evaluated? Are they evaluated against the named assessment or the TSA Core Competencies? Further, is there a weighting system in place once the equivalent assessment are evaluated?

See responses to #15 and #19

Solicitation PDF Section II, Topic 2.1.1.1, Page 8

The requirements in 2.1.1.1 state that an exception to this subparagraph will be considered on a case-by-case basis if the offeror has a testing system currently with TSA. Can TSA clarify the specific requirements from Topic 7 that this exception would override?

No, not at this time

Solicitation PDF Section II, Topic 2.2.3, Page 11

Since the TSA already has TSA-owned content that is being utilized, can they share how many total items are currently in the item bank(s)? Additionally, does the TSA have an estimated number of items it would expect to create annually?

See response to question 1. Typically there are approximately 750 items developed annually, depending upon program requirements and is always subject to change.

3Pages 16

Solicitation PDF Section II, Topic 2.6, Page 22

When TSA identifies tests that need to be developed, will the offeror conduct the job analysis work or does TSA have staff to do that?

TSA regularly conducts job analyses and can supply this data as necessary. In the event that additional data collection is necessary TSA will work with the vendor to determine how this will be conducted.

Solicitation PDF Section II, Topic 1.2, Page 7

In this section TSA references ~25,000 "off-line" assessments annually. Can TSA explain the scenario in which an off-line test is used as well as the process used? The offeror is interested to know the types of tests that are delivered off-line, the environment in which they are delivered in, how they are scored, how scores are communicated to testers, etc.

Off-line tests are typically used when system or networking issues prevent TSA test administrators from administering a test online at a TSA location. Currently offline testing is only allowed for initial certification tests after scheduled trainings. Offline tests are pdf'ed and administered via a non-networked TSA computer. Scantrons are used, the results are manually scored, and a raw score is entered into the system. The TSA Test Administrator communicates test results to test takers. The scantrons are then sent to the vendor for scanning and archiving purposes. TSA is open to other methods for administering a test in an environment without access to the internet.

Solicitation PDF Section II, Topic 2.10.2.2, Page 30

This requirement states that the Contractor shall interface with the TSA Customer Service Center to automatically feed all CATS-related inquiries to the CATS help desk. Can TSA share what application/system is uses for the TSA Customer Service Center and if it has a standard API to be used for integration? Has it been integrated into other systems already? The TSA CSC contract has not been awarded. This system will be used to track all calls related toTAP even though the TAP vendor may be responding to all inquiries.

Solicitation PDF Section II, Topic 2.10.2.8, Page 31

This requirement states that the Contractor shall use the Customer Service Center case tracking system by TSA. Can TSA share the system it currently uses and, if necessary, will TSA be responsible for securing (and paying for) any needed licenses to such system?

See response to #25. This system will be used to track all calls related to TAP even though theTAP vendor may be responding to all inquiries. TSA is responsible for securing and paying for all licenses, as such, the ownership of such stays with TSA.

Solicitation PDF, Section II, Topic 2.3.1.9, Page 13

Topic 2.3.1.9 requires that "Requested changes shall be completed within five (5) business days of the request." Are the system/content changes mentioned in this topic subject to the Configuration Management requirements mentioned in the Solicitation PDF, Section II, Topic 7.3 , including the change control oversight required by Topic 7.3.2 ? And, if so, will the time required to obtain such approval be counted against the turnaround time required under this topic or excluded from the calculation?

No

Solicitation PDF, Section II, Topic 2.3.4, Page 20

Topic 2.3.4 states that "Contractor shall upload, ensure accuracy of content and ensure content is viewable and displayed correctly at the request of TSA and within the timeframes specified below:

2.3.4.1.1 Contractor provided test content – 10 business days.

2.3.4.1.2 TSA provided test content – 10 business days.

2.3.4.1.3 Third-party provided test content – 30 calendar days."

Are the system/content changes mentioned in this topic subject to the Configuration Management requirements mentioned in the Solicitation PDF, Section II, Topic 7.3, including the change control oversight required by Topic 7.3.2? And, if so, will the time required to obtain such approval be counted against the turnaround time required under this topic or excluded from the calculation?

No

4Pages 16

Solicitation PDF, Section II, Topic 2.10.1.12, Page 28

Topic 2.10.1.12 requires that "Make minor edits such as name changes, typos missed in the review process, or other errors in the automated systems within 5 business days of the notification by the Government. Any errors found in the production version of the automated assessments shall be corrected within 48 hours of notification or detection by either TSA or the Contractor." Are the system/content changes mentioned in this topic subject to the Configuration Management requirements mentioned in the Solicitation PDF, Section II, Topic 7.3, including the change control oversight required by Topic 7.3.2? And, if so, will the time required to obtain such approval be counted against the turnaround time required under this topic excluded from the calculation?

No

Solicitation PDF, Section II, Topic 2.9.2, Page 26 Solicitation PDF, Section II, Topic 9, Deliverable #50, Page

Topic 2.9.2 requires full operational capability within six (6) months of contract award, however Topic 9 Deliverable 50 indicates that Integration/Functional testing is due eleven (11) month from the start of task order POP. Can TSA explain the discrepancy? Section 2.9.2 references the TAP system completes Certification and Accreditation (C&A) and achieves Authorization to Operate (ATO).

Deliverable 50 references section 2.11 which outlinesTAP integration with the Learning Management System and TSAs Hiring System.

Solicitation PDF, Section II, Topic 2.10.1.11, Page 31

Topic 2.10.1.11 requires the offeror's system to "Comply with TSA’s IT enterprise architecture standards and processes, and submission of all required documentation." Can TSA furnish a copy of such "standards and processes" for the offers to ensure that their systems do comply?

Yes; Page 110 SECTION IV - ATTACHMENTS adds attachment 10 DHS and TSA Enterprise Architecture Compliance to the RFP

Solicitation PDF, Section II, Topic 2.10.2.8, Page 31

Topic 2.1.2.8 indicates that "The contractor shall use the Customer Service Center case tracking system as directed by TSA." Should the offers understand this to mean that TSA will be providing the case management system and the offerors will not be using their own tracking system?

See response to question 26 and 27.

Solicitation PDF, Section II, Topic 2.11.1, Page 31 Solicitation PDF, Section II, Topic 2.11.2, Page 32

Topic 2.11.1 requires integration into "...all TSA's hiring system (Government furnished and third-party), including the Candidate Dashboard and Airport Assessor systems" and Topic 2.11.2 requires the system "...to seamlessly integrate with TSA’s LMS." Will TSA be providing project management oversight for the non-CATS portions of these integrations to ensure target dates are not missed that are outside of the offeror's control?

The vendor will not be held accountable for any schedule slippage that is not within their control and due to external factors.

Solicitation PDF, Section II, Topic 7.2.2, Page 41

Topic 7.2.2 requires "All Contractor employees shall receive initial TSA IT Security Awareness Training within 60 days of assignment to the contract." Will such training be provided by TSA or will this be developed by the Contractor? Training will be provided by TSA.

Solicitation PDF, Section II, Topic 7.2.2, Page 41 Will this requirement extend to third-party assessment provider employees? Yes

Solicitation PDF, Section II, Topic 7.2.4, Page 41

Topic 7.2.4 requires "Role-Based training for contract employees individuals with Significant Security Responsibility (SSR), whose job proficiency is required for overall network security within TSA, will be in accordance with DHS and TSA policy." Will such training be provided by TSA or will this be developed by the Contractor?

Training will be provided by TSA.

Solicitation PDF, Section II, Topic 7.2.4, Page 41 Will this requirement extend to third-party assessment provider employees? Yes

Solicitation PDF, Section II, Topic 7.3.1, Page 42

Topic 7.3.1 mentions the DHS Information Security Performance Plan (current year and any updates thereafter), the DHS Continuous Diagnostics and Mitigation (CDM) Program and TSA’s Configuration Management policy. Can TSA provide copies of these plans to the offerors so that the offerors can better understand these requirements? Information will be provided in a "Reading Room" that will be part of the OIT Industry Day.

Solicitation PDF, Section II, Topic 7.3.1, Page 42 Will this requirement extend to third-party assessment provider's systems? Yes

5Pages 16

Solicitation PDF, Section II, Topic 7.3.1, Page 42

Topic 7.3.1 indicates "The TSA Chief Information Security Officer (CISO)/ Information Assurance and Cyber Security Division (IAD) must be informed of and involved in all configuration changes to the TSA IT environment including systems, software, infrastructure architecture, infrastructure assets, and end user assets. The TSA IAD will approve any request for change prior to any development activity occurring for that change and will define the security requirements for the requested change." Can TSA provide any metrics on how long such approvals typically take? Typically 3rd party hosted systems have their own Change Control Board (CCB) which includes the TSA System Owner (SO) and

Information System Security Officer (ISSO). The TSA SO will be the Chairman of the CCB. Metrics may vary by maintenance schedule.

Solicitation PDF, Section II, Topic 7.3.1, Page 42

Topic 7.3.1 indicates "The TSA Chief Information Security Officer (CISO)/ Information Assurance and Cyber Security Division (IAD) must be informed of and involved in all configuration changes to the TSA IT environment including systems, software, infrastructure architecture, infrastructure assets, and end user assets. The TSA IAD will approve any request for change prior to any development activity occurring for that change and will define the security requirements for the requested change." Does this oversight apply to systems that are not hosted in a TSA data center?

See question 40 above.

Solicitation PDF, Section II, Topic 7.3.2, Page 42

Topic 7.3.2 indicates "Approval for immediate intervention (emergency change) requires approval of the TSA CISO, SCCB co-chairs, and the appropriate Operations Manager, at a minimum." Can TSA provide any metrics on how long such approvals typically take?

See question 40 above.

Solicitation PDF, Section II, Topic 7.3.2, Page 42

Topic 7.3.2 indicates "Approval for immediate intervention (emergency change) requires approval of the TSA CISO, SCCB co-chairs, and the appropriate Operations Manager, at a minimum." Does this oversight apply to systems that are not hosted in a TSA data center?

See question 40 above.

Solicitation PDF, Section II, Topic 7.3.2, Page 42

Topic 7.3.2 indicates "Approval for immediate intervention (emergency change) requires approval of the TSA CISO, SCCB co-chairs, and the appropriate Operations Manager, at a minimum." Does this oversight apply to systems that are not hosted in a TSA data center?

See question 40 above.

Solicitation PDF, Section II, Topic 7.3.2, Page 42 Will this requirement extend to third-party assessment provider's systems? Yes

Solicitation PDF, Section II, Topic 7.4.1, Page 43 Solicitation PDF, Section II, Topic 9, Deliverable #39, Page

Topic 7.4.1 indicates that CATS will utilize the NIST SP 800-37 and SP 800-137 Risk Management Frameworks for Security Authorization and Ongoing Authorization Process. However, Topic 9 Deliverable #39 indicates that the C&A process will take place every two (2) years in the Deliverables column and every two (2) to three (3) years in the Due Date column. Our understanding of the Ongoing Authorization process is that it eliminates the need for the recertification at the two to three year interval as the Contractor is continuously providing documentation and other artifacts that effectively extends the ATO throughout the life of the contract. Can TSA elaborate which process will govern CATS?

Any new systems will require an ATO prior to processing TSA data. An Authorized system can enter the Ongoing Authorization program but it is not guaranteed. Which process will govern TAP is to be determined.

Solicitation PDF, Section II, Topic 7.4.5, Page 44

Topic 7.4.5 mentions the "DHS Ongoing Authorization Methodology (current version) ". Can TSA provide the DHS Ongoing Authorization Methodology (current version) guide so that offerors can better understand the deliverable under this requirement? Information will be provided in a "Reading Room" that will be part of the OIT Industry Day.

6Pages 16

Solicitation PDF, Section II, Topic 7.7, Page 46

Is Topic 7.7 applicable for systems hosted on Public Cloud Service Providers (hosted on services like Microsoft Azure or Amazon Web Services) and Private Cloud Service Providers (hosted at a co-location facility on contractor owned hardware) or just for systems hosted on Public Cloud Service Providers?

It is expected any Cloud solution will be FedRAMP certified.

Solicitation PDF, Section II, Topic 7.7.1, Page 47

Can TSA elaborate on what is meant by "a commercial Trusted Internet Connection"?

https://www.fedramp.gov/draft-fedramp-tic-overlay/ http://www.dhs.gov/trusted-internet-connections

Solicitation PDF, Section II, Topic 7.7.1, Page 47

Can TSA elaborate on what is involved in deploying the EINSTEIN 3 Accelerated (E3A) capability? http://www.dhs.gov/publication/einstein-3-accelerated

Solicitation PDF, Section II, Topic 7.7.2, Page 47

Topic 7.7.2 indicates that "Security event logs and application logs shall be sent to the TSA SOC." Can TSA elaborate on how this process works?

Contractor requires firewall rules and routing to TSA SOC as well as bandwidth to send event logs from an installed ArcSight Agent or Logging device to the SOC.

Solicitation PDF, Section II, Topic 7.7.2, Page 47

Topic 7.7.2 mentions that "Security artifacts shall be created and maintained in the DHS Information Assurance Compliance Tool (IACS)." Will the Contractor be responsible for maintaining this information in the tool and be provided access, or will maintenance of these artifacts in the IASC tool be the responsibility of a TSA representative?

The ISSO does the maintenance in IACS if the contractor has no access.

Solicitation PDF, Section II, Topic 7.7.2, Page 47

Topic 7.7.2 indicates that "Security event logs and application logs shall be sent to the TSA SOC." Can TSA elaborate on if any government furnished equipment (GFE) will be required? And if so, can TSA provide any information regarding the quantity of equipment and overall physical size?

GFE will not be provided. Any equipment related to Sec II, Topic 7.7.2 must comply with the Security Authorization Process and Risk Management Framework (i.e., FIMSA and NIST requirements) and be provided by the vendor.

Solicitation PDF, Section II, Topic 7.8.2.7, Page 48

Can TSA provide the TSA Technical Standards to the offerors so that they can verify their solution is compliant with the standards. Information will be provided in a "Reading Room" that will be part of the OIT Industry Day.

Solicitation PDF, Section II, Topic 7.8.2.8, Page 48

Can TSA provide the DHS IT Security Architecture Guidance Volumes 1, 2 and 3 to the offerors so that they can verify their solution is compliant with the standards.

Information will be provided in a "Reading Room" that will be part of the OIT Industry Day.

Solicitation PDF, Section II, Topic 7.8.2.11, Page 48

Can TSA provide the DHS Ongoing Authorization Methodology (current version) to the offerors so that they can verify their solution is compliant with the standards.

See response to question 47.

Solicitation PDF, Section II, Topic 7.9.1, Page 48 Will this requirement extend to third-party assessment provider's systems? Yes

Solicitation PDF, Section III, Topic 21, Page 91

Phase 2 of the 5200.209.004 Suitability Determination for Contractor Employees (Aug 2013) FAR Clause indicates "Those contractor employees who do not have a previous federal background investigation sufficient in scope will be scheduled for the appropriate level background investigation through the submission of their security forms to the Office of Personnel Management (OPM)." Can TSA elaborate on what materials will need to be supplied by each contractor employee to OPM as part of the background investigation process?

The TSA Office of Security, PERSEC, obtains the e-QIP submission and release forms from all contractor applicants during the pre-appointment vetting phase. The e-QIP submission and release forms are then forwarded to OPM to schedule the required investigation after the contractor applicant begins working on the contract upon a favorable pre-appointment determination.

Solicitation PDF, Section III, Topic 21, Page 91

Phase 2 of the 5200.209.004 Suitability Determination for Contractor Employees (Aug 2013) FAR Clause indicates "Those contractor employees who do not have a previous federal background investigation sufficient in scope will be scheduled for the appropriate level background investigation through the submission of their security forms to the Office of Personnel Management (OPM)." Can TSA provide any metrics on how long this process takes?

Metrics are not available regarding the process. All contractor applicants are given 14 days to complete the form submission/e-QIP during the pre-appointment process. Upon receipt of the required submission, PERSEC will make an initial determination in five business days.

This initial determination can take longer on a case by case basis when issues arise during the vetting process which require contacting 3rd parties (previous employers; law enforcement entities; creditors, etc…) https://www.fedramp.gov/draft-fedramp-tic-overlay/ https://www.fedramp.gov/draft-fedramp-tic-overlay/ http://www.dhs.gov/publication/einstein-3-accelerated

7Pages 16

Solicitation PDF, Section II, Topic 2.3.3.2.1, Page 19

This requirement asks for the ability to print tests and test answers keys from the Contractor System. Can TSA elaborate on the rationale for this capability as the ability for system users to print this type of information is usually seen as a item/test security risk.

This requirement is currently needed for off-line initial certification testing purposes and is not used for all assessments.

Solicitation PDF, Section II, Topic 2.1.2.1, Page 8

This requirement discusses loading legacy assessment data into the CATS database. Can the TSA share the type of data that would be transferred. For example, sore reports, test item level responses, pass/fail scores, candidate demographics, etc. Some examples are test name, tester name, unique tester ID, location, raw score, percent correct, item labels, item level responses, test date, pass/fail.

Solicitation PDF, Section II, Topic 2.12.1.2.8, Page 34

This requirement discusses providing TSA with a list of CBT testing locations and their distance to the nearest federalized airport. Can the TSA clarify where in the Volume I/Volume II response requirements this information should be provided? Additionally, can the TSA please provide a list of the federalized airports and their corresponding city/state/zip?

Your proposal response is to be submitted in Volume I Tab A. Also, see response to question 4.

Solicitation PDF, Section II, Topic 2.9.1.1, Page 25

Is it the expectation the TSA will test with “test” data in a production environment as part of the “after initial production” UAT? If so, does the TSA require the offeror to delete testing data from a production database application after testing is complete? The offeror should be able to either delete or otherwise differentiate this data from other data collected so as not to confound aggregate reports.

Solicitation PDF, Section II, Topic 2.9.1.1, Page 25

Can TSA clarify the following statement and expectation "The TSA test lab will interface with the vendor's test environment over a secure internet connection". If this is a web application site using HTTPS protocol, will this suffice? Is it the expectation that TSA will have direct access to the database applications or to run automated testing scripts? Existing Test sites use Point to Point VPN tunnels and possibly IP security to secure the connection over which the HTTPS Web Users will connect.

Solicitation PDF, Section II, Topic 2.9.3, Page 26

Can the TSA clarify what the offline systems would consist of and if this only includes delivery of an assessment via paper and pencil? See response to question 24.

Solicitation PDF, Section II, Topic 2.9.3.4, Page 26

Can the TSA clarify if this item refers to simply the TSA UAT environment hosted by the contractor or if the expectation is the TSA host an environment at their location?

Yes. UAT will be hosted at vendor site.

Solicitation PDF, Section II, Topic 2.9.3.5, Page 26 Cam the TSA provide a number and/or types/providers of the "other systems"? See section 2.11.1 and 2.11.2 for current TSA systems. The systems may change as TSA acquires new or additional services.

Solicitation PDF, Section II, Topic 2.11.1, Page 31 Can the TSA provide a number and/or providers of "all TSA's hiring systems"? The systems may change as TSA acquires new or additional services.

Solicitation PDF, Section II, Topic 2.11.1, Page 31

Can the TSA share if there is currently a data warehouse that all TSA vendors feed data into? If so, who hosts the data warehouse and what is the frequency of transfer? Do all vendors have access and is there reporting available from the data warehouse?

Yes, TSA does have a Data Warehouse and it is currently hosted by the HRAccess Service Provider. The frequency of the data transfer or loads is dependent on the type of data and the vendor. We have nightly feeds and bi-weekly to monthly feeds. If access is required, the TSA HRAccess CO will apporve such requests

70 1.1. Background (p. 4). Please provide a list of the 450+ airports referenced in the first paragraph. See response to question 3.

71 1.2. Scope of Work (p. 6).

What tests are currently used to evaluate candidates and incumbents for the following roles?

• TSO

• SPP screeners

• FAMs

• MAP positions

• Executive positions

Currently for candidate TSO and SPP screeners, the Screener Object Recognition Test (SORT) and the Screener English Test (SET) are part of a multi-hurdle assessment process. FAM positions are administered a Situational Judgment Test (SJT), Logic Based Reasoning Test, Writing Skills Assessment, and/or Leadership Job Simulation (LJS). There is one MAP position with a computer based assessment for candidates and it is a Writing Skills Assessment. Currently there are no executive positions using computer based testing. For incumbents, job knowledge tests are used for certification processes. We are currently revising some of these assessments and expect any assessment to be subject to change pending the award of the TAP contract and evolving program requirements.

8Pages 16

72 1.2. Scope of Work (p. 6). How are the tests used?

Currently, tests can be used as part of multi-hurdle assessment processes. Other tests are used as job knowledge checks and/or pass/fail certification assessments.

73 1.2. Scope of Work (p. 6).

How much time is allowed to take an applicable test for each role? Please list (CBT) – Pre-hire Assessment testing time allowed and Post-Hire Assessment time allowed separately.

Currently applicants are given 2 hours for the TSO CBT which consists of Screener Object Recognition Test (SORT) and the Screener English Test (SET). The FAM (G Band) CBT totals 3 hours and consists of a Situational Judgment Test (SJT), Logic Based Reasoning Test, and Writing Skills Assessment, each of which are 1 hour long. The FAM (J Band) CBT totals 2 hours and consists of a Logic Based Reasoning Test and a Writing Skills Assessment, each of which are 1 hour long. The FAM (J Band) also has the Leadership Job Simulation (LJS) which totals 2 hours 15 minutes. MAP applicants are given 60 minutes for the Writing Skills Assessment. For incumbents, job knowledge certification tests can range from 30 minutes to 90 minutes. We are currently revising some of these assessments and expect any assessment to be subject to change or new positions/roles included pending the award of the CATS contract and evolving program requirements.

74 1.2. Scope of Work (p. 6).

How many candidates were tested for each role in the most recent calendar or fiscal year, whichever is applicable? Please list a number of candidates tested for (CBT) – Pre-hire Assessments and Post-Hire Assessments separately. See response to question 5.

75 1.2. Scope of Work (p. 7).

“Any internationally based proctored assessments shall be managed on a case-by-case basis.” Please provide an historical list of international site locations.

To our knowledge, TSA has not administered assessments in the past at international locations, however, wants to have that capability should the need arise.

2 SPECIFIC

REQUIREMENTS/TASKS

2.1 General System

Requirements

78 2.1.2.3 (p. 9).

“The Contractor shall not exceed four late responses per calendar year.” Please further explain what constitutes a late response. Per Section 2.1.2.3, Contractor has two business days to respond. A response that is not made within two business days is considered late.

79 2.2 Test Item and Structure

80 2.2.1.6 (p. 9).

For which testing programs does TSA currently use computer-adaptive testing (CAT)? If none, which programs are envisioned to use CAT?

Computer adaptive testing (CAT) is not currently used. Currently no specific program is envisioned to use CAT. TSA is in the process of revising its hiring and testing processes and this capability may be needed moving forward.

81 2.2.19 (p.11).

How many questions are typically in an exit survey? What question formats are used (e.g., multiple choice, open ended)?

TSA does not currently administer an exit survey at the end of assessments or test batteries but envisions having this capability in the future.

82 2.2.19 (p.11). What data management and/or reporting is required for the exit survey?

TSA does not currently administer an exit survey at the end of assessments or test batteries but envisions having this capability in the future and will determine the data management and reporting structure as needed.

83 2.2.3.1 (p. 11). What metadata are to be stored for items in the item bank for TSA-provided items? Please see SOW Section 2.2.3.1.5

2.3 Administrative

Requirements

85 2.3.2.1 (p. 12).

How often are tests for candidate evaluations expected to be administered (year around or during specific time periods (testing windows))? Please list applicable test schedules for (CBT) – Pre-hire Assessments and Post-Hire Assessments separately. Testing occurs year round. See response to question 5 for volume.

9Pages 16

86 2.3.2.1 (p. 13).

Please provide a list of each Federal Air Marshall Service (FAMS) Field Office or any other TSA facility that currently administers exams and provide annual volume per location.

See responses to question 3 and question5.

87 2.3.2.3 (p. 14).

“…have seating/testing capacity commensurate with TSA requirements for specific airports, field offices, and other TSA facilities.” Please explain further the required seating capacity.

See responses to question 3 and question 5.

88 2.3.2.4 (p. 14).

What is the desired ratio of proctors to candidates for computer-based online assessments and offline or alternative methods in physically proctored environments? Please list requirement for each test delivery method separately. TSA does not currently have a set ratio of proctors to candidates.

89 2.3.2.4 (p. 14).

“Provide TSA with the capability to administer computer-based online assessments and, if needed, offline or alternative administration methods , in physically proctored environments located on TSA properties, or on properties leased or rented by TSA for purposes of administering assessments to test takers.” Please define alternate administration methods.

See response to question 24 for TSAs current offline testing process. TSA is open to other methods for administering a test in an environment without access to the internet.

90 2.4 Reporting Requirements

91 2.4.1.4 (p.20). What statistics are required from the system for the psychometric analyses of items and tests? At a minimum, difficulty index, corrected item total correlation, reliability, raw score, and percent correct.

92 2.5 Assessment Library

93 2.5.1.1 (p. 22).

Can TSA provide the competencies that are to be measured for each of the following roles:

• TSO

• SPP screeners

• FAMs

• MAP positions

• Executive positions

The provided TSA competency catalog (Attachment 7 to the Solicitation) outlines all potential competencies that are to be evaluated for any given position. What specifically is measured will be determined by program needs and related job information.

94 2.5.1.4 (p. 22). Is there a desired or target test length for the candidate and employee exams?

The length of time will be dependent upon the test and purpose of the test. Currently, test batteries that TSA uses range from 30 minutes to 4 hours, although that is always subject to change based upon program requirements and selected tests.

95 2.6 Assessment Development

96 2.6 (p. 22).

How many forms of selection assessments for the major or critical TSA occupational series are published each year? If more than one, can the test forms overlap in test item content?

TSA develops selection assessments as needed based upon program needs and job requirements. Item content can overlap if determined to be appropriate on a case by case basis.

97 2.6.1 (p. 22). Which of the candidate and employee tests is TSA contemplating replacing with a custom assessment?

TSA is currently re-evaluating the TSO hiring process and may result in using a custom assessment(s). However, additional positions will likely be identified for custom assessments.

98 2.6.1 (p. 22).

Has a job analysis been previously completed for TSO, FAM, MAP, and other large TSA occupational groups? If so, when and are they available?

A job analysis was conducted for TSO and some of the MAP positions (LTSO, STSO, STI, CCO, and BDO) in 2012. A job analysis for MAP positions was conducted in 2008. A job analysis was conducted for FAMS was conducted in 2007. The job analysis materials may be provided to the vendor as appropriate after award.

99 2.6.1.6 (p. 23).

Is the TSA open to alternative validation strategies for instruments that would not traditionally be supported by content validation? Yes, as agreed upon when determining the validation strategy.

10Pages 16

100 2.7 Item Development

101 2.7 (p. 24). Will the contractor have any involvement in the assembly of certification and recertification examinations? Potentially, depending upon the user roles that are set up and the assistance that TSA needs.

102 2.7.1 (p. 24). How many items are currently available in the certification and recertification item bank? TSA currently has approximately 3500 items, however, TSA anticipates with the award of the TAP contract this number will increase.

103 2.7.1 (p. 24). What are the types and characteristics of the items? e.g., multiple-choice, single-choice, exhibits. The item types and characteristics that need to be supported by TAP is outlined in Section 2.2.1.

104 2.7.1 (p. 24). How many certification and recertification exams are published annually? Approximately 30 tests are finalized a year, however this number can fluctuate greatly based on program needs.

105 2.7.1 (p. 24). How often are certification and recertification exams updated?

Certification tests are updated as program needs change. Recertification tests are updated on a yearly basis, unless program changes require more frequent updates.

106 2.7.3 (p. 24). Will the TSA provide SMEs to review and approve edited test items? Yes

107 2.7.4 (p. 24). Will the contractor have any involvement in the development or updating of test specifications? Potentially, depending upon the need.

108 2.7.4 (p. 24). Can the TSA provide a copy of the test specifications and item writing guidelines?

Test specifications vary depending upon the test, but generally outline the number of items needed overall and the number of items needed per content area. Attachment # 11 has been added to page 110 of the RFP for item writing guidelines.

109 2.7.4 (p. 24). Will the TSA provide SMEs for new test item development?

TSA will inform the vendor of what content needs covered by test items and provide necessary materials. TSA will provide SMEs to review and approve vendor developed items.

110 2.7.6 (p. 24). How often are revisions and updates to the test items required? See response to question 105.

111 2.7.6 (p. 24).

Is it feasible to administer new certification items on an experimental basis to gather data for psychometric analyses? Potentially, as agreed upon on a case by case basis.

112 2.7.6 (p. 24). Will the contractor be responsible for psychometric analyses of new items? Potentially, as agreed upon on a case by case basis.

113 2.7.6 (p. 24). Is there a preference for psychometric model to calibrate items? (e.g., classical test theory or IRT) TSA does not have a preference at this time, and would determine the model on a case by case basis.

114 2.8 Validation Studies

115 2.8.1 (p. 24). What validation strategies have been used in the past?

TSA conducts validation studies in accordance with the EEOC Uniform Guidelines for Employee Selection (1978), the SIOP Principles for the Validation & Use of Personnel Selection Procedures (2003) and the Standards for Educational & Psychological Testing (1999).

116 2.8.1 (p. 24). Are prior validation study reports available for the current TSA assessments? TSA will provide as needed, if available.

117 2.8.1 (p. 24).

Will the TSA provide subject matter experts (SMEs) and job incumbents to participate in validation data gathering procedures? Yes

118 2.8.1 (p. 24). Will the TSA provide facilities for meetings and data gathering events? Yes, as needed

119 2.8.3.1 (p. 24). Does the TSA have job performance data collection forms from prior studies that may be used? There are forms available, however, the forms may need revising or updated according to the need.

120 2.8.3.1 (p. 24). Does TSA have a list of organizational data previously used as criterion? Yes, and will be shared as appropriate after contract award.

121 2.8.7 (p. 25). What method has historically been used for standard setting (i.e., to establish a cut score)?

The method used has varied depending upon the test, however, the Angoff method is historically the most common. TSA is open to other standard setting methods as appropriate.

Section VI - Instructions, Conditions, and Notices to Offerors

11Pages 16

3. Format for Proposal Submission and 4. Proposal Volume Instructions (pp. 131- 132).

The solicitation limits vendors’ responses to Factor 1: Tab A – Technical and Management Approach to 25 pages. However, under Proposal Volume Instructions, vendors are instructed to “describe and demonstrate the technical expertise, technical understanding, and detailed approach to accomplish all requirements specified in the SOW and Task Order 1,” and “All aspects of the work described in the SOW must be addressed.” Considering that the SOW in the solicitation is 63 pages long, and Task order 1 is a 40-page document, would TSA consider removing or increasing the page limit, to ensure that vendors are able to comply with the instructions for this section, and truly address all requirements? Page 131 of the RFP, Volume 1 Technical Tab "A" is now increased from 25 pages to 35 pages.

4. Proposal Volume Instructions

(p. 132).

TSA's instructions state that “All aspects of the work described in the SOW must be addressed" in Factor 1, Section 1. Does this mean that vendors should specifically address Statement of Work Sections 3-9? For example, Section 4, Travel, seems as though the vendor's response would consist of merely agreeing to TSA's travel requirements. While we acknowledge the importance of each of these sections to successful contract performance, it seems that using part of our 25 pages to discuss them would be detrimental to our responses to the more essential service requirements in Section 2, Specific Requirements/Tasks. Please clarify which sections of the SOW vendors should respond to in Factor 1, Section 1, of their Technical Proposal.

Offerors shall respond to all relevant technical aspects of the SOW that will effectively demonstrate an understanding of the tasks and performance requirements of the SOW. See answer to question 124 regarding page limit increase.

125 RFP Cover and Section VI, 4.1 Will the TSA provide an extension to the proposal submission date? See Question 2 above.

Section II. Reference 2.3.2.7;

deliverable 7

How many locations will contractor need to provide assessment materials for to support assessments administered by TSA personnel?

TSA desires materials to be provided electronically when possible, but hard copies may need to be provided to any TSA location (e.g., airport, FAM Field Office, HQ, etc.) as necessary.

Section II. Reference 2.3.2.17;

deliverable 14 Approximately how many scannable offline forms does TSA require annually?

Section 1.2 estimates 25,000 offline tests per year, and this typically corresponds with the number of scannable forms that would be needed. This number will vary.

Section II. Reference 2.7.3.

deliverable 29 Approximately how many certification items does TSA anticipate will be provided annually? TSA has approximately 3000 certification items. The number of items used in a given year vary depending upon program needs.

Section II. Reference 2.7.4, deliverable 30 Approximately how many new certification items does TSA require annually? TSA currently has approximately 3500 items, however, TSA anticipates with the award of the TAP contract this number will increase.

Section II. Reference 2.7.4, deliverable 30

Will TSA provide SME’s to assist in writing new certification items? Will item writing workshops need to be conducted at one central location or at multiple locations? If more than one location, how many and what are the geographical locations? See response to question 109. Item writing workshops will be conducted as needed and are typically conducted via video-teleconference.

Section II. Reference 2.8.1;

deliverable 32

Will the contractor be required to travel to field locations in order to conduct validation studies? If so, how many per year?

Travel to field locations will potentially be required depending upon validation methodology.

132 Section VI, 4.3 In regard to page limitations, are cover pages included in page count? No, Cover pages are not counted. The page limitation has been increased to 35 total pages for Factor 1, Tab A.

Page 5 of Task Order 001 under Background

"The Catalog of Assessments and Testing System (CATS) will be used by TSA, Office of Human Capital (OHC) to evaluate candidates and incumbents within a variety of positions including Transportation Security Officers (TSOs); Screening Partnership Program (SPP) screeners; Federal Air Marshals (FAMs);

management, administrative, and professional (MAP) positions; and executive positions. These positions will be in a variety of business, technical, and security operations-related disciplines at the entry-level, team lead, first-line supervisor, manager, and executive levels." Can TSA please estimate the number of testers to be processed by the system annually?