Attachment 1 - Scope of Work - NTEWS Bridge Panel.pdf

Scope of Work for NTEWS Bridge Panel

10 March 2022

Statement of Work

Conduct a web-based experimental survey called a “Bridge Panel” based on the National Training and

Education Workforce Survey (NTEWS) in spring 2022.

I. Overview

A. Purpose

The Contractor will conduct the inaugural NTEWS Bridge Panel.

Specifically, the Contractor will:

• Develop a probability-based sample plan for 5,000 cases of 16-75 year olds (not in high school)

• Secure appropriate frame data and select the sample

• Program a web-based instrument in English, using the 15-minute questionnaire provided by

NCSES

• Contact and recruit sampled cases to participate

• Collect and process NTEWS Bridge Panel data; develop data files for survey and para- data

• Prepare and deliver data files for the user community including creating and documenting public use files and restricted use files

• Conduct contractor-proposed research analyses (methodological and substantive) and produce publication products.

Data analysis and reporting for NCSES’ purposes is not part of this project.

B. Background

The National Center for Science and Engineering Statistics (NCSES) within the National Science

Foundation (NSF) will collect the first wave of the National Training, Education, and Workforce Survey

(NTEWS) in spring 2022. This collection is designed to better understand the Skilled Technical

Workforce (STW): those who use science, technology, engineering, and mathematics (STEM) skills and knowledge in their jobs, but do not have a Bachelor's degree. NTEWS will measure two research concepts that are of national interest: (1) the education, training, and career pathways of skilled technical workers and (2) the prevalence and interplay of education credentials (postsecondary degrees and certificates), work credentials (certifications and licenses), and work experience programs in fostering skill development among American workers.

The purpose of this RFP is to conduct a survey in parallel to the NTEWS that will explore modifications to the content through a small, representative, non-production sample. We call this experimental survey a

Bridge Panel (BP). The BP will allow NCSES to explore an expansion of and different approaches to some of the survey content in the current round of the NTEWS, particularly areas that were inconclusive during the cognitive testing. The BP is named so because it serves as a bridge between the current and future cycles of the NTEWS for question and methodological development and statistical adjustment. As named, we do intend to follow this inaugural panel of sampled cases for subsequent data collections.

While it would be ideal for the BP to mirror the essential survey conditions of the NTEWS as closely as possible, using a different data collection vendor for the BP introduces some unavoidable differences.

Mainly, the NTEWS sample is drawn from the American Community Survey which can only be accessed by the US Census Bureau. Therefore, the BP will use another frame source or sources to access the population of interest and allow for comparable statistics. Also, the NTEWS is multi-mode but the BP will be web-only. There will be additional differences in the look and format of the web instrument and contact materials. The field periods of the data collections will overlap, although the BP’s field period will be shorter.

The project originates from NCSES’ Research Program and therefore, the data are considered experimental rather than official data for federal statistics. Being such, NCSES would like the Contractor to maximize the applicability of these data for their own and others’ research purposes. This should be reflected in the plan for data collection (e.g., taking care to capture relevant contact and user paradata), plans for research analyses (e.g., proposing analyses that build on an existing body of work), and deliverables for the user community (e.g., fully transparent metadata). The Contractor has the flexibility to propose design and data collection innovations as long as the final data can be aligned with the

NTEWS survey population, allowing for comparisons between the main NTEWS data collection and the alternative versions of question in the BP.

This contract solicitation is unique because it requires data collection capabilities, operational expertise in the areas of survey and data collection methodology, and academic research interests in methodology and/or non-degree credentials and/or demographic characteristics (e.g. analyzing the results SOGI or disability questions). NCSES would ideally like to work with a university-based survey house and associated faculty interested, either methodologically or substantively, in the data produced for this project. NCSES would look favorably on the involvement of graduate students or postdocs in the project, if possible, and/or plans to use the data for further research or coursework after the close of the contract.

The overall goal of the project is simple: to design the data collection, collect the data, and deliver the data and supporting materials. NCSES is not requesting analyses for their purposes in this contract. The specifics of this project, however, require a steady emphasis on quality assessment across all stages of the project, and full transparency of these quality assessments, methods, and decisions. Any future user of the data should have all the documentation required to conduct analyses (particularly methodological) without having to request further information or make assumptions about the data collector’s decisions.

This may be more detailed documentation than typically requested in government contracts. NCSES is requiring an academic research analysis and/or use of the data for student coursework as part of deliverables. This underscores the need for detailed documentation to allow the data to be used for these purposes.

The primary tasks in the Statement of Work are:

Task 1 Overall project management

Task 2 Frame acquisition and sample selection

Task 3 Develop contact and incentive plan

Task 4 Questionnaire review and programming

Task 5 Data collection

Task 6 Data processing, weighting, and dataset preparation

Task 7 Research analysis

Task 8 Documentation and deliverables

Applicants are allowed to separate the tasks into additional subtasks beyond those provided to facilitate reporting of their proposed work plan, timeline, budget, and deliverables.

This project will be monitored by NCSES’s contracting officer’s representative (COR) and the NSF

Division of Acquisitions and Contracts contracting officer (CO) on behalf of NSF.

The Contractor shall comply with all federal requirements relating to information security and privacy, including federal privacy laws (i.e., CIPSEA, NSF Act of 1950, and the Privacy Act), the Federal

Information and Security Management Act, the E-Government Act of 2002, all relevant National Institute of Standards and Technology (NIST) Special Publication (SP) 800 series publications, such as NIST SP

800-53 Rev. 4 (Security and Privacy Controls for Federal Information Systems and Organizations);

Office of Management and Budget (OMB) memoranda related to security and privacy; and FIPS publications such as FIPS 199 and FIPS 200 (Minimum Security Requirements for Federal Information and Information Systems).

II. Specification of Work

Task 1. Overall project management

This task includes, but is not limited to, a kick-off meeting, regular check-ins with the NCSES project lead, monitoring the schedule and budget, and confirming satisfactory execution of quality assurance activities throughout the project. To facilitate the communication between Contractor and NCSES, the

Contractor will designate a main point of contact to communicate with the NCSES project lead. The proposal will include a schedule indicating the target completion date of each task and/or stage of the project. This will be monitored weekly over the course of the project and contingency plans set in motion to keep the project on schedule.

Task 2. Frame acquisition and sample selection

As noted in the Background, the BP will produce data that can be compared to the NTEWS. For context, the NTEWS will collect data from a nationally representative sample survey of the noninstitutionalized population residing in the United States or Puerto Rico who will be aged 16 to 75 years old as of March 1, 2022 and who are high school graduates or otherwise not attending a primary or secondary school. The sample will be selected from respondents to the U.S. Census Bureau's 2018 American Community Survey

(ACS), with oversamples of adults who do not have a Bachelor's degree and adults in the skilled technical workforce (STW). This sampling approach will allow NTEWS to generalize to two populations: (1) all adults ages 16-75 living in households and (2) all adults ages 16-75 in the STW.

The 2022 NTEWS sampling plan stratifies individuals by crossing education level, sex, underrepresented minority classification, and STEM vs. non-STEM workforce status. The sampling plan uses the Adult

Training and Education Survey (ATES) data to estimate proportions of the population in these strata who have (i) certifications or licenses, (ii) participated in work experience programs, and (iii) have postsecondary certifications. These proportions are then used to allocate sample across the strata to achieve a desired coefficient of variation within each stratum for these three estimates. If desired, NCSES can provide an embargoed version of the NTEWS sampling plan and data from the 2016 ATES after award of the contract.

The target population for the BP will mostly mirror the NTEWS but not include Puerto Rico, group quarters or military bases. The sample will be probability based, covering the 50 states, including the

District of Columbia. The survey will collect data from noninstitutionalized adults aged 16 to 75 years old

(inclusive) as of March 1, 2022, residing in the United States, including the District of Columbia, who are not attending primary or secondary school. Without a list of individuals on the frame, as in the ACS sample, a within household section strategy will be necessary which will likely require sampling addresses and then a within household selection.

To mirror the NTEWS, the BP sample will stratify on education (high school or less, some college or higher), sex (male, female), and minority status (underrepresented minority; not). We expect the

Contractor to append address and/or block level data to the frame as needed in order to stratify on these

(and any other proposed) characteristics. Some characteristics may also be used in a household screening process to select respondents. The Contractor may append variables not specified in the sample design in order to conduct nonresponse or other methodological analysis. For all variables appended, matching rates shall be reported.

Note in Task 4, under Programming, there may be additional sample design considerations if the requested experiment on the interactive map is implemented. For this experiment, the Contractor will divide the sample into control and experimental groups.

The Contractor will conduct quality control checks of the frame and detail the corrections made and known errors of the frame, especially if appending commercial data. Examples of quality checks include, but are not limited to: identification and removal of duplicates, address corrections for clarity and completeness to prevent undeliverable and redirected mail, expected number of elements (total and by subgroup) within sampling units given current population estimates, identification and removal of business and vacant addresses, completeness of appended variables, reasonable values and distributions for appended variables, etc. Once in production, the monitoring and recording of quality measures will continue. Final reporting of quality checks will reflect quality measures before and after data collection separately so that the weaknesses in the frame used for sampling are transparent.

A quality assurance plan including details of the documentation that will be provided will be submitted to

NCSES for approval prior to drawing the sample. Allow two weeks for NCSES review, amendment (if applicable), and final approval.

Task 3. Develop contact and incentive plan

Contact Materials

The Contractor will develop appealing and professional contact materials designed to encourage response, provide contact information for inbound contacts, and encourage contact for those needing help or confirmation of the legitimacy of the data collection. Mailed contact materials will include a unique QR code to allow would-be respondents to immediately connect to the survey instrument. The Contractor should provide their preferences for branding on communication materials (e.g., emphasizing government sponsorship vs. university sponsorship) in their proposal. NCSES will approve or amend at award of the contract.

We will treat this inaugural sample of Bridge Panelists as panelists who will be contacted again in two years. The draft questionnaire collects individual contact information but this intention to follow-up must be documented in the consent form and possibly the contact materials. We do not expect proposals to include longitudinal sample maintenance at this time but, after the close of this contract, we reserve the right to request that the Contractor either access the sample to administer between-wave maintenance contacts or transfer the sample with updated contact information (from the questionnaire) to a new vendor for between wave or wave 2 follow-up.

Contact Protocol

The Contractor shall propose a contact protocol appropriate for this population, mode of data collection, and length of field period, specifying the frequency and modes of contact. As noted above, the BP target population is 16-75 year olds who are not in high school. Without a named sample, a screener and within household selection strategy will be necessary which will likely require household contact for screening and selection and then individual contact with the selected respondent. The Contractor will develop the screener, within household selection protocol and follow-up procedures with the selected respondent. We are aware of the additional nonresponse that will likely result when the individual selected is not the person who provided the household information. Proposals should suggest procedures for mitigating this.

If not part of the proposal, NCSES will require one week for review, revision (if applicable), and approval of screener and selection protocol.

Incentives

The Contractor has permission, approved by OMB, to offer a $30 prepaid debit card (physical or electronic) to a subset of highly influential bridge panel households at week 1 of data collection. “Highly influential” refers to households that have large sampling weights and a low response/locating propensity.

The Contractor will have to use the frame data to model these propensities and designate these cases.

NCSES expects to offer the debit card incentives to approximately 1,000 of the 5,000 BP cases at the first contact. The modeling and selection strategy must be approved by NCSES prior to first contact. NCSES will need two weeks for review, revision (if applicable), and approval.

In addition, at week 6, 50% of the remaining individuals (not in households already offered an incentive) could be offered a nonresponse follow-up incentive of $30, considering which are most influential for balancing the sample at that time. The Contractor will have to monitor the sample composition and determine the 50% most influential cases. In week 1 or 2 of data collection, the Contractor will submit the modeling and selection strategy to NCSES for approval. NCSES will need two weeks for review, revision

(if applicable), and approval.

The contractor will obtain and manage the distribution of debit cards. These will have a six-month usage period at which time the cards will expire, and the unused funds will be returned to the Government.

If desired, the Contractor may propose a different incentive plan, not exceeding the total amount allocated for incentives and allowing (through modeling or other means) each sample case a non-zero chance of receiving an incentive.

Contact Paradata

Paradata will, at a minimum, capture the following.

Inbound and outbound contacts (timing and attributes) should be captured in the paradata file for an individual address. Information on incentives should be captured including the dates that the incentive was administered, mode of delivery, amount of incentive, contacts about incentives not received and re-administration of incentives, and dates of cashing incentives. Propensity scores (or other ranking) used to determine influential cases at both stages of incentive administration should be provided. A codebook or framework for the contact paradata variables must be approved by NCSES prior to commencing data collection. NCSES will need two weeks for review, revision (if applicable), and approval. Additional time may be necessary to implement the changes in the Contractor’s systems before commencing data collection.

Task 4. Questionnaire review and programming

Mode

The survey data will only be collected using web and the questionnaire design leverages the strengths of this technology to promote data quality. A paper questionnaire will not be considered. Given the small number of interviews that could be obtained using CATI, it is not worth the expense and introduction of additional error to introduce this mode for the survey data collection.

Description of Questionnaire

Not including the household screener that the Contractor will create, the survey has 101 possible questions including all skip sequences, demographics, and the request for contact information. One of those questions is a grid with 9 topics to evaluate. The questionnaire should take about 15 minutes, not accounting for any front end information like instructions, welcome, and consent. The 15-minute estimate is for a respondent answering the (at most) 31-question series about the characteristics of their non-degree credential for two different non-degree credentials, completing the series twice.

Review of Questionnaire

The draft questionnaire is available upon request and we strongly recommend reviewing this to understand the complexity and topics. We expect very little questionnaire design revision but the

Contractor may budget for an in-house “expert review” to identify any potentially significant problems. In the questionnaire review, the Contractor should advise on the 3-question series exploring position of the respondent within the household and the household’s reliance on the respondent’s income. The 2 new questions on employer support to obtain the credential would also benefit from close review.

No cognitive testing will be conducted.

The Contractor will advise as to which questions are mandatory and which can be skipped after delivery of a soft prompt. NCSES will review and approve the plan and the final version of the questionnaire before programming begins. Allow two weeks for review, amendment (if necessary) and approval.

Programming

NCSES does not require use of a specific software for administration of the web-survey so long as it is optimized for both web and mobile devices. The web survey must be capable of running on a variety of web browsers, including Chrome, Firefox, and Safari, with appropriate security procedures and protection of transmitted confidential data. In addition, the web survey data collection system and access to the system must be transportable and compliant with section 508 of the Rehabilitation Act of 1973.

FAQs should be accessible to sampled cases prior to commencing the survey. We suggest providing a link to these on the landing page and a help button throughout the instrument but the Contractor may propose other locations.

The Contractor does not have to use the variable names noted in the draft questionnaire. These were chosen by the questionnaire designer for convenience of keeping track of skips and fills and do not correspond to NTEWS or any other survey instrument.

For the question about geographic flexibility in the respondent’s job search, we request the development of an interactive map of the US, allowing respondents to select individual states. Ideally, the Contractor will propose an experiment to test the superior data quality of answering a question with this method compared to a text-based response option (serving as a Control) that they develop.

Testing

Once the survey instrument is programmed, the Contractor will conduct extensive functionality testing to make sure the autofills, skip patterns, built-in checks, and soft prompts are working and all data are recorded in the database as expected. Any functionality problems will be fixed before moving to usability testing. The usability testing phase will assess the usability of the instrument on various mobile devices

(tablet, phone) and operating systems (iOS, Android). Log all changes, edits, and updates done during both stages of testing. Provide the log and questionnaire (both can be in draft form) to NCSES along with the final electronic version for final testing and final approval. NCSES will require two weeks to conduct final testing of the instrument. Any errors found will need to be corrected and the functionality and usability tested conducted a second time. NCSES will require an additional week for final approval of the instrument if errors were found in the first attempt for final approval. This process will repeat as often as needed until final approval is given.

Paradata

Paradata will, at a minimum, capture the following.

Record the device for each login, when the FAQs are accessed, which FAQs are opened, and if possible, if the user proceeded to the questionnaire from the FAQs. A framework or codebook of the paradata requested in this section should be submitted to NCSES for approval prior to commencing data collection.

Allow two weeks for NCSES review, amendment (if necessary), and approval. Allow additional time to implement changes to the Contractor’s systems (if necessary).

Task 5. Data Collection

Data Collection Period

The BP will use a single-mode, web. The field period is expected to last a minimum of 8 weeks but no more than 12 weeks. As explained in the Task 3, incentives may be offered at weeks 1 and 6, unless another protocol is proposed and approved.

Assurance of Resources to Handle Survey Volume

Adequate and secured computer resources shall be provided by the contractor to handle the survey volume throughout the data collection phase to avoid any Internet connection interruptions. The Contractor should regularly monitor survey data submissions to identify any Internet browser issues early.

Data Quality Monitoring

As much as possible, the proposal will provide details of plans for data quality monitoring throughout the data collection. Quality issues pertain to gaining timely contact, identifying the selected respondent, securing cooperation, maximizing survey data quality (e.g., item nonresponse and straight lining), and minimizing nonresponse bias, among other things. At the point of award, NCSES may identify additional quality monitoring.

The target response rate for the NTEWS is 63%. However, we do not expect this level of response for a web-only survey, without Census branding. We would like the Contractor to aim for 40% response, providing roughly 2000 survey responses for the BP. The Contractor, in the proposal, should highlight planned efforts and interventions to maximize response and balance the representativity of the data. At the point of award, NCSES may identify additional quality monitoring.

Paradata

Paradata should, at a minimum, capture the following.

The Contractor will prepare contact history data recording the contact data noted in Task 3 as well as interim and final disposition codes with dates and times for all sample cases. Web paradata will include

(but not be limited to): login dates, times, locations, and device, page timings, timestamps and page/question for break-offs, triggering of soft prompts or error messages, use of the back button, changes to responses, original and revised responses, device, use of help materials, and other indicators for difficulty with the instrument or respondent fatigue. A framework or codebook of the paradata requested in this section should be submitted to NCSES for approval prior to commencing data collection. Allow

NCSES two weeks for review, amendment (if necessary), and approval and additional time (if necessary) for the Contractor to implement the changes in their systems.

Data Confidentiality and Protection

Provide an assurance of data confidentiality and protection throughout the project. This includes protecting the security of the frame, survey, para-, and other administrative data at the Contractor site, protecting the security of the data when transferring/delivering confidential data, protecting the confidentiality of respondents, protecting the privacy of survey respondents among staff, protecting against staff disclosure of personally identifiable information for survey respondents, and documenting any other confidentiality and access issues. As part of this assurance, the contractor shall provide to

NCSES a Confidentiality and Data Security Plan that details the steps the contractor will take to protect the data throughout its lifecycle. NCSES will provide guidance to the Contractor (in the form of a checklist) to write the CDPSP once the contract is awarded and the Contractor will have 3 weeks to submit. NCSES’ Data Confidentiality Officer will advise on the CDPSP and approve.

A duly authorized representative of the Contractor’s organization with legal authority to bind the organization must sign the NCSES Organization Data Use Agreement (DUA). All Contractor staff

(including any subcontractors) working on the BP must take the NCSES confidentiality training and sign the NCSES Individual DUA, acknowledging their responsibilities to protect the data. These must be submitted to NCSES within one month of contract award and no Contractor staff may work with data from this project until this requirement is completed.

IRB

OMB clearance has already been obtained for this work, including use of a commercial sample frame, incentive amounts, and the questionnaire. However, the Contractor will have to secure approval from their Institutional Review Board and should allow time for this in the project timeline.

Task 6. Data processing, weighting, and dataset preparation

Data Preparation

There is no specific data treatment required other than minimal general cleaning, such as for inappropriate responses. Open ends will not be coded as part of this contract. For any general editing or cleaning performed, this should be thoroughly documented in the file with flags and in accompanying documentation. The restricted use file (RUF) and public use file (PUF) will likely have different treatments due to disclosure protections in the PUF. The restricted use survey data should include a record for any person who initiated the survey but flag those cases without enough information to be considered a partial.

These data will be protected under CIPSEA. The Contractor will submit a plan for disclosure protection of both the PUF survey and para- data. NCSES will conduct a disclosure review and approve before the work proceeds (allow two weeks). Researchers working with the restricted data (RUF) will have to work within the NCSES data enclave, in which results are subject to disclosure review by NCSES before export. All data from the BP will be publicized as probability based but not for the generation of official statistics.

Weighting and Imputation

The Contractor will develop base weights to account for probability of selection according to the sampling plan implemented. Additional weight adjustments should account for nonresponse bias and coverage error. Weight trimming to control for extreme weights and adjustment to external benchmarks such as ACS totals is requested. The contractor should create sets of replicate weights to allow for adequate variance estimation or propose equivalent methods such as Taylor linearization and provide supporting variables (e.g., cluster and strata IDs).

No imputation is requested in this contract.

Task 7. Research analysis

NCSES is not requiring a report, tables and other analyses of the survey data for their purposes and, other than the frequencies included in the codebooks (weighted and unweighted), these should not be included in the proposal. NCSES welcomes proposals for analysis of the data that benefit the academic researchers involved. It is expected that, at a minimum, one manuscript will be prepared on a topic of the researchers’ choice for submission to a peer reviewed journal. Please use this section of the proposal to detail any future plans for the data in methodological or substantive research, or for coursework.

Task 8. Documentation and deliverables

Data

Note that all stages of the project request paradata or data on quality measures. These are important deliverables. As described in Task 6, both the paradata and survey data files should be prepared for both restricted and public use. The Contractor should deliver the survey data and paradata files in STATA, SAS, and CSV.

• A fully documented public-use dataset of survey data will be available for download from the

NCSES website. Data will be separated from the production survey data to ensure that Bridge

Panel data cannot be used to generate official statistics.

• A fully documented restricted-use dataset of survey data will be available for restricted-use data license holders through the NCSES data enclave. Data use will be subject to NCSES’s disclosure review.

• Replicate and nonresponse weights data file.

• Paradata files prepared for public and restricted use, including contact and web paradata as detailed in Tasks 3, 4, and 5 above.

• Data record of incentives indicating which cases received what value of incentive, date of administration and date of card activation. This file should also include the prioritization or propensities used to determine the “highly influential” cases chosen for incentives and all details noted in Task 3.

• The sample file with PII removed with appended frame variables.

Documentation

Deliver the following, at a minimum:

• Crosswalk of variable names of comparable NTEWS questions and BP questions

• Codebook with weighted and unweighted frequencies and standard errors of all survey variables, discriminating between user-initiated item missing data and planned missing data due to skip logic

• Codebook on the variables in the sample frame

• Documentation on the sample frame and sample file detailing the quality checks and edits made prior to, during, and post data collection

• Syntax used for frame cleaning and sample selection

• Examples of all contact materials

• Questionnaire in Word and PDF with skip logic.

• Questionnaire testing log as detailed in Task 4.

• Screenshot documentation of the questionnaire, capturing all front end (e.g. consent, login screen, landing page) and back end (e.g., thank you screen) information, error messages, soft prompts, and help/FAQs.

• A methodology report including, at a minimum: description of the frame and its quality, sample design and selection, weighting procedures, contact protocols, description of the data collection vendor, specifics of the data collection protocols, documented problems during data collection, data cleaning, final dispositions, summary of response rates, comparison of the response distribution compared to the population before and after weighting, and any other quality assurance procedures not noted previously.

The deliverables of data and supporting documentation are critical to supporting the goal of supporting further research using these data. In an appendix, submit a checklist of deliverables, dividing them into public and restricted use materials. This list will be kept up to date throughout the project, if additional deliverables or detail are identified. At the close of the project, the list will accompany the deliverables, noting the file name for each entry. All deliverables will be electronic.

Provide details as to how the Contractor proposes securely delivering the data and files to NCSES.

Timeline

We expect to award this contract in late April 2022 and the project will kick off immediately. The BP data collection will begin by 1 August 2022 at the latest. The data collection period is expected to last a minimum of 8 weeks but no more than 12 weeks. The data collection phase is the critical path for this project and all other tasks should be scheduled around this. The delivery of documentation and data should follow as quickly as possible following the close of data collection.