ODPPS_SOW 5.1.docx

Testing a probability-based online panel self-administration of the Consumer Expenditure Diary Survey I. Background The Consumer Expenditure (CE) Diary Survey at the Bureau of Labor Statistics (BLS) is currently fielded by using an in-person interview to place two copies of a one-week paper diary instrument that the respondent records expenditures in for two consecutive weeks. Interviewers place the diaries and train respondents on how to record the household’s daily expenditures. The interviewers also contact the respondents during the week to encourage the diary keeping task. In October 2019, the CE began a test of an online diary with the expectation of implementing the online diary into production in 2022 using the same in-person placement protocols as described above.

In-person interviews historically have higher response rates than other modes of data collection and depending on content can offer higher quality data. However, the main drawback to in-person interviewing is the higher cost compared to other modes of data collection.

Because the CE diary is already primarily self-reported, it lends itself to moving away from an in-person placement to being fully self-administered. This could lead to cost savings in the future; however, switching to a fully self-administered mode is not without risk. First, the response rates in self-administered modes tend to be lower than in-person modes. Also, there’s uncertainty surrounding the impact on quality of a fully self-administered diary without the aid of in-person training on diary-keeping procedures.

In order to explore the impact of a fully self-administered mode of the diary, we propose a test in an online probability panel. While results may not be fully comparable to a traditional address-based sample design, they will provide insight on some of the sampling and measurement error that might result from a fully self-administered mode.

A further benefit of testing a self-administered diary in an online probability panel (with recruitment from an address based sample with known probabilities) is to establish a platform that could allow for future experiments. These experiments could be more quickly designed and tested in an online panel with results feeding faster into production.

II. Objective A previously developed open-source online diary (developed using Angular JS, NGiNX, and Elasticsearch) shall be modified to reflect minor changes that were made in the CE’s large-scale feasibility test of online diaries (e.g. addition of fields to collect point of purchase information, layout of user settings, modification of username and password requirements, changes to select response options in fields). The updated diary shall be used to field an online test of a two-week Diary using an online probability panel with a sample size of 2,000 respondents divided across 3 consecutive months to meet the following research objectives of BLS:

1. Test the feasibility of administering a two-week online diary to a probability panel sample without the aid of an interviewer to assist in gaining cooperation and training the respondent on how to record expenditures.

2. Evaluate the response and cooperation rates for keeping a fully self-reported online diary, within the context of a probability based online panel.

3. Compare the sample composition of those that complete an online diary to that of the BLS CE diary production sample to determine the differences in representativeness for various population subgroups.

4. Identify any deviations in data quality from a self-administered online diary compared to CE diary production data.

The contractor shall be responsible for updating the open-source diary, developing a self-administered family questionnaire for collecting demographic and income information (as needed), administering the test in an online probability panel, and providing the resulting data tables (including paradata) based on specified output requirements to the BLS, including the assignment of given item codes to reported expenditures.

III. Government Furnished Items

1. Services. The Government will provide personnel required to answer questions about BLS data or procedures, review and provide feedback on deliverables.

2. Facilities. No facilities will be provided for this task order.

3. Equipment. No equipment will be provided for this task order.

4. Materials. The Government will provide all materials and information that are unique to the BLS or BLS programs. Examples include access to BLS data, interviewer training and materials, respondent materials (e.g. user guides, advance letters, and brochures), access to previous CE online diary instruments, and CE data requirements (See Attachment 1).

5. Software. The Government will provide current open-source online diary code and related documentation (see Attachment 2: Consumer Expenditure Diary System Overview for system architecture) IV. Tasks

1. Provide a Task Management Plan. Submit a Task Management Plan detailing the activity schedule including tasks and durations for each activity included in the project.

2. Modify Diary Instrument. Modify the code (developed using Angular JS, NGiNX, and Elasticsearch) for an existing online diary to incorporate the collection of additional fields and to reorganize some of the user layout and options. Allow for testing and feedback from BLS staff.

3. Develop Self-Administered Household Characteristics Questionnaire. With input from BLS staff, using the current CE Diary Survey Household Characteristics Questionnaire design as a guide, create a self-administered questionnaire to be administered to the respondent that collects information on the household, including member demographics and income, and typical spending. If the data are already available for the units to be sampled, then a cross-walk shall be provided between questions/response options from the respondent and the CE Diary questionnaire.

4. Respondent Materials. With input from BLS staff, develop and print (as needed) survey materials, to minimally include respondent notifications (e.g., notification letters, instructions for online diary access).

5. Develop test protocols. With input from BLS staff, develop contact strategies and protocols for the test, including recommended incentives.

6. Provide OMB materials. Provide inputs for the BLS submission of an OMB clearance package required to field the test. These include electronic copies of all materials provided to the respondent for the test (e.g., user guide, tutorials, screenshots, emails) and a written description of the test methods (and protocols). The OMB Clearance Package will be submitted in September 2021 and we estimate 9 months for OMB approval to field the test (June 2022).

7. Test in Online Probability Panel. Select representative sample in an online probability panel and administer the household characteristics questionnaire and online diary using the agreed upon protocols. The sample shall be divided into three months, with each month representing a fielding period (e.g., a participant that receives the online diary invitation in month x shall start the two week diary in month x).

8. Help Desk training and support. During the test, the contractor shall provide a help desk for users to contact with any issues concerning the use of the online diary. The help desk staff shall be trained to answer both technological questions about the diary and key survey concepts (e.g., what level of detail to include in reporting, where to report items).

9. Develop weights. Develop weights for each responding household so that the final completed sample is representative of the US population. Provide documentation on weights to allow for comparison to factors used in BLS production weighting.

10. Assign item codes. Assign an item code, defined by BLS, to each reported expenditure based on expenditure description and category collected from the online diary. As needed, BLS will provide a non-exhaustive list of example expenditure text by category for each item code. Historically, this list can provide an item code for roughly 70% of all reported expenditures through matching.

11. Attend Weekly Conference Calls. During the life of the project, calls with BLS will be conducted weekly, with the contractor providing an update on the design development, testing, and identifying any potential issues that have arisen.

12. Submit Final Report. Submit a final report describing the fielding of the test, including what protocols were used and the outcome of the responses and a summary of types of help desk calls received. The report shall include any recommendations for protocol changes based on the outcome and help desk calls.

13. Final Deliverables. Deliverables will include revised code and revised documentation for the online diary and household characteristics questionnaire (as needed), respondent materials (including online), monthly progress reports, data tables from the test and other associated household characteristics (when available) provided in CSV and/or SAS (including weights and associated para- and metadata) based on specified output requirements (including the assignment of given item codes to reported expenditures), and a final report.

V. Requirements:

1. Contractor shall be available for weekly status meetings with CE representatives to discuss progress and resolve technical issues.

2. Contractor shall submit all protocols, materials and instruments to CE for review and comment prior to the completion of diary and questionnaire development on a monthly basis.

VI. Deliverables/Milestones and Delivery Schedule

VII. Period of performance At least 17 months from contract award date.

VIII. Location of Performance All work shall be performed at the contractor’s site.

IX. Travel There will be no reimbursable travel for this requirement.

X. Confidentiality and Security Requirements

The Contractor shall comply with all BLS confidentiality and security requirements. Contractor personnel with access to BLS confidential information, including respondent identifiable and personally identifiable information, will need to become BLS-designated agents and complete BLS confidentiality training. The Contractor must ensure the secure transmission of confidential information which includes meeting the standards of Federal Information Processing Standards Publication 140 (FIPS PUBS 140) for electronic data transfers. In addition, the Contractor must implement the respective security controls catalogued in the National Institute of Standards and Technology (NIST) Special Publication 800-171, “Protecting Controlled Unclassified Information in Nonfederal Systems and Organizations”. The BLS Confidentiality and Security Requirements, which include the Confidential Information Protection and Statistical Efficiency Act and the BLS Agent Agreement, can be found in Attachments J.H.1 to J.H.3.

XI. Inspection and Acceptance Designated staff of the BLS will make final inspection and acceptance of specified deliverables.

Attachment 1. CE Data Requirements I. Food and Drinks Away from Home

1. Purchase Date

2. Point of Purchase

3. Item Description

4. Cost

5. Meal Type

6. Alcohol Type

7. Alcohol Cost

II. Food and Drinks for Home Consumption

1. Purchase Date

2. Point of Purchase

3. Item Description

4. Cost

5. Packaging Type

6. Purchased for Someone Outside of the Household

III. Clothing, Shoes, Jewelry, and Accessories

1. Purchase Date

2. Point of Purchase

3. Item Description

4. Cost

5. Gender

6. Age

7. Purchased for Someone Outside of the Household

IV. All Other Products and Services

1. Purchase Date

2. Point of Purchase

3. Item Description

4. Cost

5. Purchased for Someone Outside of the Household

Attachment 2: Consumer Expenditure Diary System Overview The BLS CED web-based system is built using a three-tiered architecture pattern. The tiers are as follows:

1. Presentation Layer – this is the UI (User Interface) for the application. This runs in the end-user’s web browser either on a computer or mobile device.

2. Business Logic – this component coordinates all objects within the application. It handles processing data requests, invoking business rules\calculations, etc. This component runs on the Application\Web Server.

3. Data Layer – this component handles the storage and retrieval of the data required by the system. The data layer runs on the Database Server.

For each tier, the open source technologies are described below.

Presentation Technologies The presentation layer provides access to the system through the UI. This tier will consist of the following technologies:

· AngularJS is JavaScript framework used to build web applications. AngularJS makes use of the following underlying technologies:

· JavaScript

· HTML5

· CSS3

Business Logic Technologies

The middle tier is responsible for process service and data requests within the application. This is where the logic for the system lives and it will be comprised of RESTful Web Services. This tier is comprised of the following technologies:

· NGiNX – Application\Web Server to host Web Services

· Spring Boot – Framework for building RESTful Web Services

· Java – Object oriented programming language Data Layer (Database) Technologies

This tier is responsible for the efficient storage and retrieval of data within the system. This tier is comprised of the following technology:

· Elasticsearch - Document oriented database. Based on Lucene, Elasticsearch offers enterprise grade performance, is highly scalable, and highly available. In previous tests of the online diary, it was implemented on a three node cluster running on Ubuntu. Elasticsearch requires a minimum cluster size of 3 and can scale to thousands of nodes. These nodes not only distribute the data, but load balance data requests as well.

References AngularJS - https://angularjs.org/ NGiNX - https://nginx.org/en/ Spring Boot - http://projects.spring.io/spring-boot/ Java - https://www.java.com/en/ Elasticsearch - https://www.elastic.co/products/elasticsearch

Application\Web Server

NGiNX

Spring Boot

Java

Database

Elasticsearch

Web Browser

AngularJS

JavaScript

HTML5

CSS3