J.6 Alerts and Warnings Hypothetical Task Order.pdf

Title: Alert and Warning Characteristics for Advanced Driver Assistance Systems

Attachment J.6 - Hypothetical Task Order

Alert and Warning Characteristics for Advanced Driver Assistance Systems

SECTION C - DESCRIPTION / SPECIFICATIONS / STATEMENT

OF OBJECTIVES

This is a Hypothetical Task Order entitled “Alert and Warning Characteristics for Advanced Driver Assistance Systems” for Solicitation No. 693JJ924R000031, “Human Factors”. The purpose of this Hypothetical Task Order is to obtain a technical plan of how the Offeror will complete the order, which includes a discussion of problem identification, study, and evaluation techniques, and reporting methodology. The purpose of this Hypothetical Task Order is also to obtain a detailed cost proposal to determine the reasonableness of the cost for future Task Orders.

Offerors’ proposals may also be used as a basis to award this Task Order to the Offeror whose proposal, conforming to the requirements of this solicitation, represents the best value to the Government.

C.1 ACRONYMS

ADAS Advanced driver assistance system AEB Automatic emergency braking ATOA After Task Order award BSW Blind spot warning CO Contracting Officer COR Contracting Officer’s Representative FCW Forward collision warning HQ Headquarters HTO Hypothetical Task Order LDW Lane departure warning NHTSA National Highway Traffic Safety Administration ODC Other direct cost OIRA Office of Information and Regulatory Affairs OMB Office of Management and Budget OST Office of the Secretary of Transportation PI Principal Investigator PRA Paperwork Reduction Act RFP Request for proposal SAE SAE International TO Task order USDOT United States Department of Transportation

C.1 INTRODUCTION

The National Highway Traffic Safety Administration (NHTSA), under the U.S. Department of Transportation, was established by the Highway Safety Act of 1970, as the successor to the National Highway Safety Bureau, to carry out safety programs under the National Traffic and Motor Vehicle Safety Act of 1966 and the Highway Safety Act of 1966. The Vehicle Safety Act has subsequently been recodified under Title 49 of the U. S. Code in Chapter 301, Motor Vehicle Safety. NHTSA also carries out consumer programs established by the Motor Vehicle Information and Cost Savings Act of 1972, which has been recodified in various Chapters under Title 49.

NHTSA is responsible for reducing deaths, injuries and economic losses resulting from motor vehicle crashes. This is accomplished by setting and enforcing safety performance standards for motor vehicles and motor vehicle equipment, and through grants to state and local governments to enable them to conduct effective local highway safety programs.

NHTSA investigates safety defects in motor vehicles, sets and enforces fuel economy standards, helps states and local communities reduce the threats of drunk drivers, promote the use of safety belts, child safety seats and air bags, investigate odometer fraud, establish and enforce vehicle anti-theft regulations and provide consumer information on motor vehicle safety topics.

NHTSA also conducts research on driver behavior and traffic safety, to develop the most efficient and effective means of bringing about safety improvements.

C.2 BACKGROUND

Advanced driver assistance systems (ADAS) often include alerts or warnings in the form of visual, auditory, and/or haptic stimuli. As ADAS technologies have become more prevalent in passenger vehicles, and include a wider range of features, automotive manufacturers have adopted a range of approaches toward providing alerts and warnings. Currently, however, there is limited publicly available evidence regarding best practices for alerts and warnings across the range of ADAS functions.

Crash data demonstrate that in-vehicle warnings can help reduce the likelihood of collisions.

The Insurance Institute for Highway Safety conducted a series of studies to determine the crash-reduction benefits of in-vehicle alert and warning systems on passenger vehicles, based on police-reported crash data. A study of crashes during 2010-2014 found that the presence of forward collision warning (FCW) on a vehicle reduced the likelihood of rear-end crashes by 27 percent, and rear-end crashes with injuries by 20 percent.1 A similar study using 2009-2015 crash data found that the presence of a blind spot monitoring system reduced the likelihood of lane-change crashes by 14 percent, and the likelihood of lane-

1 Cicchino, J. B. (2017). Effectiveness of forward collision warning and autonomous emergency braking systems in reducing front-to-rear crash rates. Accident Analysis & Prevention, 99, 142-152.

http://www.dot.gov/ http://www.dot.gov/ http://www.nhtsa.gov/Research change crashes involving injuries by 23 percent.2 Another study found that the presence of rear cross-traffic alert reduced backing crashes overall by 22 percent, and backing crashes involving two vehicles traveling in perpendicular directions by 32 percent.3

While there are demonstrated benefits of the presence of in-vehicle alert and warning systems, it is also important to consider how the characteristics of alerts and warnings influence outcomes such as conspicuity, comprehension, perceived urgency, driver response, and acceptance, and to determine the characteristics and criteria that are associated with safety benefits. Researchers have used psychophysical techniques to map signal parameters to subjective outcomes including perceived urgency,4 5 and have conducted experiments to assess the effects of different warning parameters on safety-related outcomes such as driver reaction time and crash avoidance.6 7 Guidance published by NHTSA summarizes some of the factors that are associated with perceived urgency of auditory signals, among other topics related to in-vehicle interface design.8 Driver acceptance of ADAS alerts and warnings can be important for safety, as poor acceptance can lead to drivers disabling technologies and thereby losing the potential safety benefits.9

Alerts and warnings in vehicles are most often visual, auditory, haptic, or some combination of these. Visual alerts are often presented in the instrument cluster, but may be presented in alternative locations, such as on or near side view mirrors, or in a head-up display. Haptic signals may be presented in various locations, including the driver’s seat, steering wheel, seat belt, or pedals. In some cases, vehicle control inputs initiated by the vehicle itself (e.g., brake pulses) may function as haptic signals.

There are numerous challenges in the design and assessment of alerts and warnings for ADAS. For example, drivers who are distracted, drowsy, or otherwise impaired may have different needs than attentive drivers regarding characteristics of alerts and warnings. As the number of ADAS technologies available on vehicles increases, it is also important to consider the relationships between the alerts and warnings for these different technologies.

2 Cicchino, J. B. (2018). Effects of blind spot monitoring systems on police-reported lane-change crashes. Traffic Injury Prevention, 19(6), 615-622.

3 Cicchino, J. B. (2019). Real-world effects of rear cross-traffic alert on police-reported backing crashes. Accident

Analysis & Prevention, 123, 350-355.

4 Hellier, E., & Edworthy, J. (1999). On using psychophysical techniques to achieve urgency mapping in auditory warnings. Applied Ergonomics, 30(2), 167-171.

5 Marshall, D. C., Lee, J. D., & Austria, P. A. (2007). Alerts for in-vehicle information systems: Annoyance, urgency, and appropriateness. Human Factors, 49(1), 145-157.

6 Lewis, B. A., Eisert, J. L., Baldwin, C. L., Singer, J., & Lerner, N. (2017). Urgency Coding Validations (Report No.

DOT HS 812 385). National Highway Traffic Safety Administration.

7 Marshall, D., Boyle, L. N., Wu, & Brown, T. (2018, December). Auditory alert characteristics impact on crash avoidance warning response (Report No. DOT HS 812 511). National Highway Traffic Safety Administration.

8 Campbell, J. L., Brown, J. L., Graving, J. S., Richard, C. M., Lichty, M. G., Bacon, L. P., … & Sanquist, T. (2018, August). Human factors design guidance for level 2 and level 3 automated driving concepts (Report No. DOT HS 812 555). National Highway Traffic Safety Administration.

9 Monticello, M. (2022, December 13). CR delivers advanced driver assistance systems guidelines to automakers.

https://www.consumerreports.org/cars/car-safety/cr-delivers-advanced-driver-assistance-systems-guidelines-a5361257868/ https://www.consumerreports.org/cars/car-safety/cr-delivers-advanced-driver-assistance-systems-guidelines-a5361257868/ https://www.consumerreports.org/cars/car-safety/cr-delivers-advanced-driver-assistance-systems-guidelines-a5361257868/

For example, recent model year vehicles can have FCW, lane departure warning (LDW), blind spot warning (BSW), rear cross-traffic alert, and other ADAS that may communicate a variety of alerts and warnings, among other possible technologies. When considering the characteristics of alerts and warnings, it is important to also consider the increasingly complex information space in which these signals exist, and how that might affect recommended signal characteristics. Finally, some warning systems are also paired with active safety systems that automatically take action to reduce risk or avoid a collision (e.g., automatic emergency braking [AEB] paired with FCW). While active safety systems are not the focus of this research, their presence might influence considerations for alerts and warnings and, as previously noted, the kinematic effects of active safety system interventions themselves may function as signals to the driver.

C.3 TASK OVERVIEW

The objective of this research is to identify and address critical knowledge gaps regarding the characteristics of in-vehicle alerts and warnings that are associated with detection, comprehension, perceived urgency, appropriate and timely responses, and user acceptance.

Research conducted under this Task Order shall include conducting a literature and practice review and identifying knowledge gaps (Phase 1); and conducting new research to address high-priority knowledge gaps (Phase 2). The findings of this research will help NHTSA to better understand the factors and signal criteria that are important for effective alerts and warnings used in ADAS, and to assess vehicles’ alert and warning interfaces. Specifically, the research shall address the following questions.

1. What are the characteristics of alerts and warnings used in current and near-term production vehicles? How are they similar or different across comparable ADAS features (e.g., different manufacturers’ FCW systems)?

2. What methods, measures, and metrics are appropriate for evaluating in-vehicle alerts and warnings?

3. What are best practices for in-vehicle visual, auditory, and haptic alert/warning elements

(alone or in combination) across the range of ADAS features, to promote detection, comprehension, appropriate and timely responses, and user acceptance? What criteria or ranges are recommended for various signal characteristics?

4. What implications do different driver states (e.g., attentive, distracted, drowsy) have on the appropriate characteristics of alerts/warnings?

5. What considerations should be given to the design of alerts and warnings not only in the context of a single ADAS function, but in an in-vehicle context that may include multiple alerts and warnings from various systems?

6. How do false or perceived nuisance alerts/warnings affect user trust and acceptance? What characteristics of alerts/warnings maximize user acceptance and minimize the likelihood that driver’s distrust, turn off, or disable ADAS functions?

Note that NHTSA is addressing interface characteristics to improve acceptance of lane departure warning and lane keeping assist systems in a separate, ongoing study. After task order award, the NHTSA COR (TO) will provide input to the contractor to avoid duplicative research efforts.

C.4 SPECIFIC REQUIREMENTS

C.4.1 Kickoff Meeting

The Contractor shall coordinate with the COR (TO) and begin preparatory activities for the Kickoff meeting immediately after Task Order award.

Within two (2) weeks ATOA, the Contractor shall meet with the COR (TO), Contracting Officer, and other interested NHTSA personnel by video conference (e.g., MS Teams). The purpose of this meeting is to: (1) discuss the administration of the Task Order; and (2) to discuss the project’s objectives, planned course of action, milestones and deliverables, the overall project schedule and time allocations for each task, responsibilities of subcontractors and individual staff members, anticipated resource allocations among the tasks, draft research, analysis, and risk plans, and any other issues the Contractor or Government identifies. The Contractor shall anticipate one (1) hour for this meeting. The Contractor shall provide draft slides to the COR (TO) at least two (2) business days prior to the meeting for review and comment. The finalized meeting slides shall be provided as a deliverable to the COR(TO).

C.4.2 Phase 1 Research Plan and Risk Management Plan

Within two (2) months ATOA, the Contractor shall submit a draft Phase 1 Research Plan and a draft Risk Management Plan to the NHTSA COR (TO). At minimum, the draft Phase 1 Research Plan shall include an outline of the literature and practice review (see Section C.4.4). The Principal Investigator (PI) shall track technical progress, schedule of milestones and deliverables, resource allocations, project expenses, and risk factors. Within two (2) weeks after receipt of the draft documents, the NHTSA COR (TO) will provide recommended revisions to the Contractor. Within two (2) weeks after receipt of the COR (TO) comments, the Contractor shall submit the revised documents. Within two (2) weeks after receipt of the revised documents, the COR (TO) will provide NHTSA approval of the final Phase 1 Research Plan and Risk Management Plan to the Contractor. The Contractor shall only begin the approved study upon receipt of the NHTSA approved final study plans.

C.4.3 Interim Status Briefings

The contractor shall provide five (5) interim status briefings to NHTSA following the completion of key milestones and deliverables. Interim briefings shall be provided within two (2) weeks after each of the following milestones and deliverables:

• Draft Literature and Practice Review Report submitted

• Draft Knowledge Gaps Report submitted

• Draft Phase 2 Research Plan submitted

• After PRA review is completed, but before Phase 2 data collection begins

• Phase 2 data analysis completed

Interim briefings will be held virtually, at NHTSA HQ in Washington, D.C., or at other locations mutually agreed upon between the COR (TO) and the Contractor. The Contractor shall provide draft briefing slides to the COR (TO) at least two (2) business days prior to the briefing for review and comment. The finalized briefing slides shall be provided as a deliverable to the COR(TO). In addition, the PI shall maintain ongoing timely and accurate communications with the COR (TO) on an as-needed basis. During the project, interactions with the COR (TO) may be accomplished through scheduled conversations, informal site visits, and formal technical exchange/program review briefings or other methods determined by the Contractor.

C.4.4 Literature and Practice Review

The Contractor shall conduct a Literature and Practice Review on the characteristics of in-vehicle alerts and warnings. The Literature Review should be a focused effort to address the research objectives and research questions stated in Section C.3 above. The primary focus should be on driving safety literature, but basic psychophysical research and alert and warning research from other domains may be included to address gaps and limitations in the driving safety literature.

The Practice Review should identify the key characteristics of alerts and warnings for ADAS in recent model year vehicles. This review should identify characteristics such as alert and warning modalities, visual displays and locations, auditory signal characteristics, and haptic signal characteristics and locations, across the various ADAS technologies present in these vehicles. The objective is not to develop a detailed census of all vehicle brands’ alert and warning interfaces; rather, it is to provide an overview of alert and warning characteristics that may influence safety, and to describe the range of approaches used by automotive manufacturers, and notable commonalities and differences in these approaches. The Practice Review should also identify current regulations and other requirements (e.g., NCAP) related to alert and warning characteristics.

Within four and a half (4.5) months ATOA, the Contractor shall submit a draft of the Literature and Practice Review to the NHTSA COR (TO). The COR (TO) will provide comments and recommended revisions within two (2) weeks after receipt of the draft.

Within two (2) weeks after receipt of the COR (TO) comments, the Contractor shall submit the revised documents.

C.4.5 Knowledge Gaps Analysis

Based on the findings of the Literature and Practice Review, the Contractor shall conduct a Knowledge Gaps Analysis to identify key gaps in knowledge and in the research literature regarding best practices for alert and warning characteristics of ADAS features. For each gap identified, the Contractor shall document the current state of knowledge, the potential safety benefits of addressing the gap, potential approaches to research the gap, and potential challenges in researching the gap and implementing findings in practice. In addition to addressing knowledge gaps related to alerts and warnings themselves, the Knowledge Gap Analysis should identify gaps related to methods and metrics used to evaluate characteristics of alerts and warnings in vehicles. While this Knowledge Gap Analysis should be used as a primary source to develop and justify the Contractor’s Phase 2 research approach, it should also serve as a stand-alone deliverable to help NHTSA identify and prioritize research needs.

Within six (6) months ATOA, the Contractor shall submit a draft of the Knowledge Gaps Analysis to the NHTSA COR (TO). The COR (TO) will provide comments and recommended revisions within two (2) weeks after receipt of the draft. Within two (2) weeks after receipt of the COR (TO) comments, the Contractor shall submit the revised documents.

C.4.6 Phase 2 Research Plan

Based on the findings of Literature and Practice Review and the Knowledge Gaps Analysis, as well as the stated objectives of this RFP, the Contractor shall develop a Phase 2 Research Plan to conduct new research to address knowledge gaps. The Contractor may include psychophysical or other laboratory methods as a component of the research approach;

however, methods with higher external validity (e.g., driving simulator, closed course, and/or on-road methods) are recommended to test or validate findings in a driving context.

In addition to directly assessing alerts and warnings themselves, the Contractor may propose to include methodological testing (e.g., comparing methods or metrics against one another to refine best practices for alert and warning evaluation) within the Research Plan. The Phase 2 Research Plan shall clearly and thoroughly describe the proposed research approach, including design, independent and dependent variables, apparatus, participants, procedures, and analyses. Research hypotheses should be stated, where appropriate.

Within nine (9) months ATOA, the Contractor shall submit a draft of the Phase 2 Research Plan to the NHTSA COR (TO). The COR (TO) will provide comments and recommended revisions within two (2) weeks after receipt of the draft. Within two (2) weeks after receipt of the COR (TO) comments, the Contractor shall submit the revised documents.

C.4.7 Paperwork Reduction Act (PRA) Documents and Review Process

The Contractor shall develop all study protocols and supporting materials (e.g., questionnaires) to be used in the data collection. Study procedures and materials must be approved by an Institutional Review Board prior to PRA review. The Contractor shall prepare the full set of materials (the “PRA package”) for PRA review, which include draft and final versions of the 60-day notice, 30-day notice, Part A, Part B, Supporting Statements, Privacy Threshold and Privacy Impact Assessments, and other materials as required. The team shall also participate in virtual meetings as needed with NHTSA, OST, and OMB as needed to secure PRA approval. Due to the nature of the PRA process, the timeline for multiple revisions to the PRA package will vary depending on the various review processes (and associated meetings) within NHTSA, OST, and OMB but is generally 1.5 to 2 years in total. The overall project plan shall include this process, knowing that research requiring more than 9 participants cannot begin until PRA approval is received. The research team shall plan for multiple rounds of edits, meetings, and drafts prior to final PRA approval by OIRA/OMB. Given the importance of PRA and risks (e.g., scheduling) that are associated with the process, knowledge of the PRA process is vital to successful completion of this task, and Offerors will be expected to demonstrate relevant knowledge.

Within twelve (12) months ATOA, the Contractor shall submit draft PRA documents to the NHTSA COR (TO). The COR (TO) will provide comments and recommended revisions within two (2) weeks after receipt of the draft. Within two (2) weeks after receipt of the COR (TO) comments, the Contractor shall submit the revised documents to the COR (TO).

C.4.8 Conduct Phase 2 Research

Following PRA approval, the Contractor shall conduct the Phase 2 research.

C.4.9 Final Report, Research Note, and Final Briefing

The Contractor shall submit a final report to the COR (TO), addressing the outcomes and lessons learned from the project. The final report shall include a unique report title that is suitable for publication and shall adhere to the writing guidelines as described in Section C of this RFP. The final report shall include an Executive Summary that provides a high-level overview of the report. The draft final report shall be reviewed and edited by a Technical Editor prior to submission to the COR.

The Contractor shall also submit a Research Note document that provides a brief overview of the study background/problem, objectives, methods, and key findings, followed by a discussion of the applicability and value of this research to practice. It is particularly important that the Research Note communicates the contribution of the research findings to practice. The Research Note shall be written so that it can be understood by a non-technical audience, should be brief (approximately 2-4 pages), and may include figures and tables.

Within forty-five (45) months ATOA, the Contractor shall submit the draft final report and draft Research Note to the COR (TO). Within three (3) weeks after receipt of the draft report and draft Research Note, the COR (TO) will provide feedback and recommended revisions (if required) to the Contractor. Within three (3) weeks after receipt of the COR (TO) comments, the Contractor shall submit the revised final report and Research Note for NHTSA agency review.

Within one (1) week after submission of the revised final report and Research Note for NHTSA agency review, the Contractor shall brief the COR (TO) and other interested NHTSA staff on the entire study in a Final Briefing to be held virtually. The Contractor shall provide draft briefing slides to the COR (TO) at least two (2) business days prior to the briefing for review and comment. The finalized briefing slides shall be provided as a deliverable to the COR(TO).

Within four (4) weeks after receipt of the agency review comments, the Contractor shall submit the final version of the final report and Research Note to the COR (TO), along with a comments resolution matrix showing how each agency review comment was addressed.

Note that the project schedule allows for up to twelve (12) months for the agency review process to be completed.

C.4.10 Monthly Progress Reports and Biweekly Meetings

Beginning thirty (30) days after Task Order award, the Contractor shall provide monthly project status reports to the COR (TO) by the 15th calendar day of the month, following the month being reported. The purpose of this report is to describe the status of the technical tasks, costs, and schedule. The monthly reports shall include the following information:

General Information:

− Contractor name − NHTSA contract number − Task Order number − Project title − Period of performance − Reporting period − Name and phone number for the NHTSA COR (TO) − Name and phone number for NHTSA COR (IDIQ) (if different than COR (TO)) − Name and phone number for Project Manager

Project Status:

− Accomplishments made during the reporting period.

− Preliminary or interim results, conclusions, trends, or other items of information that are of interest to NHTSA.

− Plans for accomplishments in the next reporting period.

− Problems that the Contractor has experienced in conducting their services; and specific action that NHTSA should undertake to alleviate a problem.

− Remaining milestones and deliverables, and their status.

− Monthly obligations and cumulative total to date.

Beginning one (1) month ATOA and recurring every two weeks thereafter, the COR (TO) and the Contractor shall meet for 30-minute virtual (e.g., MS Teams) biweekly meetings to discuss project status and progress. Individual meetings may be replaced by email updates by mutual agreement of the COR (TO) and the Contractor if there are no priority issues to discuss, if the biweekly meeting is replaced by another meeting (e.g., Interim Briefing or site visit), if there has been no notable project progress (e.g., during PRA review process), or if an unresolvable scheduling conflict exists.

(End of Section C)

SECTION F – DELIVERIES OR PERFORMANCE

The terms and conditions under Section V of the Base IDIQ will be observed under this Hypothetical Task Order, as applicable.

F.1 PERFORMANCE MILESTONES AND DELIVERABLES

The milestones and deliverables in the table below apply to this Hypothetical Task Order and are considered important to the successful completion of the Hypothetical Task Order.

All deliverables are subject to COR (TO) review and approval, and changes may be required. The schedule outlined in the table is an estimate. Offerors may modify this schedule with three exceptions: 1) the duration of time allotted for NHTSA review of deliverables shall not be shortened, 2) Offerors shall assume a duration of 24 months from submitting draft PRA documents (Item No. 19 in the table below) until final PRA approval (which occurs prior to Item No. 22 in the table), and 3) the total period of performance shall not exceed 60 months.

Item No.

Section No.

Deliverables/Milestones Due Date/After Task Order Award (ATOA)

1 C.4.1 Kickoff meeting (M), briefing slides (D) Within 2 weeks ATOA

2 C.4.2 Draft Phase 1 Research Plan and Risk Management Plan (D)

Within 2 months ATOA

3 C.4.2 COR (TO) provides feedback on draft Phase 1 Research Plan and Risk Management Plan (M)

Within 2 weeks after receipt of draft plans

4 C.4.2 Submit revised Phase 1 Research Plan and Risk Management Plan (D)

Within 2 weeks after receipt of COR (TO) comments

5 C.4.2 NHTSA approval of final Phase 1 Research Plan and Risk Management Plan (M)

Within 2 weeks after receipt of revised plans

6 C.4.4 Draft Literature and Practice Review (D) Within 4.5 months ATOA

7 C.4.4 COR (TO) provides feedback on draft Literature and Practice Review (M)

Within 2 weeks after receipt of draft

8 C.4.4 Final Literature and Practice Review (D) Within 2 weeks after receipt of COR (TO) comments

9 C.4.3 Interim Briefing 1: Literature and Practice Review (M), briefing slides (D)

Within 2 weeks after submitting draft Literature and Practice Review

10 C.4.5 Draft Knowledge Gap Analysis (D) Within 6 months ATOA

11 C.4.5 COR (TO) provides feedback on draft Knowledge Gap Analysis (M)

Within 2 weeks after submitting draft

12 C.4.5 Final Knowledge Gap Analysis (D) Within 2 weeks after receipt of COR (TO) comments

13 C.4.3 Interim Briefing 2: Knowledge Gap Analysis (M), briefing slides (D)

Within 2 weeks after submitting draft Knowledge Gap Analysis

14 C.4.6 Draft Phase 2 Research Plan (D) Within 9 months ATOA

15 C.4.6 COR (TO) provides feedback on draft Phase 2 Research Plan (M)

Within 2 weeks after submitting draft

16 C.4.6 Final Phase 2 Research Plan (D) Within 2 weeks after receipt of COR (TO) comments

17 C.4.3 Interim Briefing 3: Phase 2 Research Plan (M), briefing slides (D)

Within 2 weeks after submitting draft Phase 2 Research Plan

18 C.4.6 NHTSA approval of final Phase 2 Research Plan (M)

Within 2 weeks after receipt of revised plans

19 C.4.7 Draft PRA documents (D) Within 12 months ATOA

20 C.4.7 COR (TO) provides feedback on draft PRA documents (M)

Within 2 weeks after submitting draft

21 C.4.7 Submit revised PRA documents to COR

(TO) (D)

Within 2 weeks after receipt of COR (TO) comments

22 C.4.3 Interim Briefing 4: Preparation for Phase 2 Data Collection (M), briefing slides (D)

After PRA approval, but before Phase 2 data collection begins

23 C.4.8 Begin Phase 2 data collection (M) After Interim Briefing 4

24 C.4.3 Interim Briefing 5: Preliminary findings (M), briefing slides (D)

Within 2 weeks after data analysis is completed

25 C.4.9 Draft Final Report (D) Within 45 months ATOA

26 C.4.9 COR (TO) provides feedback on draft Final Report (M)

Within 3 weeks after submitting draft

27 C.4.9 Revised Final Report (D) Within 3 weeks after receipt of COR (TO) comments

28 C.4.9 Final Briefing (M), briefing slides (D)

Within 1 week after submission of revised Final Report

29 C.4.9 COR (TO) provides NHTSA agency review comments (M)

Within 59 months ATOA

30 C.4.9 Revised Final Report with NHTSA agency comments resolved (D)

Within 4 weeks after receiving NHTSA agency review comments

31 C.4.10 Monthly Progress Reports (D) 15th calendar day of each month following the month being reported, beginning 30 days ATOA

32 C.4.10 Biweekly Meetings (M) Every two weeks, beginning 1-month ATOA

(End of Section F)

SECTION L – INSTRUCTIONS, CONDITIONS, AND NOTICES TO THE

OFFERORS OR RESPONDENTS

L.1 HYPOTHETICAL TASK ORDER PROPOSAL DELIVERY

Questions: All questions concerning the Hypothetical Task Order shall be submitted in accordance with the procedures and deadlines specified for the RFP 6933JJ924R000031 (see SECTION L)

When/Where/How to Submit: Hypothetical Task Order Technical and Cost Proposal volumes shall be submitted as separate attachments to the Offeror’s base IDIQ Proposal, in accordance with the procedures and deadlines specified for the base IDIQ Proposal.

L.2 HYPOTHETICAL TASK ORDER PROPOSAL FORMAT

The Offeror shall submit/present the Hypothetical Task Order Technical and Cost Proposals as two (2) separate volumes:

VOLUME I – HYPOTHETICAL TASK ORDER TECHNICAL PROPOSAL

• The Hypothetical Task Order Technical Proposal shall be submitted as a stand-alone document (i.e., not as a section of the base IDIQ proposal). It shall contain a cover sheet that clearly identifies the Offeror’s name.

• The electronic submission(s) shall be submitted in a format compatible with Microsoft Office 2016.

• Hypothetical Task Order technical proposals shall be limited to 15 pages.

Cover page, table of contents, and references list, if provided, do not count toward the page limit.

• Technical proposals shall use Times New Roman 12-point font and 1-inch margins for body text. Alternative fonts and font sizes may be used sparingly for other report elements, such as table contents and headings, while still ensuring legibility and formatting consistency.

• No cost/price data shall be included in Volume I: Hypothetical Task Order Technical Proposal.

VOLUME II – HYPOTHETICAL TASK ORDER COST PROPOSAL

• The Hypothetical Task Order Cost Proposal shall be submitted as a stand-alone document (i.e., not as a section of the base IDIQ proposal). It shall contain a cover sheet that clearly identifies the Offeror’s name.

• The electronic submission(s) of the Hypothetical Task Order Cost Proposal shall be submitted in a format compatible with Microsoft Office 2016. The electronic submission shall allow the Government to interact with the data.

• Subcontractor cost/pricing shall be submitted in the same format and level of detail as the prime Contractor’s proposal.

• Supporting documentation for other direct costs (ODCs) (if applicable) shall be provided.

• An SF-33 form is not required as part of this submission.

• Page Limit: None

Note: NHTSA will not be liable for the failure to read and/or evaluate an HTO cost proposal that contains a virus and/or similar problem.

L.3 HYPOTHETICAL TASK ORDER TECHNICAL PROPOSAL INSTRUCTIONS

A written technical proposal is required for this Hypothetical Task Order. The Hypothetical Task Order technical proposal shall be bound separately from the business proposal. The Hypothetical Task Order Technical Proposal, Volume I, will be evaluated in accordance with Section M.2 of this Hypothetical Task Order RFP.

Price/cost information shall not be provided in the forwarding letter or in any part of the Hypothetical Task Order technical proposal.

L.3.1 PART I: Technical Approach and Program Management Plan

The Offeror’s technical proposal shall demonstrate the Offeror’s thorough understanding of the key technical issues involved in this Hypothetical Task Order;

the logistical requirements to successfully conduct the Hypothetical Task Order; and the Offeror’s strategy for completing the effort. In addition to demonstrating knowledge of the objectives and tasks described in Sections C.3 and C.4, the technical proposal shall demonstrate procedural knowledge of PRA by discussing necessary steps and identifying procedural bottlenecks. The Offeror’s technical proposal shall:

• Be clear, complete, reasonable, and feasible.

• Include work activities for both Phase 1 and Phase 2, including detailed information on the proposed project preparatory activities, implementation activities, and quality control procedures.

• Only propose to use resources (e.g., staff, facilities, equipment) that are available, or could realistically be acquired to conduct this research; costs to acquire resources shall be included in the cost proposal.

• Include a description of proposed schedule as well as all milestones and deliverables

• Demonstrate the Offeror’s approach and work plan for achieving the overall Task Order objectives

• Propose a realistic and achievable schedule for this Task Order in accordance with the instructions provided in Section F of this Hypothetical Task Order RFP, and include interim milestones and/or deliverables for PRA that demonstrate knowledge of the PRA process.

• Provide information indicating a plan for overall management of the project, including a strategy to ensure that work and deliverables are satisfactory, within budget, and on schedule.

• Provide a risk management plan including the identification of potential issues that may arise during the conduct of the project and strategies for handling risks and opportunities.

• Demonstrate how Offerors will manage and coordinate with subcontractor and consultants, if applicable.

• Describe the approach to the project staffing and organization, including designation of the Task Order project manager and other key staff and subcontractors.

• Describe the approach to managing the Task Order; meeting the task requirements and schedules; meeting performance measures; and communicating with the NHTSA COR (TO).

The program management plan shall demonstrate how Offerors will manage and coordinate with subcontractor and consultants, if used. The management plan shall describe the Offeror’s staffing and organizational approach to the project including designation of the Task Order project manager and other key staff and subcontractors.

This also includes the Offeror’s plan for managing the Task Order; meeting the task requirements and schedules; meeting performance measures; and communicating with the NHTSA COR (TO).

L.3.2 PART II: Qualifications of Project Personnel

The Hypothetical Task Order technical proposal shall demonstrate the qualifications of the Offeror’s proposed professional and technical personnel (including any proposed subcontractor personnel). This includes identifying staff names, proposed roles on the Hypothetical Task Order, and proposed level of effort. The Hypothetical Task Order technical proposal shall demonstrate the skill level and experience of the proposed staff in the following areas: Human factors literature and practice review, applied human factors experimental research, alert and warning interface development and/or evaluation, research design, experiment instrumentation, statistical analysis, and technical report writing and editing.

NHTSA will review the qualifications of the proposed staff to ensure a depth of technical background relevant to the effort, particularly of the proposed project manager.

Offerors shall not submit resumes as part of the Hypothetical Task Order proposal;

however, NHTSA will review resumes submitted as part of the base IDIQ proposal to assess the qualifications of proposed staff for this Hypothetical Task Order.

Resumes from the base IDIQ proposal will be used to evaluate staff with substantial roles proposed in the Hypothetical Task Order.

L.4 HYPOTHETICAL TASK ORDER COST PROPOSAL INSTRUCTIONS

This volume shall be separate from Volume I and shall contain all pricing information as described below.

L.4.1 Cost/Price Information Assumptions:

Offerors shall support their total cost for the objectives and task(s) outlined in the Statement of Objectives (SOO), Section C and the proposed Performance Work Statement, and shall contain the Offeror’s best terms from a cost standpoint with applicable discounts for consideration of the total effort. The cost proposal shall be prepared in accordance with the Offeror’s accounting system and practices and the format of the Cost Proposal template provided as Exhibit E.1. The Cost Proposal shall illustrate cost data by cost element (material, labor, indirect costs, etc.) used to develop the Loaded Fixed Rate. Further, the proposed cost shall be summarized by task.

L.4.2 Cost/Price Information Breakdown:

Cost Proposals shall include, at a minimum, all costs by task under Section C for the following categories:

1. Labor Rates/Categories: Provide labor rates/categories for all proposed staff.

Please include the number of hours needed to complete each task. Proposal should also include yearly escalation rates. Anticipated promotions for any personnel shall be included with the escalation calculation. The Government encourages Offerors to propose rates consistent with those proposed in the Cost Proposal volume for the base IDIQ proposal. The Government encourages Offerors to provide price discounts. The number of hours needed to complete each task shall be included.

2. Subcontracts/Consultants [as appropriate]: Offerors shall include subcontract information to be evaluated when proposing subcontractors/independent consultants (e.g., name, labor rates, etc.).

a. Name and address of the Subcontractor or consultant;

b. Identify the position and the portion of the work to be completed;

c. Extent to which adequate price competition was obtained, or justification in its absence. This information should be in a Negotiation Memorandum Format describing the Prime Contractors approach for identifying appropriate subcontractors and/or consultants inclusive of methods used to determine cost reasonableness.

d. Statement as to whether the proposed Subcontractor is on the list of Parties

Excluded from Federal Procurement and Non-Procurement Programs.

e. Cost proposal inclusive of any supporting documentation needed to substantiate proposed rates, hours, and overall costs. Note: Prime Contractors/Offerors are responsible for performing a cost reasonableness analysis on all its proposed Subcontractors and Consultants in accordance with FAR 15.404-3 and this analysis should be included in the Negotiation Memorandum described above in Section C.

3. Other Direct Costs (ODCs): This category includes materials, equipment, participant incentive payments, etc. Offerors shall consolidate ODCs into one line item. However, a separate schedule shall be provided to break down ODCs. The schedule shall provide a detailed itemization of each ODC proposed and shall also provide rationale for the need of each item.

Other Divisions: If other divisions, subsidiaries and parent/affiliated companies will perform work or furnish materials under this proposed contract, please provide the name, location, and inter-company pricing agreement for evaluation.

L.4.3 Estimated Level of Effort

For proposal purposes, the following is the Government’s estimated level of effort to complete the Task Order. The Government’s estimated range for this work is $650,000 to $750,000. This estimate is provided for informational purposes and is not intended as a floor or ceiling of the potential offer. These estimates are advisory and Offerors shall propose the appropriate hours and labor required to meet the requirements of the solicitation.

Labor Category Estimated No.

Hours Senior Technical Program Manager 250

Project Manager 600

Senior Researcher 800

Associate Researcher 1,000

Assistant Researcher 1,200

Senior Instrumentation Engineer 350

Instrumentation Engineer/Technician 350

Senior Statistician 100

Statistician 150

Technical Editor 100

Administrative Assistant 100

Total Estimated Hours 5,000

(End of Section L)

SECTION M – EVALUATION FACTORS FOR HYPOTHETICAL

TASK ORDER

M.1 GENERAL PROCEDURES FOR HYPOTHETICAL TASK ORDER EVALUATION

A. Evaluation Process: The evaluation process consists of two separate phases.

• PHASE ONE - The Offeror’s technical proposal will be evaluated for technical merit according to the criteria set forth in M.2, EVALUATION CRITERIA. If the Hypothetical Task Order technical proposal is determined to be “technically unacceptable,” the Offeror’s Hypothetical Task Order cost proposal will not be evaluated.

• PHASE TWO – If an Offeror’s Hypothetical Task Order technical proposal was not determined to be “technically unacceptable,” the Offeror’s Hypothetical Task Order cost proposal will be evaluated in accordance with Section M.3, PRICE/COST AND

OTHER BUSINESS FACTORS.

M.2 EVALUATION CRITERIA

The Technical Evaluation Committee (TEC) will evaluate and score each Hypothetical Task Order proposal in accordance with L.3 and the elements discussed below. All factors and sub-factors are listed in descending order of importance.

All technical evaluation factors, when combined, are significantly more important than price.

M.2.1 Technical Approach and Program Management Plan

Offerors will be evaluated on how well the proposal demonstrates understanding of the key technical issues involved in this Task Order; the logistical requirements to successfully conduct the Task Order; and the Offeror’s strategy for completing the effort. Offerors will also be evaluated on their technical proposal’s clarity, completeness, reasonableness, feasibility, and adherence to the instructions provided in this Hypothetical Task Order RFP.

Offerors will be evaluated on how well the proposal describes and justifies choices regarding the schedule, milestones, and deliverables. The technical proposal shall demonstrate the Offeror’s approach and work plan for achieving the overall Task Order objectives.

Offerors will be evaluated on how well the proposal provides detailed information on the proposed project preparatory activities, implementation activities, and quality control procedures. This includes discussion of:

• A plan for overall management of the project

• A strategy to ensure that work and deliverables are satisfactory, within budget, and on schedule

• A risk management plan including the identification of potential issues that may arise during the conduct of the project

• Description of the rationale for research methods and metrics proposed (e.g., indicators of cognitive workload)

• Strategies for handling risks and opportunities.

Offerors will be evaluated on how they will manage and coordinate with subcontractor and consultants, if used. The management plan shall describe the Offeror’s staffing and organizational approach to the project including designation of the Task Order project manager and other key staff and subcontractors. This also includes the Offeror’s plan for managing the Task Order; meeting the task requirements and schedules; meeting performance measures; and communicating with the NHTSA COR (TO).

Offerors will be evaluated on how well the proposal develops the Performance Work Statement (i.e., Offeror’s solution) for accomplishing the objectives of the solicitation.

Offerors shall develop the performance metrics and a measurement plan, and a quality assurance plan for accomplishing the objectives of the solicitation.

M.2.2 Qualifications of Project Personnel

Offerors will be evaluated on the extent to which:

• Proposed personnel have expertise and work-related experience in relevant areas, such as human factors literature and practice review, applied human factors experimental research, alert and warning interface development and/or evaluation, research design, experiment instrumentation, statistical analysis, technical report writing and editing, and any other areas relevant to the Offeror’s approach.

• The education, experience, and availability of key technical personnel is demonstrated, and proposed project roles and levels of effort are defined.

Subcontractors and consultants (if applicable) will be evaluated to the same standards as above.

M.2.3 PRICE/COST AND OTHER BUSINESS FACTORS

The completeness and accuracy of cost/price schedules will be reviewed. A determination will be made as to whether the Offeror has properly understood the cost/price instructions and provided a complete price for all the requirements.

The cost proposal evaluation will be based on the following factors:

• Cost analysis, price reasonableness, and comparison with the independent government cost estimate provided in Section L above.

• Whether the cost proposal is reasonable for the work to be performed, realistic, complete, reflects a clear understanding of the requirements, and is consistent with the technical proposal. The evaluation of the Offeror’s cost proposal on this factor will also affect the Offeror’s technical proposal score, to the extent that the cost proposal reflects the technical approach.

(End of Section M)