Sources Sought Notice.docx

Sources Sought Notice: National Registry Test Development and Analysis

Solicitation Number:

TBD

Notice Type:

Sources Sought Notice

NAICS Code 611710 – Educational Support Services

Title:

National Registry of Certified Medical Examiners Test Development and Analysis

Set-Aside:

Small Business

Contract Type:

Firm Fixed Price

General Information:

This Sources Sought Notice (Notice) is for market research and planning purposes only and shall not be considered to be an obligation by the Government to acquire any products or services. This notice is to determine the availability of potential small businesses (e.g., 8(a), service-disabled veteran owned small business, HUBZone small business, small disadvantaged business, veteran-owned small business, and women-owned small business) that can provide the required work outlined in the notice. This is not a solicitation announcement for proposals and no contract will be awarded from this Notice. No reimbursement will be made for any costs associated with providing information in response to this Notice. Respondents will not be notified of the results of this evaluation. Capability statements will not be returned and will not be accepted after the due date. This synopsis does not constitute a Request for Proposal (RFP).

Synopsis:

Background The Federal Motor Carrier Safety Administration’s (FMCSA’s) National Registry of Certified Medical Examiners (National Registry) was created to ensure that medical examiners can effectively determine if interstate CMV drivers meet FMCSA’s physical qualification standards.

FMCSA contractors have provided technical support services in the development and implementation of the National Registry Medical Examiner Certification test since 2005. The development included a Role Delineation Study from which a Detailed Content Outline (DCO) was produced to serve as a blueprint for the development of test forms. Working with panels of ME experts representative of the four professions, Physicians [Medical Doctors (M.D.) and/or Doctors of Osteopathic Medicine (D.O.)], Physician Assistants (PA), Advanced Practice Nurses (APRN), and Chiropractors (D.C.), which constitute the majority of MEs, FMCSA contractors developed and piloted the initial test forms and determined the cut scores for the test forms.

Test form sets were developed so that each test would have a Part A and a Part B, meaning each test form set contained two test forms (i.e., form 01A and 01B). Each test form set contains 100 scored test items, presented in a different order on Part A from Part B, and 40 unscored (pre-test) items (20 on Part A and 20 on Part B) for the purpose of vetting test items. FMCSA contractors continued to work with the ME experts and held quarterly meetings where they would revise and draft items for the test bank ensuring that each are strongly linked to the intended topic in the DCO. They analyzed test results for two different test form sets that included up to 80 pre-test items and 200 scored test items per year. The contractor also analyzed test results from the test forms annually, and revised, accepted, or discarded unscored test items (pre-test), based on psychometric analysis of test results from a large number of ME candidates and review by a panel of ME experts. The contractors and panel members also reviewed scored test items and changed them for use in new test forms.

Standard construction guidelines were followed to ensure each question yields a fair assessment. By adhering to design specifications as each version of the test is assembled, a substantially similar challenge is presented to each candidate. The goal about challenging each candidate equally is reinforced by the fact that there are statistical targets associated with items that are met for each version of the test. Adherence to statistical targets while satisfying design specifications for content domains and item complexities, are critical to achieving the goal of creating an equal challenge for each candidate. One of FMCSA’s goals are to incrementally strengthen the psychometric properties of items in the bank over time. In short, better items are retained in the bank while weaker items are revised or discarded while at the same time new items are added, some of which will perform sufficiently well. Hence, incremental improvement will occur.

Currently, there are a total of 8 test form sets, each containing 2 test forms, a Part A and Part B (e.g., form 01A and 01B, form 02A and 02B, etc.) for a total of 16 test forms. Each of the 8 test form sets contain 100 different scored test questions so that test forms 01A and 01B have 100 different scored test questions from test forms 02A and 02B etc. In each test set, Part A and Part B contain 20 different unscored pre-test questions but the same 100 scored test questions that are presented in a different order on Part A versus Part B. As discussed above, each test form contains 100 scored test items and 20 unscored (pre-test) items for the purpose of vetting test items. Since the initial implementation of the National Registry and with FMCSA’s hiring of a Chief Medical Officer (CMO), FMCSA has changed the approach in the way in which the DCO and test questions will be updated. FMCSA will no longer use a panel of ME experts, instead the CMO with input from FMCSA’s Medical Review Board (MRB) will work together to update the test questions based on the updated DCO. The test item bank of 2,400 questions was reviewed by FMCSA and many outdated and duplicate items were discarded, leaving 450 questions to be used in the development of new tests. In addition, FMCSA wrote 100 new test questions that were reviewed by the MRB in April 2020, and were determined to be either approved, edited, or discarded. Those questions that were approved are being added to the current item bank of 450 questions to produce new tests to be put into circulation. FMCSA’s CMO is updating the DCO in accordance with changes in the FMCSRs and current clinical best practices pertaining to physical qualifications and driver examinations. After updating the DCO, FMCSA will continue to consult with its MRB to draft and edit future questions to be added to the test item bank. FMCSA will provide the contractor with updates to the current test item bank as they occur.

Scope of Work The Contractor shall provide technical services for FMCSA’s Medical Programs Division to support the National Registry certification test implementation with test development, analysis, reporting, and consultation. The services shall be provided at the Contractor’s site. FMCSA will provide the Contractor with the current revised test item bank, test forms, the updated DCO and analytical data developed under a previous contract. All information that FMCSA shares with the contractor and all information generated by the contractor under this contract are the sole property of FMCSA.

The Contractor shall maintain the test item bank in accordance with Federal security requirements identified by FMCSA including limited access controls. Working with FMCSA, the Contractor shall add, revise, and delete test items as needed. In accordance with current psychometric methodology customized for certified healthcare professionals, the Contractor shall organize and evaluate the test questions provided by FMCSA at the start of the contract to determine, and subsequently provide, the max number of test form sets they can develop during the base year of the contract with a goal of three or more test form sets. In addition, the Contractor shall provide two or more test form sets during each option year of the contract. Each of the test forms will have different questions and a Part A and a Part B, so that there are different pre-test items for vetting on each test form. The Contractor shall deliver, to FMCSA or other contractor as identified by FMCSA, once per contract year, via secure overnight delivery on an encrypted CD, all test forms in XML format, and cut scores and test delivery instructions in PDF format, along with an inventory list describing the type and size of each file, and instructions for the data’s use and manipulation (e.g., how to open the file, how to sort the data).

The Contractor shall be responsible for having the data storage and analysis tools necessary to accept and evaluate test results from FMCSA or another contractor as identified by FMCSA. The test results can be up to 50,000 records in a single transfer to the Contractor. Although the test results provided to the Contractor will not contain any Personally Identifiable Information, the Contractor shall follow all FMCSA security requirements regarding the storage of data and the transfer of test forms and test results between the Contractor and FMCSA or other contractor as identified by FMCSA.

The Contractor shall prepare an annual statistical report on test performance aggregated by professional type (including Physicians, Physician Assistants, Advanced Practice Nurses, and Chiropractors). This report will include the analysis of question performance and types of items that various healthcare professionals (including Physicians, Physician Assistants, Advanced Practice Nurses, and Chiropractors) answered correctly and incorrectly. This report will include the analysis of test results to show areas of weakness in ME candidate training, overall and by healthcare profession. The report format will include a title page, table of contents, list of tables with titles, list of figures, executive summary, introduction, methodology, body, and conclusion.

At the end of the final contract year, the Contractor shall provide and implement a transition plan for the transfer of all information received from FMCSA or developed under this contract to FMCSA. The Contractor shall submit a final report for the contract to FMCSA and dispose of all information developed under the Contract or received from FMCSA. FMCSA will conduct a site visit to verify the disposal.

Instructions:

Interested small business contractors are invited to submit a technical capability statement which clearly demonstrates their ability to meet the governments requirement as summarized above. Contractors should include the following information in their capability statement:

1. Company name, DUNS number, company mailing address, point of contact information (One person's name, telephone number and e-mail address).

1. Company's applicable socioeconomic status and available contract ordering vehicles (include contract numbers, if GSA Schedule, also include Special Identification Number (SIN)), if available.

Responses should be no longer than ten (10) pages, excluding cover page, single-spaced, with font no smaller than 12-point. Responses must be submitted via email to Elizabeth.Johnson@dot.gov with a carbon copy to Deila.Johnson@dot.gov. Reference in the subject line of the email: National Registry Test Development and Analysis. Responses must be received by 12:00 p.m. ET on July 7, 2020. .