Sources Sought Notice_Officer Verbal Ability Test.pdf

Ver. 20DEC2016

W911NF-23-S-0010

SOURCES SOUGHT NOTICE

REQUEST FOR WHITE PAPERS

BAA TOPIC II A.2.a.ii: HOLISTIC PERSONNEL ASSESSMENT

“Enhancing Army Officer Selection: The Development and Validation of an Open-Ended Verbal Ability Test”

INTRODUCTION

Broad Agency Announcement (BAA) W911NF-23-S-0010 was publicized on SAM.gov and Grants.gov on 01 May 2023. This Sources Sought Notice calls for White Paper submissions in reference to the BAA Topic II A.2.a.ii: HOLISTIC PERSONNEL ASSESSMENT. The United States Army Research Institute for the Behavioral and Social Sciences (ARI) Broad Agency announcement W911NF-23-S-0010, issued under the provisions of paragraph 6.102(d)(2) of the Federal Acquisition Regulation, provides for the competitive selection of basic and applied research and that part of development not related to the development of a specific system or hardware procurement. A Proposal submitted in response to this BAA and selected for award is considered to be the result of full and open competition and in full compliance with the provisions of Public Law 98-369, “The Competition in Contracting Act of 1984," and subsequent amendments. Funding of research and development (R&D) within ARI areas of interest will be determined by funding constraints and priorities set during each budget cycle. Any award related to the submission of a White Paper and subsequent Proposal requested by this Notice is subject to funds availability and priorities. ARI may choose not to select any new award due to unavailability of funds or other factors.

The sequence of steps leading to an award is:

1) Request for White Paper initiated by ARI through this Sources Sought Notice.

2) Optional but highly encouraged submission of a timely White Paper no more than six pages in length (one page is the cover page) to the POC for the U.S. Army Contracting Command, megan.a.deluca.civ@army.mil, and copy furnish (CC) the ARI Technical Points of Contact (TPOCs), david.martinez689.civ@army.mil and katherine.e.ciarlante.civ@army.mil.

3) ARI will provide written or telephonic feedback for whitepapers submitted and will provide a response with either “encouraged to submit a proposal” or “not encouraged to submit a proposal” as per established criteria presented in Part III.

4) If the White Paper merits it, a request of a formal proposal initiated by ARI.

5) Submission of a timely, formal proposal.

6) Evaluation of the formal proposal as per established criteria presented in Part III.

7) Award for selected proposal based on availability of funds or other factors.

This sequence allows earliest determination of the potential for funding and minimizes the labor and cost associated with submission of a full proposal that has minimal probability of being selected for funding.

mailto:david.martinez689.civ@army.mil

BAA W911NF-23-S-0010 allows several potential instrument types (e.g., contract, grant, cooperative agreement) to result from a successful proposal. For this Notice, the intention of the Government is to award a contract.

THOSE SUBMITTING A WHITE PAPER/PROPOSAL ARE CAUTIONED THAT ONLY A

GOVERNMENT CONTRACTING OR GRANTS OFFICER CAN OBLIGATE THE GOVERNMENT

THROUGH AWARD OF A LEGAL INSTRUMENT INVOLVING EXPENDITURE OF

GOVERNMENT FUNDS.

This Sources Sought Notice for a Requested White Paper consists of seven parts as follows:

• Part I: Research and Development Interests of the Requested White Paper

• Part II: Preparation and Submission

• Part III: Evaluation Criteria

• Part IV: Feedback

• Part V: Deadlines

• Part VI: Inquiries

• Part VII: References

ACC (APG) RTP Agency Point of Contact:

The POC for the US Army Contracting Command (Aberdeen Proving Ground) Research Triangle Park Division is: Ms. Megan Deluca, (919) 541-4682, megan.a.deluca.civ@army.mil.

ARI Agency Points of Contact:

The ARI POCs for technical matters for this White Paper topic are: Dr. David Martinez, david.martinez689.civ@army.mil and Dr. Katherine Ciarlante, katherine.e.ciarlante.civ@army.mil.

I. RESEARCH AND DEVELOPMENT INTERESTS OF THE REQUESTED WHITE PAPER:

The United States Army Research Institute for the Behavioral and Social Sciences is the Army’s lead agency for the conduct of research, development, and analyses for Army readiness and performance via research advances and applications of the behavioral and social sciences that address personnel, organization, training, and leader development issues. ARI contracts with educational institutions, non-profit organizations, and private industry for research and development (R&D) in different areas, including the areas specifically identified in Section II - A W911NF-23-S-0010. Efforts funded under this White Paper request will only include Applied Research and/or Advanced Technology Development.

Applied Research provides a systematic expansion and application of knowledge to design and develop useful strategies, techniques, methods, tests, or measures that provide the means to meet a recognized and specific Army need. Applied Research precedes system specific technology investigations or development, but it should have a high potential to transition into the Advanced Technology Development (ATD) Program.

The ARI ATD Program includes the development of technologies, components, or prototypes that can be tested in field experiments and/or simulated environments. Projects in this category have a direct relevance to identified military needs. These projects should demonstrate the general military utility or cost reduction potential of technology in the areas of personnel selection, assignment, and retention;

training strategies and techniques; leader education and development; performance measurement; and team and inter-organizational mission effectiveness. These projects should be focused on a more direct operational benefit and if successful, the technology should be available for transition.

For this Notice, the intention of the Government is to fund Applied Research.

WHITE PAPER TOPIC: Enhancing Army Officer Selection: The Development and Validation of an Open-Ended Verbal Ability Test

To ensure the Army has the right officers in the right place at the right time, it needs validated, objective, and standardized selection assessments specifically designed for officers. Currently, the Army lacks such tests and instead relies on information from sources such as interviews, college transcripts, and self-report measures. Ongoing efforts at ARI are developing officer assessments (e.g., Byrd et al., 2023), including multiple-choice assessments of fluid abilities. The current effort aims to develop an open-ended assessment of crystallized verbal abilities (see Carroll, 1993).

Prior research has identified several constructs that are relevant to officer performance (Boyle et al., 2022;

Melvin et al., 2021; Royston & Lin, 2022). These include, but are not limited to, oral communication, written communication, and reading comprehension. More generally, research suggests that verbal abilities are among the best predictors of performance across numerous domains (Postlethwaite, 2011).

Verbal abilities are frequently assessed using closed-ended (e.g., multiple-choice) tests; however, open-ended tests are used, often for complementary information. Closed-ended tests are easy to score, which facilitates the administration of a greater number of items and a broader assessment of a construct. Open-ended tests have, historically, required more resources to score and are therefore used sparingly to assess depth of knowledge or skills and abilities that are not amenable to closed-ended testing (Ackerman & Smith, 1988; Campbell, 1999; Ku, 2009; see also Rodriguez, 2003).

Large language models (LLMs) now make open-ended tests more practicable. Research suggests that LLMs can be used to score open-ended assessments with performance levels comparable to—or, in some cases, exceeding—human scorers, at a fraction of the time and financial cost (Impey et al., 2025; Lee et al., 2024; Saretzki & Benedek, 2026).

The current project will develop a cost-efficient, open-ended assessment of verbal abilities to support the selection of aspiring officers. Because the Officer Verbal Ability Test may eventually be fielded alongside other assessments, it should take no more than 30 to 45 minutes to administer.

To accomplish this objective, the offeror shall:

• Identify specific verbal abilities to measure in the Officer Verbal Ability Test.

• Produce a fixed-form version of the Officer Verbal Ability Test.

• Develop one or more LLM scoring agents that operate on government systems.

• Provide initial evidence of the reliability and validity of the test (and accompanying scoring agent) and any subsections.

o As one source of validity, the offeror must compare human-derived scores with LLM-derived scores.

A successful White Paper will describe:

• The method(s) that will be used to identify specific verbal abilities to be included in the Officer Verbal Ability Test.

• The process and method that will be taken to produce a fixed-form version of the test.

• The methods that will be used to develop one or more valid scoring agents.

• The validation approach and methodology (e.g., concurrent, alternative validity generalization).

Offerors should keep the following in mind:

• The assessment is to be used to select aspiring officers from a population that may have no knowledge of the Army or prior experience in leadership.

• The assessment should be occupation-agnostic (e.g., not specifically to select infantry officers).

• ARI will work to obtain performance data (e.g., course grades) but cannot guarantee it. Thus, the offeror should offer alternative sources of validity evidence (e.g., for convergent validity).

• The LLM(s) and any agent applications used to score responses must be approved by the

Department of War and function on Department of War systems. ARI currently maintains access to various LLMs, though the available models and versions are subject to change. Thus, offerors should propose a method that could work across models.

• The assessment and related materials will be owned by the Army at the completion of the project.

The offeror should include how all code and documentation will be collated for delivery.

• ARI will work to obtain a suitable sample. The total sample size for development in this effort is unlikely to exceed 300 individuals. Offerors may also identify their own sample.

• ARI will work to ensure computer access for participants; however, it cannot guarantee that all or any of the participants will have computers to complete the assessment. Offerors should be able to securely digitize responses (e.g., manually) for scoring.

• While the test itself should be 30-45 minutes or less, data collection sessions can be up to two hours. Importantly, the longer the session, the more difficult it is to achieve an adequate sample size.

The award will be a 24-month period of performance (Base, 12 months, not to exceed $400,000;

Option 1, 12 months, not to exceed $315,000) with a total budget not to exceed $715,000.

The Army Contracting Command- Aberdeen Proving Ground, RTP Division has the authority to award a variety of instruments, to include contracts, grants, and cooperative agreements. The ACC (APG) RTP Division reserves the right to use the type of instrument most appropriate for the effort proposed (contract, cooperative, or grant).

II. PREPARATION AND SUBMISSION OF A WHITE PAPER:

Preparation of White Paper

A White Paper should focus on describing details of the proposed research for both the base and if applicable, option (s) approach, including how it is innovative and how it could substantially advance the state of the science. Army relevance and potential impact should also be described, as well as an estimate of total cost for both the base and option approach. White Papers should present the effort in sufficient detail to allow evaluation of the concept's technical merit and its potential contributions to the Army mission.

A White Paper must be limited to six (6) pages (page one is the cover page) and an addendum in which the Applicant must include a biographical sketch (up to 300 words per individual) of all key personnel (i.e., Principal Investigators and Co-Principal Investigators) who will perform the research, highlighting their qualifications and experience as discussed below. All files and forms must be compiled into a single PDF file or MS Word document before submitting. Reviewers will be advised that they are only to review the cover page and up to five pages plus the addendum. Any pages submitted in excess of the six (6) page limit will not be reviewed or evaluated.

TECHNICAL INFORMATION FOR WHITE PAPERS:

1. Technical Approach: A detailed discussion of the effort's scientific research objectives, approach, relationship to similar research, level of effort, and estimated total cost; include the nature and extent of the anticipated results, and if known, the manner in which the work will contribute to the accomplishment of the Army's mission related to this request and how this would be demonstrated.

2. Requests for Government Support: The type of support, if any that the Applicant requests of the Government (such as facilities, equipment, demonstration sites, test ranges, software, personnel or materials) shall be identified as Government Furnished Equipment (GFE), Government Furnished Information (GFI), Government Furnished Property (GFP), or Government Furnished Data (GFD). The Applicant shall indicate any Government coordination that may be required for obtaining equipment or facilities necessary to perform any simulations or exercises that would demonstrate the proposed capability.

3. The cost portion of the whitepaper shall contain a brief cost estimate including research hours, burden, material costs, travel, etc.

4. Key Personnel Biographical Information: As an addendum to the White Paper, the Applicant must include a biographical sketch (up to 300 words per individual) of all key personnel (i.e., Principal Investigators and Co-Principal Investigators) who will perform the research, highlighting their qualifications and experience.

RESTRICTIVE MARKINGS ON WHITE PAPERS:

1. The Applicant must identify any proprietary data the Applicant intends to be used only by the Government. The Applicant must also identify any technical data or computer software contained in the White Paper that is to be treated by the Government as limited rights or restricted rights respectively. In the absence of such identification, the Government will assume to have unlimited rights to all technical data or computer software presented in the White Paper. Records or data bearing a restrictive legend may be included in the White Paper but must be clearly marked. It is the intent of the Army to treat all White Papers as procurement sensitive information before the award and to disclose their contents only to Government employees or designated support contractors for the purpose of procurement related activities only. Classified, sensitive, or critical information on technologies should not be included in a White Paper.

2. An Applicant is cautioned that portions of White Papers may be subject to release under terms of the Freedom of Information Act, 5 U.S.C. 552, as amended.

Submission of White Paper

White Papers must be submitted by e-mail to the POC for the U.S. Army Contracting Command, megan.a.deluca.civ@army.mil, and cc’d to the ARI Points of Contact (POCs), david.martinez689.civ@army.mil and katherine.e.ciarlante.civ@army.mil, in electronic MS Word document format or PDF file format. Cite “ARI BAA W911NF-23-S-0010, Enhancing Army Officer Selection: The Development and Validation of an Open-Ended Verbal Ability Test” in the e-mail subject line.

III. EVALUATION CRITERIA:

White Papers and full Proposals received in response to this request will be evaluated by the ARI designated points of contact identified in this request using the following factors/criteria:

1. Scientific and Technical Merit- The overall scientific and/or technical merits of the proposed research.

2. Potential Contribution- The potential contributions to ARI’s mission.

3. Qualifications/Capabilities- Proposed principal investigator and key personnel qualifications, capabilities, related experience, and techniques and also institutional resources and facilities.

4. Cost- Addresses the level of support requested. Will be considered for realism, affordability, and appropriateness, and may be grounds for rejection independent of evaluation on other factors.

The request for a proposal will be made based on the overall evaluation of a White Paper using the four criteria listed above. The overall scientific and/or technical merit of the proposed approach will be weighted more strongly than all of the other non-cost factors combined. All evaluation factors/criteria other than cost, when combined, are significantly more important than cost or price. A request for proposal may not necessarily be made to the lowest proposed price. During the evaluation of White Papers, ARI’s POCs for technical matters may request a telecon with an Applicant, but telecons are not guaranteed nor required for competition and award purposes. ARI’s POCs for technical matters reserves the right to evaluate a White Paper and request a proposal without discussions. The Applicant’s initial submission should contain the Applicant’s best terms from a technical and price standpoint. Once a full proposal has been requested, all communications must go through the POCs for the U.S. Army Contracting Command.

If the White Paper evaluation results in the request and submission of a full proposal, the proposal will be evaluated by a panel of scientific peers using the same factors/criteria as those listed above under Evaluation Criteria. A request for a full proposal does not guarantee an award. The decision to award will be based on feedback from the panel, considerations presented by ARI’s POCs for technical matters identified in this document, and other factors like budgetary constraints. ARI may choose not to select any award due to unavailability of funds or other factors.

IV. FEEDBACK:

Written or telephonic feedback will be provided to the Applicant regarding the White Paper’s scientific merit and potential contributions to the ARI’s mission. If the Government decides to request a full proposal, a written request will be sent to the Applicant. The Written Request will, at a minimum, invite a full proposal. The request may also include feedback intended to improve the proposal’s potential for award.

V. DEADLINES:

Electronic versions of the White Paper must be received by the POC for the U.S. Army Contracting Command and the ARI POCs, with e-mail subject line “ARI BAA W911NF-23-S-0010, Enhancing Army Officer Selection: The Development and Validation of an Open-Ended Verbal Ability Test” by e-mail no later than 5:00 PM Eastern Daylight Time on 3 July 2026. Any extension to the White Paper submission deadline will be posted to SAM.gov and Grants.gov as an amendment to this Notice. Note that a White Paper received within the deadline will be evaluated. For a White Paper received by the deadline, the offeror will receive feedback by end of day 10 July 2026 with notice of if they are “encouraged to submit a proposal” or “not encouraged to submit a proposal.”

Offerors may also choose to submit a Proposal without submitting a white paper as outlined in the BAA, W911NF-23-S-0010 (D.2.Section 2). The deadline for full Proposal submission is 5:00 PM Eastern Daylight Time on 4 August 2026. The proposal due date will be the same for offerors who submit a white paper and offerors who submit their proposals without white papers.

If edits are required for a Proposal, the offeror will be notified by end of day 13 August 2026, and edits to the Proposal will be due by 1 September 2026.

Please refer to the BAA, W911NF-23-S-0010 for instructions for the submission of a full Proposal.

An Applicant is responsible for submitting an electronic White Paper or full proposal to be received and accepted at the Government site indicated in this Notice no later than the date and time specified above.

When sending electronic files, an Applicant shall account for potential delays in file transfer from the originator’s computer to the Government website/computer server. An Applicant is encouraged to submit their response early (48 hours if possible) to avoid potential file transfer delays due to high demand or problems encountered during submission.

An Applicant should receive confirmation of delivery at the Government site, not just successful relay from the Applicant’s system. Acceptable evidence to establish the time of receipt at the Government site includes documentary and electronic evidence of receipt maintained by the Government site. All submissions shall be submitted before the deadline identified above to be considered – no exceptions.

If an emergency or unanticipated event interrupts normal Government processes so that a White Paper or full proposal cannot be received at the site designated for receipt by the date and time specified above, then the date and time specified for receipt will be deemed to be extended to the same day and time specified in this Notice on the first work day on which normal Government processes resume.

An Applicant agrees to hold the terms of their White Paper and any subsequent proposal valid for 180 calendar days from the date of submission.

VI. INQUIRIES:

ACC (APG) RTP Agency Point of Contact (Contractual Questions)

The POC for the US Army Contracting Command (Aberdeen Proving Ground) Research Triangle Park Division is: Ms. Megan Deluca, (919) 541-4682, megan.a.deluca.civ@army.mil.

ARI Agency Points of Contact (Technical Questions)

The ARI POCs for technical matters for this White Paper topic are: Dr. David Martinez, david.martinez689.civ@army.mil and Dr. Katherine Ciarlante, katherine.e.ciarlante.civ@army.mil.

VII. REFERENCES:

Ackerman, T. A., & Smith, P. L. (1988). A comparison of the information provided by essay, multiple-choice, and free-response writing tests. Applied Psychological Measurement, 12(2), 117–128. https://doi.org/10.1177/014662168801200202

Boyle, B. E., Darbor, K. E., Cato, C. R., Kammerer, W., & Thurston, J. (2022). Survey of Talent Requirements: Talents, Skills, and Abilities of Army Officers by Branch and Functional Area (Technical Report # 1418). U.S. Army Research Institute for the Behavioral and Social Sciences. (DTIC No. AD1188586)

Byrd, C., Glorioso, M., Adis, C., Engelsted, L., Pfleger, J., & Wisecarver, M. (2023).

Development of Three Cognitive Assessments for Officer Classification (ARI Research Note # 2023-04). U.S. Army Research Institute for the Behavioral and Social Sciences.

(DTIC No. AD1190661)

Campbell, J. R. 1999. Cognitive processes elicited by multiple-choice and constructed response questions on an assessment of reading comprehension (Unpublished doctoral dissertation). Philadelphia: Temple University.

Carroll, J. (1993). Human cognitive abilities: A survey of factor-analytic studies. Cambridge University Press. https://doi.org/ 10.1017/CBO9780511571312

Impey, C., Wenger, M., Garuda, N., Golchin, S., & Stamer, S. (2025). Using large language models for automated grading of student writing about science. International Journal of Artificial Intelligence in Education, 35(4), 1825-1859. https://doi.org/10.1007/s40593- 024-00453-7

Ku, K. Y. (2009). Assessing students’ critical thinking performance: Urging for measurements using multi-response format. Thinking Skills and Creativity, 4(1), 70-76.

https://doi.org/10.1016/j.tsc.2009.02.001

Lee, G., Latif, E., Wu, X., Liu, N., & Zhai, X. (2024). Applying large language models and chain-of-thought for automatic scoring. Computers and Education: Artificial Intelligence, 6, 100213. https://doi.org/10.1016/j.caeai.2024.100213

Melvin, E.S., Legree, P., Young, M.C., & Kilcullen, R.N. (2021). Optimizing Officer Classification: Selection of Predictor Constructs and Measures. (ARI Research Note #

2021-03). U.S. Army Research Institute for the Behavioral and Social Sciences. (DTIC No. AD1147775)

Postlethwaite, B. E. (2011). Fluid ability, crystallized ability, and performance across multiple domains: A meta-analysis [Unpublished doctoral dissertation]. The University of Iowa.

Rodriguez, M.C. (2003). Construct equivalence of multiple-choice and constructed-response items: A random effects synthesis of correlations. Journal of Educational Measurement, 40,163-184. https://doi.org/10.1111/j.1745-3984.2003.tb01102.x

Royston, R. P. & Lin, N. (2022). Army-wide Job Analysis for Officers: Branch and Functional Area-specific Knowledge, Skills, and Behaviors Requirements from the Army Talent Attribute Framework Volume I: Main Report (ARI Technical Report # 1425). U.S. Army Research Institute for the Behavioral and Social Sciences. (DTIC No. AD1194370)

Saretzki, J., & Benedek, M. (2026). Investigating the validity evidence of automated scoring methods for divergent thinking assessments. Creativity Research Journal, 1-17.

https://doi.org/10.1080/10400419.2025.2607669

The ARI POCs for technical matters for this White Paper topic are: Dr. David Martinez, david.martinez689.civ@army.mil and Dr. Katherine Ciarlante, katherine.e.ciarlante.civ@army.mil.