NCO Promotion SJT Additional QandA for BAA.docx

Q&A for BAA W911NF-23-S-0010 “Developing NCO Promotion Situational Judgment Test” Is the intent of this effort to develop a single assessment to be used across all junior NCO ranks and MOS? Yes.

Would an approach that initially addresses one rank/MOS and scales to additional ranks/MOS be acceptable? Yes.

Similarly, the Army Talent Attribute Framework lists many leadership KSAOs. Is the intent to develop a single assessment that measures all of them, or would a subset be acceptable as an initial effort? If there is a subset in mind, where should we focus? The job analysis report that is referenced identifies NCO ratings of KSAOs by importance.

What is meant by “high stakes?” Does it mean that if a Soldier does not pass, they do not get promoted, or does it refer to the challenging situations NCOs often find themselves in? A high-stakes assessment results in an outcome that is important to the test taker. The intended application, assuming the research is successful, is for use in the promotion process.

What are the requirements to host and deliver the adaptive test? What sort of platform do you plan to use to host and deliver it? Army Analytics Group supports ARI’s hosting and delivery needs. Programmers from AAG will attend the project kickoff meeting to answer questions and will be included in follow-on consultation to ensure the project plan and test development align with AAG requirements. In the past, many different types of files and computer languages have been acceptable, e.g., JSON, Python, .net although the versions and types have changed as security risks have been identified. It is critical that the files can be scanned for vulnerabilities and corrections may be required.

Given the vision for the adaptive assessment is a narrative that unfolds based on responses, what would be considered “innovative?” For example, would a simulation-based or virtual environment-based approach be acceptable? Or is the vision something text-based, potentially with multimedia (e.g., videos)? We welcome a variety of ideas for a computer adaptive SJT but note there may be limitations based on what the Army Analytics Group can support.

In addition to item difficulty, there are a variety of ways of adapting items. Are we limited to adapting based on increasing or decreasing difficulty? The goal is to develop a computer adaptive SJT that assesses Army leader attributes in the most efficient way possible.

Will ARI provide access to participants for multiple data collections to collect critical incidents, select SJT items, and perform validation studies? Yes, ARI will assist in gaining access to NCO participants when it is required and efforts have been made to minimize the burden for data collection support (e.g., build on prior research, capitalize on external SME consultants, when appropriate) Should the prototype be a comprehensive and fully realized version that’s near-ready for deployment or a more rudimentary, proof-of-concept version?

The computer-adaptive SJT deliverable is intended to represent significant development towards a computer-adaptive measure. It is not expected to yield a deployment-ready computer-adaptive test.

Should the computer adaptive prototype be fully validated within the scope of this project, or is that something that would occur outside the scope of the project?

Criterion-related validation of the computer-adaptive SJT is not a requirement for this project.