Sources Sought Automatically Generated Situational Judgment Tests Q and A.docx

Question: Do white papers that are based on or derived from serious games and simulation-based assessment research fall within the scope of the sources sought?

Answer: More important than any particular application context is that an offeror’s method generate items appropriate for a designated set of constructs. Per the sources sought notice, “Preference will be given to White Papers and proposals that focus on developing automatic SJT item generation approaches that are scalable and can be applied across contexts and domains. White Papers that permit flexibility or customization in the constructs that are targeted is also preferable” (p. 4).

Question: Is research described in the “context” sources sought notice (e.g., “Development of an Army-Specific Work Contexts and Situations Taxonomy”, W911NF-18-S-0005-FY23) related to this requirement?

Answer: The two sources sought notices are for concurrent but independent efforts. While one effort might ultimately inform the other, the two should not be viewed as smaller parts of a larger overall effort.

Question: Is the intention that the resulting prototype be usable by anyone without specialized skillsets?

Answer: The intention is to have a proof-of-concept that does not require specialized or high-level programming skills on the part of users – for example, a user should not need to edit java/C++ classes or compile programs. However, for a project at this stage of research, it would be appropriate to assume some level of computer skills on the part of users – simpler tasks like editing an XML/JSON file in a text editor or typing simple commands on a command-line, for example, would be acceptable.

Question: Page 3 in the solicitation mentions that this effort should consider the examinee (e.g., things like cognitive load, etc.). That does not seem to be a big part of what is described on page 4 in terms of what the white paper should address. How important is that piece?

Answer: Considerations of examinee reactions and perceptions of the items, cognitive load levels, and response processes underlying the SJT could be considered a subset of potentially relevant evaluation methods as described in 3.(b) on page 4.

Question: Are there specific models of interest to you, e.g., LSTM/RNN approaches?

Answer: No.

Question: Do you place more emphasis on the modeling piece (i.e., should we be working to improve state-of-the-art models) or on the design process of SJTs using modeling?

Answer: The elements of SJT design and natural language understanding/generation are both significant aspects of the requirement and are likely interdependent to a degree.

Question: Would any existing (archival) data be available for this work? Would we have access to participants/SMEs for data collections?

Answer: The Army can facilitate access to research participants through a few mechanisms (e.g., umbrella weeks, online survey platforms like Verint), although not all Army populations have ready access to computers. It should not be assumed that existing Army SJTs (tests or participant data) will be available for this research.

Question: Are there specific deliverables/end states that need to be produced at the end of the base and each option period?

Answer: No – which deliverables/end states make sense for particulars periods of performance will depend on the specific nature of each proposal received.

Question: Should we propose travel? If so, are travel estimates accounted for in the dollar values provided, or do those only include labor?

Answer: IPRs and regular meetings can be conducted virtually (e.g., using MS Teams), but data collection activities might potentially require travel, depending on the mode of data collection being proposed. The dollar limits mentioned in the sources sought notice are inclusive of all categories (labor, travel, etc.).