C3 SSN Additional Questions and Answers.docx

Question: Is this primarily a validation effort?

Answer: The purpose is to update the construct definitions, how they are measured, and then conduct the necessary construct validation to support the changes.

Question: Would adding new constructs be considered?

Answer: The existing constructs were identified by SMEs as consistent and important across related jobs, and cross-referenced with similar occupations on O*NET. The primary focus of this research is updating definitions, changing how they are measured, and demonstrating construct validity.

Question: Should the proposal take the form of a grant or cooperative agreement?

Answer: As stated in the Sources Sought Notice, the proposal will be a contract.

Question: What are the specifications of the deployment platform?

Answer: The application uses a flexible architecture built on Node.js, Angular.io, and MongoDB. The assessment will need to be uploaded to our research platform, preferably as a .NET DLL. The application cannot use ActiveX control, applets, or flash for scan and must work on Edge, Chrome, or Firefox.

Question: Can a demonstration of the assessment be provided?

Answer: Unfortunately, we cannot do a demonstration. The technical report (Adis et al., 2022) should provide an adequate overview of the assessment.

Question: Why were personality traits were not part of this assessment?

Answer: The Army already has personality assessments that were designed for selection and assignment. C^3 is meant to supplement those assessments by providing a measure of the potential to learn and succeed specifically in cyber jobs.

Question: There is no direct assessment of social cognitive biases that can influence optimal decision making. Is there a reason for this?

Answer: The C^3 is meant to assess the cognitive skills used in cyber work—things like decisions made about problems such as technological errors, rather than situations involving things like anchoring and adjustment, the availability heuristic, and similar decision biases. The constructs we have were identified by SMEs as important and consistent across related jobs, and cross-referenced with similar occupations on O*NET.

Question: How much do you want to keep from the current C^3 assessment, and to what extent are you open to include new items and paradigms?

Answer: The existing constructs were identified by SMEs as consistent and important across related jobs, and cross-referenced with similar occupations on O*NET. The primary focus of this research is updating definitions, changing how they are measured (including adding or changing items), and demonstrating construct validity. We are open to any modifications within those parameters.

Question: Do offerors need to have a security clearance?

Answer: The data for this effort should be collected using Army personnel. In order to analyze the data collected from Army personnel, the contractor will be required to use an Army-provided computer and must be able to obtain a clearance for use of that computer.

Question: How many awards will be given under this BAA?

Answer: Only one contract will be awarded under this announcement.

Question: Can you clarify whether the full existing instrument will be provided to the ultimate recipient of the contract? (i.e. is it just during this whitepaper stage that you are unable to demonstrate the current assessment? Or would all project development be expected to proceed based solely on the technical report descriptions?) And would the existing instrument be available to the contract recipient in an editable form (aka full set of source code) or would any revisions we design need to be built out as new items instead of being able to modify the prior items directly?

Answer: We will provide the existing assessment battery, including its code, as well as a guide to setup and maintenance. The code will be editable.

Question: What we would need to account for on our project team vs. what aspects of the technical platform are supported by ARI?

Answer: Project teams will need to have the capacity to edit the assessment, as well as support the transition of the assessment to the Army’s hosting platform. ARI will not be able to provide technical support.

Additional Questions Question: Can you provide clarification on what is meant by ‘improve functionality?” Is this asking for hardware solutions? We are referring to “Update the assessment battery to incorporate new measures and improve functionality.”

Answer: Functionality refers to the guidelines that the revised battery should be no more than 30-40 minutes in length, and be user-friendly in terms of ease-of-use, navigability, and interactivity.

Question: Are there specific constraints (test format, hardware/software) that the new and revised measures should adhere to? Is ARI open to changes to the scenarios used in the current C^3, or would they be open to alternative task environments that would be developed with the same priorities of avoiding confounding effects of context and technology familiarity?

Answer: The application uses a flexible architecture built on Node.js, Angular.io, and MongoDB. The assessment will need to be uploaded to our research platform, preferably as a .NET DLL. The application cannot use ActiveX control, applets, or flash for scan and must work on Edge, Chrome, or Firefox.

We are open to changes to the scenarios used in the current C^3, as well as alternative task environments.

Question: The request specifies developing new measures – presumably there are limitations on test length which would need to be considered when incorporating new measures. Are all of the constructs currently in the assessment going to be retained or can we suggest a process for reducing the number of measures?

Answer: The constructs cannot be eliminated, but one major goal is to reduce the test length. This could be achieved by eliminating the existing way a construct is measured and replacing with a new measure.

Question: The request discusses low predictive utility of many of the constructs – are there any publications or tech reports related to those findings?

Answer: A technical report is currently in the review process regarding the results of the criterion validation, but the Adis et al report does discuss definitions of the constructs, their operationalizations in the assessment, and construct validity evidence.

Question: Regarding the research using Army personnel would there be a need of having an IRB approval for human subjects research or do you have one in place through the Army?

Answer: We have an IRB in place through the Army.