SOO Adaptive Quiz Engine draft 1 Jul 2025.pdf

STATEMENT OF OBJECTIVE

(SOO)

Adaptive Quiz Engine (AQE)

Version: 1 July 2025

1.0.BACKGROUND

Air Education and Training Command (AETC) tasked the 19th Air Force, Detachment 24 “Pilot Training Next” to prototype a next generation synthetic training environment that can support the new Fighter/Bomber Fundamentals (FBF) tactical jet training course. A quizzing system was prototyped by Det 24 as part of Air Mobility Fundamentals - Simulator (Only) (AMF-S) and FBF 2.0. This quizzing system was found to be unusable at scale due to limitations in the platform to segregate students by class and base.

1.1. Customer Organization and Mission: 19 AF Flying Training Center of Excellence (19AF/FTCOE) mission is to train the world’s finest Airmen and Aviators to deliver 21st Century Air Power for America.

1.2. Program Objective: The program objective is to streamline pilot training utilizing state of the art commercial solutions. Focus areas include software solutions for user portal, simulation environment, Artificial Intelligent (AI) flight instructor, learning management system, and network and data architecture to fully functional T-38C enhanced Immersive Training Devices (eITDs) and the Instructor Operating Stations (IOSs) platforms at the Undergraduate Pilot Training (UPT) bases.

1.3. Project Overview: This project aims to create an adaptive quiz engine for the FBF. An AQE should enhance pilot training effectiveness and efficiency, preparing students for better retention of the material in long-term memory.

2.0. REQUIREMENT

The Contractor shall provide an adaptive quiz engine that fulfills the requirements outlined in this section. This may be accomplished either by deploying a Contractor-developed adaptive quiz engine or by sustaining and upgrading the Government’s current platform (Section 4), provided that all requirements specified below are fully met. The contractor shall be responsible for providing all necessary software licenses and costs required for its implementation and hosting and sustain Authority to Operate (ATO) either path they choose.

2.1. The Contractor shall deliver the following on 15 September 2025:

Adaptive Quizzing Functionality:

• The quizzing system must require the association of individual questions with specific competencies, including Knowledge, Skills, and Abilities (KSAs) and Learning Objectives (LOs).

The system must allow unique identifiers for each question to be associated to the Government owned Competencies and Skills System (CaSS).

Spaced quizzing (questions related to the same KSA / LO are spread out in the quiz and not asked back-to-back).

Questions should be presented in a semi-random manner while still in compliance with the previous requirement (spaced quizzing).

Each question must have a provision for one or more “follow-up” question(s) covering the same KSA / LO.

• The quiz shall adaptively continue (not advance the student to the next lesson) until the system determines that the student has achieved a satisfactory level of mastery for all designated KSAs and LOs based on a mastery assessment.

Each question must provide targeted remediation resources (e.g., simple text, links, documents, graphics, rich media) in the event of an incorrect student response or low confidence (when indicated).

The quiz will present follow-up questions for those questions answered incorrectly, or when low confidence is indicated, until reaching 100% correct per KSA prior to advancing the student.

A question shall not be repeated unless all questions have been presented prior to achieving 100% correct per KSA.

Hosting and Security:

• The solution must be hosted and maintained by the Contractor within a government cloud environment that already possesses a valid ATO and adheres to all applicable security requirements.

Reporting:

• The system must provide automated assessment and reporting of student mastery levels, aggregated at the question (either by unique identifier or KSA and LO), quiz, student, class, unit, base, and program levels.

• Student accomplishment reporting to the LMS for Fighter Bomber Fundamentals (FBF) must be available for government testing no later than 11 November 2025.

Accomplishment is a minimum reporting of start, stop, and assessment of each individual question and overall assessment of the quiz.

Content Management:

• The solution must deliver content in the LMS for FBF course by 11 November 2025. Roughly, 1,800 predefined Government questions with 0 - 3 supporting images per question.

Training:

• The contractor shall provide comprehensive training for individual users and program managers on the operation and administration of the quizzing system.

Users:

• The system shall provide, at a minimum, three (3) roles: Student, Content Manager, Site Manager.

Student role: student account access to the AQE from within the LMS must be provided by 11 November 2025

2.2. The Contractor shall propose a delivery schedule to integrate the following features:

Reporting:

• The system must provide the capability to add labels and create filter criteria for report generation.

• All quiz results and student performance data must be reported to the Government's Learning Management System (LMS) via xAPI.

1. Quiz results should include, but not limited to, the date and time of the quiz (to establish the version the student was presented), along with the question, the student’s response, and the correct answer for all questions presented.

2. The maintenance of the API/xAPI will be completed by the contractor in accordance with the standards provided by the government.

Content Management:

• The solution must include content creation tools that allow the Government to upload, edit, and modify all content, including questions and remediation materials.

The Government will be responsible for creating all quiz content.

• Each question within the question bank shall have development, review, and publish options.

• The question banks(s) must be exportable in two formats:

1. Student viewable, printable, offline format (.pdf,.docx, etc.) with a sampling of questions to include all associated graphics, remediation, and answer key;

1. This “paper” quiz is primarily for preparing quizzes in advance in the event participants do not have access to the LMS.

2. The sampling size and association of questions to KSAs should be selectable by the Content Manager

2. System ingestible format that allows the question bank to be transferred to alternate systems/databases.

1. Acceptable formats can be anything that allows full association of the question to answers and remediation, which also shall include all graphics and other associated material and links.

• The system must allow Content or Site Manager role users to both review the quiz adaptively (“student mode”), with the ability to clear score data, and review all questions unrestricted, including follow-up questions.

Integration Capability:

• The system must be capable of seamless integration with the Government's LMS, allowing it to be displayed within the government’s authorized web-based LMS.

Confidence Level:

Ability to add confidence level for a quiz as a whole or individual questions.

Confidence level is stored in the same manner as the responses and becomes available for reporting. Confidence level for questions may be used to trigger remediation and a follow-up question.

Question Formats:

Multiple choice: 3 or more possible answers Binary: true/false, yes/no, a) or b)

Multiple answer: 3 or more possible answers with more than one correct answer.

Drag and drop matching: pair content from one list by dragging to another.

Hotspot: multiple selectable areas over an image with one or more being the correct answer.

Version Control:

The question bank(s) shall include the ability for catalog versioning control.

Versions of the questions banks are provided so that specific versions can be associated with lessons within a syllabus. This will provide the ability to update questions for newer syllabi without altering the existing questions in the question bank associated with a syllabus already in training.

Users:

Content Manager: User role managed by the Adaptive Quiz Engine. Manages question bank.

Site Manager: Manages Content Manager roles and can perform all Content

Manager functions.

3.0. DELIVERY

The Contractor shall deliver the most updated version and deliver in accordance with Section 3.1.

3.1 Delivery Period: 15 September 2025 – 14 September 2026.

3.2 Deliverables: The Contractor shall deliver a functional adaptive quiz engine, support plan, documentation, and warranties (if applicable) to 19 AF/FTCOE, Joint Base San Antonio (JBSA) - Randolph AFB, Texas.

4.1 Current Government platform

The government has a primary contractor that is providing cloud hosting with Amplifire as a sub-contractor. The primary contractor is not able to continue providing the hosting. The replacement contractor would be required to take over as the primary contractor. This includes the Google Cloud Platform (GCP) hosting environment, which needs to be renewed in March 2026 and sustainment of the cloud hosting infrastructure within GCP for the sub-contractor Amplifire. The replacement contractor would also be taking over the subcontract with Amplifire and providing the delivery schedule for the updates to the environment or Amplifire to meet the requirement gaps in this contract.

Contract Evaluation Criteria

I. Technical Approach (Weight: 40%) This section assesses the contractor's understanding of the requirements and the feasibility and effectiveness of their proposed technical solution.

• Adaptive Quizzing Functionality (15%):

o Completeness: Does the proposal fully address all aspects of adaptive quizzing, including association with KSAs/LOs, unique identifiers for CaSS, targeted remediation, mastery assessment, spaced quizzing, alignment with KSAs/LOs, and quiz repetition? (Measurable by: Compliance matrix showing how the proposal addresses each requirement, demonstration of functionality) o Effectiveness: How well does the proposed approach demonstrate an effective adaptive learning strategy to achieve mastery? (Measurable by: Explanation of the adaptive algorithm, examples of how the system will adapt to different student performance levels) o Innovation: Does the proposal offer any innovative or value-added approaches to adaptive quizzing? (Measurable by: Description of unique features or functionalities)

• Hosting and Security (10%):

o ATO Compliance: Does the proposal demonstrate a clear understanding of

ATO requirements and how the solution will be hosted and maintained in a government cloud environment with a valid ATO? (Measurable by:

Description of the hosting environment, security certifications, and compliance with security standards) o Security Approach: How robust and secure is the proposed hosting environment and the overall solution architecture? (Measurable by: Security plan, data protection measures, vulnerability assessment procedures)

• Content Management (5%):

o Content Delivery: How effectively does the proposal address the delivery of content in the LMS for the FBF course, including the handling of 1,800 predefined questions and supporting images? (Measurable by: Description of the content delivery process, data migration plan) o Content Creation Tools (Future): Does the proposal demonstrate an understanding of content creation and management capabilities for future requirements? (Measurable by: Description of the content creation tools, version control capabilities)

• Integration Capability (5%):

o LMS Integration: How well does the proposal address the seamless integration with the Government's LMS? (Measurable by: Description of the integration process, API specifications, data exchange protocols)

• Question Formats (5%):

o Variety and Functionality: Does the proposal demonstrate the ability to support the required question formats (multiple choice, binary, multiple answer, drag and drop matching, hotspot) and their associated functionalities?

(Measurable by: Description of the question formats, examples of their implementation)

II. Management Approach (Weight: 30%) This section assesses the contractor's ability to manage the project effectively, including planning, staffing, and risk management.

• Project Management Plan (10%):

o Completeness: Is the project management plan comprehensive and well defined, covering all key aspects of the project? (Measurable by: Review of the project plan, schedule, resource allocation, and communication plan) o Feasibility: Is the proposed schedule realistic and achievable? (Measurable by:

Assessment of the schedule, milestones, and dependencies)

• Staffing and Key Personnel (10%):

o Qualifications: Do the proposed key personnel possess the necessary skills and experience to successfully execute the project? (Measurable by: Review of resumes, experience with similar projects) o Availability: Are the proposed key personnel committed to the project and available for the required level of effort? (Measurable by: Letters of commitment, resource allocation plan)

• Risk Management Plan (10%):

o Identification: Does the proposal identify potential risks and propose mitigation strategies? (Measurable by: Review of the risk register, mitigation plans) o Responsiveness: How well does the risk management plan address potential challenges and contingencies? (Measurable by: Assessment of the risk mitigation strategies, contingency plans)

III. Past Performance (Weight: 20%) This section assesses the contractor's past performance on similar projects.

• Relevance: How relevant are the contractor's past projects to the requirements of this SOO? (Measurable by: Similarity of past projects to the current requirements)

• Quality of Work: What is the documented quality of work on past projects?

(Measurable by: Review of past performance reports, customer feedback)

• Timeliness: Did the contractor deliver past projects on time and within budget?

(Measurable by: Review of past project schedules, cost performance data)

• Customer Satisfaction: What is the documented level of customer satisfaction on past projects? (Measurable by: Review of customer surveys, references)

IV. Price (Weight: 10%) This section assesses the reasonableness and competitiveness of the proposed price.

• Cost Reasonableness: Is the proposed price reasonable and justified? (Measurable by:

Cost analysis, comparison to market rates)

• Cost Realism: Is the proposed cost realistic given the proposed technical approach and level of effort? (Measurable by: Assessment of the cost breakdown, labor rates, and material costs)

• Value for Money: Does the proposal offer the best value for money, considering both technical merit and price? (Measurable by: Overall assessment of the proposal, considering all evaluation criteria)

V. Other Considerations (Pass/Fail)

• Training: Does the contractor provide comprehensive training for individual users and program managers? (Measurable by: Training plan and materials)

• User Roles: Does the system support the required user roles (Student, Content Manager, Site Manager)? (Measurable by: User role descriptions and access control mechanisms)

• Delivery Schedule: Does the proposed delivery schedule meet the required deadlines?

(Measurable by: Review of the delivery schedule and milestones)

Scoring System:

• Use a consistent scoring system for each criterion (e.g., Excellent, Good, Fair, Poor).

• Provide clear definitions for each score level.

• Document the rationale for each score.

Important Notes:

• This is a template and should be customized to your specific needs.

• Consult with legal and contracting professionals to ensure compliance.

• Maintain transparency and fairness throughout the evaluation process.

By using these comprehensive evaluation criteria, you can ensure that you select a contractor that is well-qualified to deliver a successful Adaptive Quiz Engine that meets the needs of the 19 AF/FTCOE. Good luck!