Data Driven Test_PWS DRAFT.pdf

CONTRACTOR SUPPORT FOR USAF TEST PILOT SCHOOL DATA SCIENCE &

EMERGING TECHNOLOGIES CURRICULUM AND INSTRUCTION

Background

The U.S. Air Force Test Pilot School’s (USAF TPS) mission is to "Create Highly-Adaptive Critical-Thinking Test Leaders to Accelerate Multi-Domain Capabilities for The Warfighter."

We prepare mid-career aviators, satellite operators, and engineers to test and evaluate highly complex systems in conditions of significant risk, time pressure, senior leadership/Congressional oversight, and resource constraints. Our school is world-class in preparing students for the test and evaluation profession, but in keeping with our mission, we need to further prepare students for advanced and emerging technologies which have the potential to significantly disrupt the government's acquisition, test, and evaluation efforts.

The collection, analysis and utilization of data is transforming the test profession. Going forward, USAF TPS graduates will be testing systems which are as much computers and algorithms as they are aircraft and spacecraft. In developing these systems, graduates will coordinate with multiple AFMC, DoD and contractor agencies – each responsible for a different part of the “system of systems” that composes a modern capability. USAF TPS graduates must be truly collaborative design partners in an iterative process of defining requirements, designing, testing, and improving systems integrating both hardware and software.

For graduates to effectively participate in the design and test of modern capabilities, it is imperative that they gain fluency in the principles of data collection, data analysis and data-driven techniques like machine learning/AI, in addition to mastering conventional flight and space test techniques. USAF TPS is seeking to partner with leading experts in industry and academia who can provide instruction on, and practical experience with, the development, test, and analysis of data-driven systems.

Request Overview USAF TPS is seeking qualified contractors to support the development, instruction and sustainment of curriculum focused on preparing graduates to test autonomous aircraft and data-driven mission systems. The curriculum should focus on the following Curriculum Focus Areas:

(1) Data-Analysis and Controls for Flight Sciences, (2) Data-Driven Analysis and Decision Making for Mission Systems and (3) practical capstone exercises in simulation and testing autonomous aircraft.

Definitions

• Class: A cohort of ~40 students consisting of both flight and space testers completing a year-long course at USAF TPS. Note that USAF TPS always hosts two classes simultaneously which are off phase by 6 months (the senior class is 6 months ahead of the junior class). USNTPS’s cohort consists of ~36 flight testers completing a year-long course. USNTPS also hosts two classes simultaneously annually.

• Contact time: Time the instructor spends with one or more students. The time students spend doing homework and exams is not counted as contact time.

Description of Proposal Requirements Proposals must meet all the requirements below:

• Eligibility: Eligible contractors must have at least 2 years of experience in the development and test of autonomy algorithms for military aircraft. Contractors must have practical experience with deploying autonomy algorithms both in simulation and in hardware.

• Instructors: The lead instructor must have at least 2 years of professional experience in military flight test and 2 years of experience in the development and test of autonomous systems. Experience in these fields can be gained concurrently. The lead instructor can be assisted by other qualified co-instructors with at least 1 year of experience in the development and test of autonomous systems. Instructors with formal teaching experience, especially at an established Test Pilot School, are preferred.

• Location: Instruction will be offered in person at USAF TPS at Edwards AFB, CA.

Instructors are expected to travel to Edwards to teach and to collaborate on curriculum development with USAF Test Pilot School faculty. Instruction at Patuxent River, MD for the United States Naval Test Pilot School (USNTPS) may be required.

• Services: In accordance with the Curriculum Focus Areas and Desired Learning Objectives, contractors should collaborate with USAF TPS to refine the curriculum objectives and syllabus, develop new curriculum, deliver the curriculum to USAF TPS/USNTPS students, and train USAF TPS/USNTPS faculty so they are proficient in the curriculum material.

• Intellectual Property: The intent is for contractors to eventually pass the curriculum on so it is taught solely by USAF TPS/USNTPS faculty. As such, all educational materials produced by the contractor for USAF TPS/USNTPS including lesson plans, labs/activities and course infrastructure is the property of the government.

• Student Instruction Scope: Contractors will teach the curriculum to two separate 40 person classes of TPS students annually. All students will go through the same curriculum progression. Note that students are enrolled at USAF TPS/USNTPS for 1 year and start the course approximately 6 months offset from one another. 60-80 hours of student contact time per class is expected (totaling 120-160 hours annually for both classes). Contact time estimations in this document are notional. Proposals may suggest more or less hours as required.

• Instructor Training Scope: Contractors will also provide instructor training to USAF TPS/USNTPS faculty. 60 hours of instructor training contact time annually is expected.

The intent of instructor training is both to help USAF TPS/USNTPS faculty to gain fluency in the contractor-produced curriculum and to assist USAF TPS/USNTPS faculty in incorporating data science into their own existing courses. Contact time estimations in this document are notional. Proposals may suggest more or less hours as required.

• Template Instruction Calendar: USAF TPS expects that approximately 4 weeks annually will be necessary to instruct each class and 2 weeks annually will be necessary for instructor training. Contractors may choose to instruct one or both classes simultaneously while at Edwards AFB. Student content must be delivered in the order of:

(1) Data-Analysis and Controls for Flight Sciences, (2) Data-Driven Analysis and Decision Making for Mission Systems and, (3) practical capstone exercises in simulation and testing autonomous aircraft. The suggested time allocation per class is outlined below. Contractors may deviate from this template if desired.

o 1 week block per class for Data-Analysis and Controls for Flight Sciences o 1 week block per class for Data-Driven Analysis and Decision Making for

Mission Systems o 2-week block per class for practical capstone exercises in simulation and testing autonomous aircraft o 2 weeks total for training USAF TPS instructors. Scheduled when convenient.

Instruction calendar for USNTPS will be determined post award.

• Syllabus: The proposal shall provide a draft syllabus, with the intent that a more detailed syllabus will be refined through a course design process following contractor selection.

The proposed syllabus should include, at a minimum:

o A notional list of lessons and activities with the estimated duration of instruction and associated curriculum development time in hours. USAF TPS/USNTPS strongly prefers curriculum focusing on hands-on exercises and practical examples. USAF TPS/USNTPS students can be expected to have beginner coding proficiency.

o A proposal for the frequency and duration that instructors will be present at both locations for instruction, faculty training and curriculum development.

o Instructor experience, expertise, and qualifications.

• Resumes: The contractor should submit resumes for its lead instructor and any co-instructors.

• Curriculum Focus Areas: Proposals shall include content in all of the following focus areas.

o Data-Analysis and Controls for Flight Sciences: Understanding and applying data-driven approaches for the analysis of flight sciences data and learning-based approaches for the control of aircraft. Instruction should focus on the technical underpinnings of data-driven approaches, comparisons and contrasts to classical approaches, and considerations for safe and efficient system characterization, flying qualities/performance testing, and envelope expansion.

o Data-Driven Analysis and Decision Making for Mission Systems:

Understanding and applying data-driven approaches for perception, navigation, networking, sensor fusion and management, and electronic warfare. Instruction should focus on the technical underpinnings of data-driven approaches, comparisons and contrasts to classical approaches, and considerations for safe and efficient system characterization and testing.

o Practical Capstone Exercises: Designing and coordinating practical capstone exercises that mimic real developmental flight test and data analysis utilizing the USAF TPS X-62 VISTA, the Joint Simulation Environment, or other suitable systems.

• Desired Learning Objectives: Proposals shall include content in all of the following Desired Learning Outcomes themes. Sub-bullets are listed for reference and may or may not be included in the contractor proposal.

o Data-Driven Decision Making and Control: Using learning-based approaches to develop control algorithms for autonomous aircraft.

Autonomy stacks Reinforcement learning techniques AI performance, robustness, and trustworthiness System characterization and developing simulations Simulation training and test techniques The sim-to-real gap Real world test of autonomous systems o Data-Driven Perception: Using learning-based approaches for classifying, localizing, and filtering sensor data.

Common sensor modalities and data formats Principles of artificial neural networks Training neural networks Neural network architectures for perception Evaluating and testing learning-based perception algorithms Edge deployment o Data-Driven Analysis: Techniques for collecting, curating, and drawing insights from large amounts of unstructured or structured data.

Basic techniques in statistical inference, correlation, regression, sampling, and probability Data visualization Data collection Data labeling Data pipelines and database management System characterization from structured and unstructured data Statistical data analysis techniques

• Timeline: If awarded, vendors must complete curriculum development and be ready for instruction at USAF TPS/USNTPS in January 2025.