N6134021R0045 - NEW NAWCTSD BAA (FY21).pdf

NAVAL AIR WARFARE CENTER

TRAINING SYSTEMS DIVISION

BROAD AGENCY ANNOUNCEMENT

N61340-21-R-0045

EFFECTIVE DATE: 26 January 2021 EXPIRATION DATE: 25 January 2026

THIS BAA SHALL REMAIN IN EFFECT UNTIL SUPERCEDED OR EXPIRED

Summary of Changes

Change Level Change Date Affected Section Reason

- 01/04/2016 Initial Posting

Amendment 3 05/07/2019 Cover page Corrected Expiration Date from 26 Jan 2021 to 03 Jan 2021

1.2.1 Updated Points of Contact (POCs)

1.2.3 Updated RFP process

3.1 Updated proposal prep info and POCs

3.1.3 Updated proposal prep info

3.1.4 Deleted clause 252.227-7019

3.1.5 Deleted broken link

3.1.6 Deleted outdated language

3.1.8 Updated info regarding contract types

3.1.9 Corrected Expiration Date 03 Jan 2021

3.1.11/3.1.12 Updated info requesting electronic copies

3.2.1.5 Updated to three years Past Performance

5.0 Updated POCs

Attachments 3-5 Deleted attachments

-i-

TABLE OF CONTENTS

Page

1.0 INTRODUCTION ------------------------------------------------------------------------------------------------- 1

1.1 AUTHORITY -------------------------------------------------------------------------------------------------------------- 1

1.2 RECOMMENDED PROCESS --------------------------------------------------------------------------------------------- 1

1.2.1 STEP 1 - TECHNICAL DIALOG (TELEPHONE CALL) --------------------------------------------------------------- 1

1.2.2 STEP 2 - TECHNICAL DIALOG (INFORMAL WHITE PAPER) ------------------------------------------------------ 1

1.2.3 STEP 3 - SUBMISSION OF FORMAL RESEARCH PROPOSAL ------------------------------------------------------- 1

1.2.4 STEP 4 - CONTRACT AWARD FOR SELECTED PROJECTS --------------------------------------------------------- 1

1.3 GOVERNMENT OBLIGATION ------------------------------------------------------------------------------------------ 2

2.0 RESEARCH AREAS AND TOPICS --------------------------------------------------------------------------- 3

2.1 TRAINING TECHNOLOGY AND METHODOLOGY RESEARCH AREA ---------------------------------------------- 4

2.1.1 ADAPTIVE, SIMULATION-BASED TRAINING AND ASSESSMENT ------------------------------------------------- 4

2.1.2 ADVANCED DISTRIBUTED LEARNING (ADL) --------------------------------------------------------------------- 4

2.1.3 HUMAN SOCIAL CULTURAL AND BEHAVIORAL MODELING (HSCB) ------------------------------------------ 4

2.1.4 ADVANCED INSTRUCTIONAL TECHNOLOGY ---------------------------------------------------------------------- 4

2.1.5 AUTOMATED SYSTEMS ---------------------------------------------------------------------------------------------- 5

2.1.6 COALITION WARFARE ----------------------------------------------------------------------------------------------- 5

2.1.7 DECISION-MAKING UNDER STRESS -------------------------------------------------------------------------------- 5

2.1.8 DEPLOYABLE TRAINING SUPPORT TECHNOLOGIES -------------------------------------------------------------- 5

2.1.9 DISTANCE LEARNING ------------------------------------------------------------------------------------------------ 5

2.1.10 DISTRIBUTED DEBRIEF AND AFTER ACTION REVIEW SYSTEMS ------------------------------------------------ 5

2.1.11 EMBEDDED TRAINING TECHNOLOGY ------------------------------------------------------------------------------ 6

2.1.12 GAMES AND GAMING ------------------------------------------------------------------------------------------------ 6

2.1.13 GRAPHICAL USER INTERFACE (GUI) DESIGN AND INFORMATION VISUALIZATION -------------------------- 6

2.1.14 HUMAN SYSTEMS INTEGRATION (HSI) ---------------------------------------------------------------------------- 6

2.1.15 INDIVIDUAL AND TEAM SMALL ARMS TRAINING SYSTEMS ---------------------------------------------------- 7

2.1.16 INNOVATIVE SUBMARINE SYSTEMS TRAINING ------------------------------------------------------------------- 7

2.1.17 INSTRUCTIONAL STRATEGIES AND TEAM MODELING ----------------------------------------------------------- 7

2.1.18 INTELLIGENT TUTORING AND EXPERT SYSTEMS ----------------------------------------------------------------- 7

2.1.19 KNOWLEDGE PRESENTATION FORMATS --------------------------------------------------------------------------- 8

2.1.20 LEADERSHIP DEVELOPMENT --------------------------------------------------------------------------------------- 8

2.1.21 MAINTENANCE TRAINING AND SUPPORT ------------------------------------------------------------------------- 8

2.1.22 MANPOWER AND PERSONNEL SELECTION RESEARCH ----------------------------------------------------------- 8

2.1.23 MOBILE TRAINING TECHNOLOGIES -------------------------------------------------------------------------------- 8

2.1.24 PERFORMANCE MEASUREMENT ------------------------------------------------------------------------------------ 9

2.1.25 SIMULATOR AND VIRTUAL ENVIRONMENT SICKNESS (CYBERSICKNESS) ------------------------------------- 9

2.1.26 TEAM TRAINING AND PERFORMANCE MEASUREMENT ---------------------------------------------------------- 9

2.1.27 THE COGNITIVE SCIENCE OF LEARNING: IMPLICATIONS FOR INSTRUCTION AND MODELING -------------- 9

2.1.28 TRAINING EFFECTIVENESS RESEARCH -------------------------------------------------------------------------- 10

2.1.29 TRAINING TECHNOLOGY FOR DISTRIBUTED AND JOINT SYSTEMS ------------------------------------------- 10

2.1.30 MEDICAL TEAM PERFORMANCE AND SIMULATION TRAINING ------------------------------------------------ 10

2.2 SIMULATION SYSTEMS RESEARCH AREA ------------------------------------------------------------------------- 10

2.2.1 DISPLAY PROJECTOR TECHNOLOGY ----------------------------------------------------------------------------- 10

2.2.2 HELMET-MOUNTED DISPLAYS ----------------------------------------------------------------------------------- 10

2.2.3 LIVE/VIRTUAL/CONSTRUCTIVE (VLC) INTEGRATION --------------------------------------------------------- 11

2.2.4 SENSOR SIMULATION TECHNOLOGY ----------------------------------------------------------------------------- 11

2.2.5 VEHICLE DYNAMIC SIMULATION TECHNOLOGY --------------------------------------------------------------- 11

2.2.6 VISUAL SIMULATION TECHNOLOGY ----------------------------------------------------------------------------- 11

2.3 COMPUTER APPLICATIONS RESEARCH AREA -------------------------------------------------------------------- 13

2.3.1 ADVANCED HUMAN BEHAVIORAL REPRESENTATION --------------------------------------------------------- 13

2.3.2 ANTI-SUBMARINE WARFARE (ASW) AND SUBMARINE OPERATIONS --------------------------------------- 13

2.3.3 ASYMMETRIC ENTITIES ------------------------------------------------------------------------------------------- 13

2.3.4 CAPABILITY MATURITY MODEL INTEGRATION (CMMI) ----------------------------------------------------- 13

2.3.5 EMBEDDED TRAINING --------------------------------------------------------------------------------------------- 13

-ii-

2.3.6 EXPERT SYSTEMS FOR TRAINING -------------------------------------------------------------------------------- 13

2.3.7 FIDELITY FOR TRAINING DEVICES ------------------------------------------------------------------------------- 13

2.3.8 HIGH PERFORMANCE COMPUTING ------------------------------------------------------------------------------- 13

2.3.9 INFORMATION MANAGEMENT FOR SUPPORT OF MODELING AND SIMULATION ---------------------------- 14

2.3.10 REUSABLE SOFTWARE --------------------------------------------------------------------------------------------- 14

2.3.11 SIMULATION NETWORKING --------------------------------------------------------------------------------------- 14

2.3.12 SPEECH RECOGNITION TECHNOLOGY --------------------------------------------------------------------------- 14

2.4 Science, Technology, Engineering and Mathematics (STEM) Education….……….……………….18

3.0 PROPOSAL PREPARATION AND SUBMISSION ------------------------------------------------------ 15

3.1 GENERAL INFORMATION -------------------------------------------------------------------------------------------- 15

3.1.1 ELIGIBILITY --------------------------------------------------------------------------------------------------------- 15

3.1.2 POST-EMPLOYMENT CONFLICT OF INTEREST ------------------------------------------------------------------- 15

3.1.3 RESTRICTIVE MARKINGS ON PROPOSALS ----------------------------------------------------------------------- 15

3.1.4 DATA AND SOFTWARE CLAUSES --------------------------------------------------------------------------------- 16

3.1.5 REPORTING REQUIREMENTS -------------------------------------------------------------------------------------- 16

3.1.6 FACILITIES ---------------------------------------------------------------------------------------------------------- 16

3.1.7 PERIOD OF PERFORMANCE ---------------------------------------------------------------------------------------- 16

3.1.8 CONTRACT TYPES -------------------------------------------------------------------------------------------------- 17

3.1.9 PROPOSAL SUBMISSION CUT-OFF DATE ------------------------------------------------------------------------ 17

3.1.10 FOLLOW-ON CONTRACTS ----------------------------------------------------------------------------------------- 17

3.1.11 PROPOSAL COPIES ------------------------------------------------------------------------------------------------- 17

3.1.12 MAILING ADDRESS ------------------------------------------------------------------------------------------------ 17

3.1.13 NON-U.S. CITIZEN PARTICIPATION ------------------------------------------------------------------------------ 17

3.2 RESEARCH PROPOSAL CONTENTS --------------------------------------------------------------------------------- 17

3.2.1 TECHNICAL SECTION ---------------------------------------------------------------------------------------------- 17

3.2.1.1 Proposed Research ------------------------------------------------------------------------------------------------ 18

3.2.1.2 Potential Contribution -------------------------------------------------------------------------------------------- 18

3.2.1.3 Offeror’s Qualifications ------------------------------------------------------------------------------------------ 18

3.2.1.4 Personnel ----------------------------------------------------------------------------------------------------------- 18

3.2.1.5 Past Performance -------------------------------------------------------------------------------------------------- 19

3.2.1.6 Statement of Work ------------------------------------------------------------------------------------------------ 19

3.2.2 ADMINISTRATIVE SECTION --------------------------------------------------------------------------------------- 19

3.2.2.1 Contract Type ------------------------------------------------------------------------------------------------------ 19

3.2.2.2 Environmental Considerations ----------------------------------------------------------------------------------- 19

3.2.2.3 Organizational Conflicts of Interest ----------------------------------------------------------------------------- 19

3.2.2.4 Security Issues ----------------------------------------------------------------------------------------------------- 19

3.2.2.5 Disclosure Preference --------------------------------------------------------------------------------------------- 20

3.2.2.6 Understanding of Evaluation Policy ---------------------------------------------------------------------------- 20

3.2.2.7 Representations, Certifications and Other Statements of Offerors or Quoters----------------------------- 20

3.2.2.8 Subcontracting Plan ----------------------------------------------------------------------------------------------- 20

3.2.3 COST SECTION ------------------------------------------------------------------------------------------------------ 20

3.2.3.1 Period of Performance -------------------------------------------------------------------------------------------- 20

3.2.3.2 Direct Labor -------------------------------------------------------------------------------------------------------- 20

3.2.3.3 Materials ------------------------------------------------------------------------------------------------------------ 27

3.2.3.4 Other Direct Costs ------------------------------------------------------------------------------------------------- 20

3.2.3.5 Indirect Costs ------------------------------------------------------------------------------------------------------ 21

3.2.3.6 Fee/Profit ----------------------------------------------------------------------------------------------------------- 21

4.0 PROPOSAL EVALUATION ---------------------------------------------------------------------------------- 22

4.1 INITIAL REVIEW ------------------------------------------------------------------------------------------------------- 22

4.2 PEER REVIEW---------------------------------------------------------------------------------------------------------- 22

4.2.1 PROPOSED RESEARCH --------------------------------------------------------------------------------------------- 22

4.2.2 POTENTIAL CONTRIBUTION --------------------------------------------------------------------------------------- 22

4.2.3 OFFEROR’S QUALIFICATIONS ------------------------------------------------------------------------------------- 22

4.2.4 PERSONNEL --------------------------------------------------------------------------------------------------------- 22

-iii-

4.2.5 PAST PERFORMANCE ---------------------------------------------------------------------------------------------- 22

4.2.6 COST REALISM ----------------------------------------------------------------------------------------------------- 22

4.3 PROPOSAL COMPARISONS ----------------------------------------------------- ERROR! BOOKMARK NOT DEFINED.

5.0 ASSISTANCE INSTRUMENTS ------------------------------------------------------------------------------ 23

6.0 FORMAL PROPOSAL FORMS ----------------------------------------------------------------------------- 24

-1-

1.0 INTRODUCTION

1.1 AUTHORITY

The Research and Engineering Competency of the Naval Air Warfare Center Training Systems Division (NAWCTSD) issues this Broad Agency Announcement (BAA) under the provisions of paragraphs 35.016 and 6.102(d)(2) of the Federal Acquisition Regulation (FAR), which provides for the competitive selection of research proposals. Contract(s) based on responses to this BAA are in full compliance with the provisions of The Competition in Contracting Act of 1984 (PL 98-369) as codified in 10 USC 2304.

NAWCTSD contracts with educational institutions, nonprofit organizations, and private industry for research and development (R&D) in those areas covered in Section 2.0 of this BAA. This BAA is intended to cover, in general, all R&D areas of interest to NAWCTSD and its customers relating to simulation and training technology.

1.2 RECOMMENDED PROCESS

The following four-step sequence is recommended for offerors contemplating submission of a proposal under this BAA.

This sequence allows for an early determination of the potential for interest and funding, and limits offeror and Government expenditure of effort to prepare and review formal proposals for research that may have little chance of being supported.

1.2.1 Step 1 - Technical Dialog (Telephone Call)

This step initiates a technical dialog between the Government and the potential offeror. The initial point of contact may direct callers to a specific scientific point of contact based on the topic area and specifics of the proposed research project. The initial contact points for each area of research interest identified in Section 2.0 are shown below:

Research Area Point of Contact Phone

2.1 Training Technology and Methodology Mr. John Hodak (407)380-4737

2.2 Simulation Systems Mr. Benito Graniela (407)380-8031

2.3 Computer Applications Mr. Tyson Griffin (407)380-4671

2.4 Science, Technology, Engineering and

Mathematics (STEM) Education

Mr. Robert (Bob) Seltzer

(407)380-4115

1.2.2 Step 2 - Technical Dialog (Informal White Paper)

This step is a continuation of the technical dialog for projects of interest. The scientific point of contact may request submission of an informal white paper to facilitate understanding of the scientific and technical aspects of the proposed research project. Although there are no restrictions or formal requirements, use of the white paper is intended to determine which efforts are of sufficient scientific and technical merit preparatory to submission of a formal research proposal as described in Section 3.0; therefore, white papers should not be so lengthy or detailed as to constitute a formal proposal. White papers may contain a bottom-line cost estimate.

1.2.3 Step 3 - Submission of Formal Research Proposal

This step ends the technical dialog. If there is sufficient interest in a proposed research project, the Contract Specialist will issue a written Request for Proposal to the offeror.. Once the Contracting Office receives a formal research proposal, communication between scientific personnel and the offeror is permitted only as authorized by the Contracting Officer.

1.2.4 Step 4 - Contract Award for Selected Projects

Regardless of whether the four-step process is used, all proposals will receive an initial review (see Section 4.1) and the Contracting Officer will notify the offeror, in writing, whether the proposal will be processed for award. The primary basis for selecting proposals for award shall be scientific/technical merit, importance to agency programs, funds

-2-availability, and cost (to include realism and reasonableness to the extent appropriate). See Section 4.2 for specific evaluation criteria.

1.3 GOVERNMENT OBLIGATION

PERSONS SUBMITTING PROPOSALS ARE CAUTIONED THAT ONLY A CONTRACTING OFFICER

MAY OBLIGATE THE GOVERNMENT TO ANY AGREEMENT INVOLVING EXPENDITURE OF

GOVERNMENT FUNDS.

-3-

2.0 RESEARCH AREAS AND TOPICS

NAWCTSD has comprehensive simulation and training systems responsibilities ranging from research and technology base development through system acquisition and life cycle support. The Research and Engineering Competency is NAWCTSD’s arm of the laboratory system. Its mission is to plan and perform a full range of directed R&D in support of Naval training systems. The work covers the broad spectrum of training simulation technology as applied across mission areas and all stages of training. It is intended that programs under the BAA include Fundamental research related to this mission. “Fundamental Research” means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

Collaborative arrangements between universities and industrial companies are encouraged. Projects should take maximum advantage of existing university and industry research and engineering programs and facilities.

Capabilities are needed to promote and conduct multi-disciplinary (e.g., engineer, psychologist, instructional specialists) research in training and simulation technology; enhance the development of training devices; foster productive working relationships with NAWCTSD scientists; and be a source of innovation for the application of instructional principles in training systems.

The requirement for R&D conducted under this BAA is to explore unique training techniques incorporating innovative behavioral and engineering technologies, which are needed for more effective and/or less expensive training systems. Technology products may include empirical research, software and computer models, test beds and proof of concept demonstrations. Projects should provide insight to NAWCTSD personnel to optimize the use of training systems. Research areas that are described in the remainder of this chapter are important, but other R&D topics supporting training systems may also be considered. The following research areas and topics are not intended to be mutually exclusive but rather are often interdependent and may be exercised in various combinations at any time. In fact, proposals that involve interdisciplinary teams are especially encouraged.

-4-

2.1 TRAINING TECHNOLOGY AND METHODOLOGY RESEARCH AREA

Research is sought for the following topics:

2.1.1 Adaptive, Simulation-Based Training and Assessment

Modern simulations afford a wide variety of capabilities for personnel training and assessment in complex task domains, such as the ability to customize instructional content to an individual learner’s needs. Accurate skill assessment during simulation-based training provides the foundation for identifying and correcting skill deficiencies via adaptive training. Potential topics of research for the adaptive, simulation-based training research and development topic include: (1) innovative, simulation-based assessment strategies; (2) task analysis methodologies in support of adaptive training; (3) development of methodologies and associated hardware and software requirements (e.g., intelligent agents) driving the real-time customization of training content; (4) adaptive feedback/after-action review strategies; (5) the development of measurement frameworks (e.g., Bayesian networks, data mining algorithms) supporting performance measurement and adaptive training; and (6) training effectiveness evaluation research with respect to adaptive training tools

2.1.2 Advanced Distributed Learning (ADL)

The Navy, the other services, the Department of Defense, the Federal Government, academia, and private industry have made the commitment to develop capabilities in ADL. The goal of ADL is to deliver instructional and job performance aiding information anytime, anywhere, to anyone who needs it. Potential topics of research include:

(1) learner-centric, adaptive instructional techniques; (2) effective use of web-unique instructional and performance aiding techniques; (3) design of effective ADL resource centers; (4) design and use of learning management systems; (5) effective use of sharable content objects; (6) techniques for performance measurement and feedback;

(7) use of intelligent tutoring systems and intelligent agents; (8) techniques for continuous, career-long learning; (9) student motivational techniques; (10) techniques for configuration management of ADL; (11) role of the instructor or facilitator in ADL; (12) tradeoff analyses between instructional approaches and organizational constraints (e.g., firewalls); (13) tools and techniques for ADL-specific front-end analysis; and (14) tools and techniques for evaluation of ADL systems.

2.1.3 Human Social Cultural and Behavioral Modeling (HSCB)

Irregular Warfare missions require the Navy to increasingly operate in joint and coalition operations (e.g., stability, security, transition, and reconstruction or SSTR). These expanded missions require developing effective and integrated collaborative multi-team decision making processes that range from identification of adversarial intent, to establishing trust and conducting negotiations. Research is needed to develop: 1) valid methods for developing and analyzing behavioral signatures in culturally embedded contexts; 2) formal mathematical models to incorporate behavioral signatures into Human Behavior Representations (HBRs); 3) effective, adaptive training regimes, and 4) real-time decision aids to facilitate on-the-job performance of operational personnel.

2.1.4 Advanced Instructional Technology

Numerous instructional features have been implemented and proposed for use in simulators and other training devices. Because of the changes in Navy training policies, some features need to be reexamined and/or explored.

Examples include (a) scenario management features; (b) instructor on-line, help and tutorials; (c) automated performance measurement; (d) instructorless training features; (e) management of multiple warfare areas and management of Semi-Automated Forces (SAFORs); (f) augmented reality; and (g) instructor communications.

Research is required to determine: (1) optimal applications of instructional features; (2) optimal display methodologies for instructional information; (3) training effectiveness of features; and (4) development of new features that take advantage of emerging technologies, such as virtual environment, distributed training systems, and the use of video, audio and animation in instructor training.

-5-

2.1.5 Automated Systems

Future systems are growing increasingly complex and becoming highly automated. Experiences in other industries have identified a number of problems associated with human operators working in an environment with a significant amount of automation. Research is needed to help identify effective design and training strategies that will minimize problems associated with loss of situation awareness, under- and over-reliance on automation, etc.

2.1.6 Coalition Warfare

Coalition warfare is proving to be an effective means of dealing with hostile nations and terrorist threats. However, despite the political, financial, and military advantages coalition warfare represents, it also poses significant logistical problems, specifically in the domains of training and readiness. Each individual nation thoroughly trains its armed services to pre-specified customs and standards, and routinely participates in coalition exercises.

Advances in Network-Centric Operations (NCOs) are expected to enhance coalition operations; however, more research is needed to understand the human systems integration and training requirements. The objective of this effort is to apply recent developments in decision theory, individual and team training, leadership and commander's intent, multi-cultural diversity, and collaborative support technologies to enhance coalition warfare in a NCO. To accomplish this objective, research is required that addresses: (1) the developmental stages of culturally diverse leaders and teams who are working over networks; (2) adaptive team and leader performance; and (3) training and collaborative support tools for distributed decision making.

2.1.7 Decision-Making Under Stress

Conflicts today continue to be characterized by rapidly unfolding ambiguous and stressful situations that impact individual and team decision-making in combat. The objective of this effort, therefore, is to apply recent developments in decision theory, individual and team training, and collaborative technologies to enhance decision quality under stressful conditions. To accomplish this objective, research is required in the following three areas:

(1) performance measurement; (2) stress; and (3) training for complex team decision making in face-to-face and distributed environments.

2.1.8 Deployable Training Support Technologies

Advances in weapon system complexities have significantly increased the demand on human operators and teams.

One of the main problems with deployable training systems is a lack of support or training for instructors. What is needed are systems that provide training support tools for individuals and teams of individuals who are responsible for training in deployed contexts; for example, to support crucial instructor functions at sea. Potential areas of research include: (1) automated performance measurement; (2) integration of observational measures with automated performance measures; (3) human performance modeling technologies; (4) instructor training technologies (i.e., interpretation of automated performance measures, debriefing skills); (5) techniques for automated instruction; and (6) techniques and tools for delivering feedback and debrief.

2.1.9 Distance Learning

More than any other service branch, the Navy is depending on e-learning to train and educate its personnel. One of the greatest challenges for distance learning developers and designers is how to create courseware that is engaging and effective; in short, how to avoid churning out page-turning, passive material that fails to fully captivate the learner. More emphasis is being placed on a wide array of active learning activities, such as team mission rehearsal at a distance, joint training, embedded simulations, game-based learning, and job performance aids. Research is needed to determine how to maximize these approaches, to exploit or adapt available groupware products, and how to address the social/human issues associated with collaborative training and education over networks.

2.1.10 Distributed Debrief and After Action Review Systems

Debriefing and after action review is critical to the training effectiveness of distributed training exercises. Currently, debrief technologies are focused on single simulation systems, and are not readily available to address distributed simulation-based exercises with multiple teams. The content and formation of information, as well as instructional

-6-features, are not responsive to the needs of multiple team and cross team debriefs. Research needs to focus on the application of debrief systems to multi-platform debriefing and after action review, and should include: (1) identification and implementation of optimal information content and format; (2) identification and development of instructional features; (3) techniques for the transfer of data from the training device to the debrief station; and (4) design and development of a repository of debrief data and information.

2.1.11 Embedded Training Technology

Embedded training systems include training capabilities that are resident on operational deployed equipment or are interfaced with it. Embedded training ranges from single equipment operator training up to full system team training. Embedded training maximizes fidelity and accessibility by putting the training site on board deployed weapon platforms. Four instruction technologies that have been identified as appropriate for embedded training R&D are: (1) performance measurement (dynamic assessment) and explicit feedback; (2) missing team/team member simulation; (3) automatic intelligent platforms; and (4) automated adaptive instruction. Other technologies may be appropriate as well, including application of eye tracking, speech recognition, latent semantic analysis, adaptive training technologies, and intelligent tutoring to embedded training systems.

2.1.12 Games and Gaming

The use of games is aggressively pursued for its instructional value in today’s society and military; however, it is a relatively new instructional technology that has limited empirical support. Theoretically, if built properly, these training games have the potential to improve skill acquisition, recall of facts, increase situation awareness, and improve ability to effectively multi-task, for example. The question is how can games be developed to ensure they meet specific training objectives. Potential topics of research include: how to identify the essential characteristics of effective games, how to implement these characteristics into a training game, how games engage users in game play, and how this engagement of users improves training.

2.1.13 Graphical User Interface (GUI) Design and Information Visualization

Future training systems and network centric environments generate exponential increases in workload for operators and leadership due to heightened levels of incoming data that require logging, monitoring, integration and interpretation. The complexity of information management in multi-team training and operations is further complicated by distribution and asynchrony. Intelligent graphic and multimodal interface design solutions are critical to support future Network Centric Warfare (NCW) and training environments where team members and decision makers are separated geographically and temporally. Research should address how to support the team as well as the individual. Potential research areas include: (1) information visualizations to support shared cognition and/or decision making; (2) common collaboration tools; (3) multi-modal and intuitive user interfaces and their respective combinations; and (4) usability methodologies. This list is not exhaustive and additional approaches are likely to emerge as science and technology advances.

2.1.14 Human Systems Integration (HSI)

In order to meet the challenges of developing systems that meet required mission capabilities at the lowest lifecycle cost, it is crucial that human operators and maintainers are considered at all phases of the acquisition lifecycle from early concept development to disposal of the system. HSI is a systems engineering approach to optimize cost, schedule, and performance of designed systems through insuring that these human considerations, which include manpower, personnel, training, and human engineering, are included within the systems acquisition process along with hardware and software considerations. The development of tools, technologies, and processes to support HSI practitioners is necessary to effectively achieve these goals. Potential research and development opportunities in this area include: (1) tools and technologies to support task analysis, human performance modeling and simulation-based acquisition; (2) validated approaches of measuring, assessing, analyzing, and evaluating human performance within the context of military systems; (3) planning and decision support tools for acquisition program managers; (4) methods for the prudent application of automation and decision support to include considerations of operator and team workload and situational awareness; and (5) tools to support the sharing of data across program management, engineering, and HSI disciplines

-7-

2.1.15 Individual and Team Small Arms Training Systems

Small arms training is inherent in the military. However, in addition to basic weapons handling and marksmanship, this training must also include as a significant component opportunities to train correct tactics, procedures, teamwork, and decision-making. The wide range of environments, adaptability of hostile forces, and the increasing variety of mission objectives, coupled with longer or more numerous deployment cycles, has increased the need for training alternatives in this domain. Research areas include: (1) analysis of training requirements; (2) creation of databases; (3) weapon modeling and simulation; (4) computer controlled hostiles and neutrals; (5) instructional features; (6) networking; (7) computer generated graphics; (8) deployability issues (e.g., foot print, instructor support, simulation sickness); and (9) the development, demonstration, and evaluation of training approaches.

2.1.16 Innovative Submarine Systems Training

Submarine systems have unique requirements for shore-based as well as on-board training. Submarine piloting and navigation places unusual demands as compared with surface ships. Tactical operations involve particularly complex data gathering and analysis techniques. Potential areas of training research for such submarine capabilities include: (1) innovative on-board computer-based training system design; (2) training requirements measurement techniques capable of discriminating between requirements for on-board vs. shore-based training; (3) fidelity requirements measurement techniques; (4) analysis and display of measures of effectiveness and performance for at-sea, on-board, and classroom; (5) cost reduction techniques for operator, sub-team, and full-team training; (6) techniques to motivate students, especially on-board, to engage in training; and (7) virtual environment technology to provide on-board “presence” to classroom instruction. Other submarine-specific training research may be appropriate as well.

2.1.17 Instructional Strategies and Team Modeling

Researchers and developers of Intelligent Tutoring Systems (ITSs) have demonstrated improvements in learning in traditional academic, static domains. ITSs apply advanced cognitive modeling and diagnosis to develop objective-based instructional materials “on-the-fly” that are tailored and adaptive to each student. Research is necessary to extend these types of technologies and approaches to support objective-based training in dynamic, team contexts.

An extension of a traditional ITS that is capable of supporting military team training could include components such as: automated performance measurement and diagnosis at the individual and team level for teamwork and taskwork, adaptive scenario modification, simulated teammates, and intelligent selection of instructional strategies (i.e., normative feedback, process feedback, scaffolding, on-line feedback, off-line feedback). In order to extend traditional ITSs, fundamental differences between academic and operational domains must be addressed. Some of these differences include scenario-driven versus student-driven pacing of decision-making and problem-solving activities and the existence of multiple expert models of problem-solving approaches. Given these differences, significant research is essential in determining effective way(s) to choose, develop, and provide instructional strategies in order to minimize cognitive disruptions and maximize learning in a scenario-based, team training environment.

2.1.18 Intelligent Tutoring and Expert Systems

As the technological complexity of Naval weapon systems has increased, there has been a corresponding increase in the number of jobs in the Navy that are classified as technical or highly technical. Although research indicates that one-on-one tutoring is the best way to teach complex skills, it is an economic impossibility. The student-to-teacher ratio is too high to allow an instructor to provide one-on-one tutoring for each student. Intelligent tutors based upon intelligent agents, expert systems, and other methods and technologies can provide individualized instruction by tailoring pace, sequence, content, presentation style, and feedback for individual learners. Reductions in instructor workload and training time, and increases in the student motivation and level of learning, will result in direct readiness and economic benefits. Potential topics of research include: (1) individual tutors and job aid experts embedded in devices, "in the pocket," or residing in distributed systems; (2) natural language interfaces and other innovative interface designs for tutors and job aids; (3) model-based diagnostic and feedback systems; (4) methods and tools for automatic explanation generation;(5) knowledge engineering tools for capturing subject matter and instructional expertise; and (6) tools for automatic generation of instructional content and job aids.

-8-

2.1.19 Knowledge Presentation Formats

Complex tactical decision making performance requires decision-makers to harness and apply large quantities of declarative and procedural knowledge. Research is needed to determine optimal methods for presenting this knowledge to trainees so that they learn and encode it in a manner that is consistent with cognitive processing requirements. In particular, the relative merits of various multi-media formats must be determined, and issues such as authoring multi-media instructional systems must be addressed. Research is also needed in the application of cognitive task analysis techniques to the authoring of multimedia instruction and the translation of such analysis into appropriate presentation strategies.

2.1.20 Leadership Development

Network-centric warfare initiatives have placed significant challenges on leaders. Leadership responsibilities have been pushed further down the chain of command then ever before. Additionally, leaders in this environment are forced to cope with leading ad hoc rotating teams at a distance in complex multi-team systems. Situations such as this outstrip the current theories of research. Potential areas of research include: (1) the role of the leader in a multi-team, distributed system; (2) methods, strategies and tools to facilitate the development of leadership expertise earlier in the career pipeline; (3) leaders' communication and dissemination of information in a Network Centric Warfare (NCW) environment; (4) effective issuance of command intent in a NCW environment; (5) systems to aid leaders in obtaining and maintaining situational awareness in a networked environment; (6) the leaders role in creating conditions for team effectiveness in a networked environment; (7) the impact of leadership style/skill in distributed, multicultural teams; and (8) technologies to facilitate distributed team leadership.

2.1.21 Maintenance Training and Support

As the complexity of modern military systems increases, there is a strong need to exploit existing and emerging technologies to provide effective training and performance support strategies and systems for maintainers and maintenance related operations. Potential areas for research include: (1) intelligent tutoring for maintainers; (2) embedded assessment capabilities to track maintainer performance in operational equipment; (3) distance learning capabilities to allow for remote training of maintainers and remote performance support; (4) training for conditioned based maintenance; and (5) integrated Interactive Electronic Technical Manuals (IETMs). Other technologies may be appropriate as well, including application of augmented reality, speech recognition and eye tracking to embedded training or performance aiding systems.

2.1.22 Manpower and Personnel Selection Research

Essential to maximizing Fleet readiness is the recruitment, assessment, selection, and retention of qualified Navy personnel. Applied manpower and personnel selection R&D training technology is necessary to identify and meet current Navy manpower requirements, and to develop innovative, research-based solutions for the future Fleet. This BAA research topic covers areas including: (1) job/task analytic strategies (e.g., job analysis methods applied to training, cognitive task analyses); (2) personnel assessment and test development (e.g., cognitive and noncognitive test development, knowledge testing, problem-based learning assessment, computer-based and adaptive testing, psychometric theory); (3) performance criterion development (e.g., simulation-based performance measurement;

training performance measurement; portable, practical, or embedded measurement devices); (4) test validation strategies (e.g., innovative validation approaches drawing from advances in psychometric theory); and (5) personnel attraction, recruitment, and retention research.

2.1.23 Mobile Training Technologies

One of the current trends in workforce training and education is the transition from eLearning to mLearning.

mLearning refers to mobile learning content to support the growing mobile and remote workforce independent of location in time or space. mLearning is the intersection of mobile computing and eLearning: accessible resources wherever you are, strong search capabilities, rich interaction, powerful support for effective learning, and performance-based assessment. Basically it can be thought of as eLearning through mobile computational devices:

Palms, Windows CE machines, even your digital cell phone.

-9-

Many entities (including the Advanced Distributed Learning (ADL) Labs) are researching the technological aspects and challenges associated with mLearning. However, the pedagogical research on what types of learning and what strategies are best suited to this unique delivery methodology is limited. Research is needed to determine optimal methods for presenting knowledge to learners to maximize the effectiveness of learning and encoding content consistent with cognitive processing requirements.

2.1.24 Performance Measurement

The NAVAIR training systems community requires basic and applied research and development in a variety of areas related to the measurement of human performance at the individual, team, and multi-team level. Increased reliance of simulation to meet mission-level training requirements has created an urgent requirement to develop measurement capabilities in this environment. Specific areas of research that require attention include, but are not limited to: (1) improved data collection technologies in both live and simulation-based environments; (2) diagnosis of the root causes of performance deficits; (3) rapid and efficient creation of accurate human performance models;

(4) creation and validation of mathematical algorithms to compute higher-order integration of automated performance measurement systems with human observer/evaluator data; (5) technologies and strategies to enhance the capabilities and offset the limitations of human observers and raters; (6) valid and reliable methods for aggregating or integrating multiple observations to produce evaluations at the team and multi-team levels; (7) data presentation formats and strategies for effective debriefing preparation and delivery; (8) linking observed performance to specific individual, team, and multi-team competencies; (9) technologies and strategies for measuring performance in distributed, simulation based training exercises; and (10) technical solutions for effectively sharing training data over simulation networks.

2.1.25 Simulator and Virtual Environment Sickness (Cybersickness)

Numerous reports have documented the occurrence of psycho-physiological disturbances, balance problems, visual illusions, and sickness of trainees following the use of some simulators and Virtual Environments (VEs). The result has been compromised training, decreased simulator use, and aftereffects that may occur as long as 8 to 24 hours after training. Potential areas of research include: (1) survey the frequency of “simulator sickness” or “cybersickness;” (2) isolate the design and operating characteristics which contribute to sickness; (3) develop human factors design and procedure guidelines to minimize simulator or VE sickness; and (4) develop instrumentation to test and accept simulations based on system parameters correlated with simulator and VE sickness."

2.1.26 Team Training and Performance Measurement

The training community places a high priority on R&D for team performance, emphasizing the need for systematic analysis and design of team training technology. Still, methodological and practical problems for individual and multiple teams continue to exist. Potential areas of interest should be applied to the individual and distributed team problem, to include: (1) performance measurement techniques; (2) debriefing feedback procedures and tools; (3) team training design; (4) measures of effectiveness (to include process and outcome measures) and criterion development; (5) design, development and evaluation of team training approaches; (6) team modeling; (7) instructional strategies; (8) applications of learning principles to team performance; (9) hierarchical and distributed team performance measurement; and (10) specific and general measurement criteria and techniques for distributed training operations ranging from small to large scale exercises.

2.1.27 The Cognitive Science of Learning: Implications for Instruction and Modeling

Human learning, just like other human activities, takes place at the multiple levels described by Allan Newell in his 1990 book, Unified Theories of Cognition: biological (milliseconds), cognitive (seconds), rational (minutes – days), and social/organizational (week – decades). A comprehensive cognitive science of learning must address the relevant variables, processes and relationships at all of these levels. As theoretical advances are made in these areas, additional work is needed to translate this progress into useful results. Two fields of interest to the military that are ripe for practicable infusions from a cognitive science of learning are education/training and human behavioral modeling. Both applications are growing increasingly complex as military technology, operations and teaming arrangements continue to expand, and both have the capability to significantly impact our mission readiness.

-10-

Research should advance our knowledge in the cognitive science of learning and demonstrate applicability to the design of effective instruction and/or the development of valid human behavioral models.

2.1.28 Training Effectiveness Research

Training theories and applications suggest that training effectiveness is a complex, multi-dimensional construct.

Therefore, in order to assess readiness, training performance, and other important outcomes, research is needed to:

(1) define categories of Measures of Effectiveness (MOEs) and Measures of Performance (MOPs) for a wide range of training systems; (2) develop multi-component approaches to training evaluations; and (3) specify the relationship among training requirements, knowledge, skills and attitudes, and MOPs. In addition, methods to forecast knowledge, skills and attitude requirements with associated MOEs/MOPs for evolving and newly developed systems are required.

2.1.29 Training Technology for Distributed and Joint Systems

Distributed interactive simulation provides unique opportunities for a coordinated training environment via networked simulations. There are multiple simulators and associated systems capabilities needed to support the life cycle of a distributed simulation-based training exercise. There is a strong need to exploit existing and emerging training technologies to identify effective training strategies for these distributed teams. Potential areas for research include: (1) specifications of distributed training requirements; (2) distributed performance measurement procedures and techniques; (3) identification of techniques and tools for delivering distributed feedback and conducting distributed debriefs; (4) distributed scenario generation; (5) instructional strategies for distributed missions; and (6) evaluation procedures for distributed training systems.