Sources Sought Draft PWS.pdf

PERFORMANCE WORK STATEMENT

Analytical Operational Test and Evaluation Support Services

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA

1. Scope of Tasking. Contractor will provide technical and analytical support to Operational Test and Evaluation Force (OPTEVFOR) in the definition, conduct and analysis of structured Operational Test and Evaluation (OT&E) of Information Warfare (IW) systems. This support will be required in one or more of the following forms: Documentation Review, Operational Test Design, Test Planning Support, Test Execution, Data Management, Test Operations Support, Test Analysis and Reporting, Modeling and Simulation (M&S) Support, and Program Management Support. OPTEVFOR is charged with leading the Operational Test community with highly skilled testers and staff that adapt to change, and provide credible, prompt, warfighting-focused test results to Navy, Marine Corps and Coast Guard forces and acquisition leadership.

2. Background There are five warfare divisions at OPTEVFOR that are supported by competency divisions.

The warfare divisions include Undersea Warfare (40), Air Warfare (50), Information Warfare (60), Surface Warfare (70), Expeditionary Warfare/Littoral Combat Ship (80) Division. Each warfare division has a Navy Captain as the division Director with a senior civil servant as the Deputy or a senior civil servant as the division Director and a Navy Commander as the Deputy.

The warfare divisions represent the traditional core of the OPTEVFOR organization. This is where the Fleet operators reside. The work there allows OPTEVFOR to bridge the gap from technical to the tactical, successfully. The warfare divisions are staffed by uniformed military personnel and Navy government civilians, augmented by a contractor workforce.

There are four competency divisions: Policy and Operations (01A), Test Design (01B), Test Planning and Analysis (01C) and Cybersecurity Testing (01D). In addition, the Technical Director (00TD) oversees the technical aspects of the Command’s mission. Other support divisions include the Staff Commanding Officer and Administration (10), Chief Information Officer (CIO) (20), Contracts (01K), and the Comptroller (30). The members of the competency divisions generally work within the test team to ensure that the Commander’s policies are adhered to and that best practices are applied.

• Test Policy, Operations and Training (Code 01A) ensures OPTEVFOR policy conforms to Navy Test and Evaluation (T&E) Policy; monitors production of T&E products; leads continuous process improvement efforts; coordinates external OT&E inquiries Inspector General and Government Accountability Office;

manages all government T&E training (including budget requests); provides T&E editorial support to the command; and plans, analyzes and evaluates the effectiveness of command processes.

• Test Design and M&S (Code 01B) is responsible for the Mission- Based Test Design (MBTD) process, M&S technical review, test planning and post-test iterative process analysis support, test design production, Capabilities Based Test and Evaluation coordination, and Targets, Ranges, and Test Resources coordination.

• Test Planning and Analysis Division (Code 01C) is responsible for policy, processes and standardization for test planning, test execution, post-test analysis and evaluation (reporting). This includes Lead Test Engineer support to the OPTEVFOR Warfare Divisions and Fleet Resources coordination to support execution of OT&E.

• Cybersecurity Division (Code O1D) OT&E integrates critical cybersecurity testing into the acquisition lifecycle and tests Systems under Test (SUT) per Department of Defense Instruction 5000.02 and Director, Operational Test and Evaluation (DOT&E) Memorandum of 01 August 2014. The Warfare division will be responsible for executing acquisition test design, planning support to ultimately provide the Cybersecurity OT&E group (O1D) support in cybersecurity execution and reporting. Through a rigorous and iterative test process, Cybersecurity OT&E assists in the development and fielding of more secure and resilient Information Technology (IT) systems supporting the warfighters. Code 01D Cybersecurity OT&E group includes the following support positions: Technical Director, Future Capabilities and Threat Emulation Developer, Analysis Manager, Training and Lab Manager, Test Team Leads and Certified Ethical Hackers.

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA

3. Information Warfare Division and Supported Programs The Information Warfare (IW) Division (Code

60) tests and evaluates all ashore and afloat Command, Control, Communications, Computers, Intelligence, Space, Business, and Enterprise Information systems such as radars, radio frequency transmitters/receivers, digital switching networks, identification friend or foe, tactical decision-making systems, intelligence systems, combat direction systems, tactical links, information warfare systems, cryptologic systems, space and satellite systems, and non-tactical automated information systems. The division's main effort is to assist, through early involvement, in the acquisition process conduct tests, and provide timely and accurate evaluations to characterize system performance and resulting capability. Current IW test and evaluation programs include, but are not limited to:

• Automated Digital Network System (ADNS) Software Defined Network (SDN)

• Advanced Extremely High Frequency Satellite Communication Sys (AEHF)

• Amphibious Tactical Communications System (ATCS)

• Advanced Tactical Data Link Systems (ATDLS)/ LINK16

• Battle Force Tactical Network (BFTN) Resilient Command, Control, and Communications (RC3) System

Enhancement (BRSE)

• Command and Control Air Operations Suite (C2AOS)

• Command and Control Processor (C2P)

• Consolidated Afloat Network and Enterprise Services (CANES)

• Commercial Broadband Satellite Program (CBSP)Satellite Terminal Transportable Non-Geostationary (STtNG)

• Cryptologic Carry-On Program (CCOP)

• Counter Insider Threat Capability (CITC)

• Computer Network Defense (CND)

• Distributed Common Ground System – Navy Increment 2 (DCGS-N Inc 2)

• Department of Defense Health Management System Modernization (DHMSM)

• Digital Modular Radio (DMR)

• Electronic Procurement System (ePS)

• Enhance Polar Satellite – Replacement (EPS-R)

• Enterprise Resource Planning (ERP)/ Next Generation ERP (ERP+)

• Environmental Satellite Receiver Processors (ESRP)

• Force Level Integration Suite (FLINT)

• Global Broadcasting System (GBS)

• GPS-Base Positioning, Navigation, and Timing Service (GPNTS)

• Joint Military Global Positioning System (GPS) User Equipment (MGUE)

• Global—Theatre Security Cooperation Management System (G-TSCMIS)

• Hazardous Weather Detection and Display Capability (HWDDC)

• HORIZON

• Integrated Communications and Data System (ICADS)

• Intelligence Carry-On Program (ICOP)

• Identification Friend or Foe (IFF) Mode 5

• Joint Common Access Program (JCAP)

• Long Endurance Electronic Decoy (LEED)

• LINK Monitoring and Management Tool (LMMT)

• AN/SPN-50 Maritime Air Radar for Tactical Aircraft (MARTA)

• Mobile Advanced EHF Terminal (MAT) https://items.navair.navy.mil/iBoss/Programs/Details?unid=d0a5f5b5-9aef-4c79-a8bd-b4a608a82017 https://items.navair.navy.mil/iBoss/Programs/Details?unid=82f98ff4-ada0-4282-b122-cbe4bab22e1c

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA

• Maritime Tactical Command and Control (MTC2)

• Mobile User Objective System (MUOS)

• MyNavyHR portfolio (MyNavyHR)

• Nuclear Command, Control, and Communications Navy Modernized Hybrid System (NC3 NMHS)

• Navy Cyber Situational Awareness (NCSA)

• Naval Court Martial and Reporting System (NCORS)

• Navy—Electronic Chart Display and Information System (N-ECDIS)

• Naval Integrated Tactical Environment System—Next Generation (NITES Next)

• Naval Maintenance, Repair, and Overhaul (N-MRO)

• Navy Multiband Terminal (NMT)

• Naval Operational Supply System (NOSS)

• Navy Personnel and Pay (NP2)

• Network Tactical Common Data Links (NTCDL)

• Project OVERMATCH (OM)

• Protected Tactical Enterprise Service (PTES)

• Protected Tactical SATCOM (PTS)

• Real Time Spectrum Operations (RTSO)

• Sea Services Security Assistance Logistics and Execution (S3ALE)

• Scaled Onboard Electronic Attack (SOEA)

• SCORPION

• SHARK CAGE

• Surface Electronic Warfare Improvement Program (SEWIP) o SEWIP Block 2 o SEWIP Block 3 o SEWIP Lite

• SPECTRAL

• Shipboard Panoramic Electro-Optic/Infrared (SPEIR)

• Ship’s Signal Exploitation Equipment Increment F (SSEE Inc F) o SSEE Modifications (SSEE Mods) o SSEE Tactical Cryptologic System (SSEE TCS)

• United States Naval Observatory—Precise Time and Astronomy (USNO-PTA)

• Wideband Anti-Jam Modem System (WAMS)

• Weather Satellite Follow-On Microwave (WSF-M)

4. Period of Performance and Location.

4.1. Period of Performance. See schedule.

4.2. Place of Performance. Unless required to support mission accomplishment, work shall primarily be performed at the customer’s site. If the number of personnel exceeds onsite capabilities, unclassified work will be expected to be performed offsite. The government will not determine a situation where the contractor may utilize “Telework” as this is a professional services contract, so any work performed by the contractor away from the customer location will be upon the contractor to ensure all requirements are being met within the respective task orders issued under this contract. The customer location is:

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA

Operational Test and Evaluation Force 7970 Diven Street Norfolk, VA 23505-1498

While onsite at OPTEVFOR, the contractor shall comply with OPNAVINST 5239 (series), with specific regard to non-government hardware and software. Normal workdays at OPTEVFOR are Monday through Friday and normal working hours are from 0730 through 1600 EST. The manned quarter deck watch will unlock the front and back doors to CA-10 at 0500 (M-F) and lock the front and back doors to CA-10 at 1830 (M-F). Any contractor who desires to remain past 1830 in CA-10 to perform services under their support contract MUST notify the quarterdeck CDO/watch stander NLT 1700 on the day they will be working past 1830. Once the contractor has secured all work and access in their respective workspace, they will notify the CDO/watch stander that they are securing for the day.

4.3. Travel. Travel can be expected to occur CONUS and OCONUS and will be required throughout the course of this contract. The need for travel associated with this PWS shall be determined in advance by the COR at the Task Order level and authorized by the COR. Travel shall be paid by the Contractor and reimbursed by the Government. Travel costs/expenses will be considered reasonable and allowable only to the extent that they do not exceed on a daily basis, the maximum per diem rates in effect at the time of the travel. Travel expenses include airfare, per diem, rental car, local travel and miscellaneous expenses as authorized by the Joint Travel Regulations (JTR). Costs for transportation may be based upon mileage rates, actual costs incurred or a combination thereof, provided the method used results in a reasonable charge. No mileage costs will be incurred when travel is within a 50-mile radius of the primary work location. The JTR, while not wholly applicable to Contractors, shall provide the basis for the determination as to reasonable and allowable. Maximum use is to be made of the lowest available customary standard coach or equivalent airfare accommodations available during normal business hours. Exceptions to these guidelines shall be approved in advance by the Contracting Officer or Designee. Travel dates and locations are to be confirmed through the TA and approved by the COR in writing via email, prior to commencement of any travel.

4.4. Observed Federal Holidays

• Columbus Day

• Veterans Day

• Thanksgiving

• Christmas Day

• New Years’s Day

• MLK Jr. Day/Inauguration Day Mon

• Washington’s Birthday Mon

• Spring Holiday (MIL Only)

• Memorial Day

• Juneteenth National Independence Day

• Independence Day

• Labor Day

4.5. Travel Instructions

4.5.1. Ensure all Contractors supporting this requirement meet the minimum requirements of the Commander, U.S. Fleet Forces Command/Commander, U.S. Pacific Fleet Instruction 6320.3B, dated 07Apr14, Medical Screening for U.S. Government Civilian Employees, Contractor Employee’s, and Guests and Visitors Prior to Embarking Fleet Units.

4.5.2. Any travel OCONUS by the Contractor will need to review the Foreign Clearance Guide at http://www.fcg.pentagon.mil and select the country of intended travel. During OCONUS travel, the contractor shall operate in accordance with International Traffic in Arms Regulation (ITAR) Part 124, approved by US Department of State, Office of Defense Trade Controls.

4.5.3. The Contractor is not authorized to perform any travel that is not in conjunction with this contract. It is the Department of the Navy policy not to allow a charge of profit or fee on http://www.fcg.pentagon.mil/

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA reimbursable items.

4.5.4. The Government will make no reimbursements for a contractor’s local travel (defined as less than 50 miles from the place of performance).

4.5.5. In no instance shall the contractor exceed the amount specified in the contract for travel

Contract Line Item Number (CLIN) expenses.

4.5.6. Locations include CONUS, OCONUS and at sea locations and any other location, which may be added on a Task Order (TO) basis based on future test requirements.

4.5.7. For planning purposes any travel location OCONUS must be considered as a possible travel location unless travel restrictions at time of test do not permit entry. Possible OCONUS locations include, but are not limited to the following: Sasebo, Japan, Naples Italy, Australia, Hawaii, Halifax, Nova Scotia, Nanoose, British Colombia, and any other location, which may be added on a TO basis based on future test requirements.

5. Specific Tasking. The contractor shall provide support in evaluating the operational effectiveness and suitability of the system under test that has been assigned to OPTEVFOR for OT&E in accordance with the processes and procedures prescribed in OPTEVFOR M-3980.2 (series), the Operational Test and Evaluation Manual. This manual and other governing OT&E references are listed in paragraph 13 and are available from the OT Resource Library on the OPTEVFOR Sharepoint site:

https://flankspeed.sharepoint-mil.us/sites/OPTEVFOR/SitePages/01A-Division-OT-Resources.aspx.

5.1. All government furnished Non-Secure Internet Protocol Router (NIPR) templates to be used by the contractor shall only be obtained from the authoritative source, the OT Resource Library on the OPTEVFOR Sharepoint site https://flankspeed.sharepoint-mil.us/sites/OPTEVFOR/SitePages/01A-Division-OT- Resources.aspx.

5.2. All government furnished Secure Internet Protocol Router (SIPR) templates to be used by the contractor shall only be obtained from the authoritative source, the OPTEVFOR SIPR Share Drive:

Y: / OT&E Production Library.

5.3. The contractor shall sign up for the OPTEVFOR “Alerts for OT Resource Changes” to receive updates (which can be tailored on what you receive) on all instructions, Best Practices, and templates. Instructions to sign up are found on the OPTEVFOR SharePoint page under 01A – Operations, OT Instructions and Guidance:

https://flankspeed.sharepoint-mil.us/sites/OPTEVFOR/SitePages/01A-Division-OT-Resources.aspx

5.4. The relevant process that requires the use of government furnished templates are (but not limited to):

a. Test Design o LTD Brief o LTD Letter o IEF and TIEF o IEF Change Letter o SWP TIEF Approval Letter o E-IPR Brief o RCRM

b. Test Strategy o TEMP or MTS Letter (O6 Comment, Signature, Change) o TECP Brief

c. Test Planning o Test Plan (IOT&E, FOT&E, OA/EOA, QRA, DCP, VCD) o Concept of Test Brief o Cyber Survivability Test Plan o CP0 Brief o CP2 Brief

d. Post-Test Iterative Process (PTIP) https://flankspeed.sharepoint-mil.us/sites/OPTEVFOR/SitePages/01A-Division-OT-Resources.aspx https://flankspeed.sharepoint-mil.us/sites/OPTEVFOR/SitePages/01A-Division-OT-Resources.aspx https://flankspeed.sharepoint-mil.us/sites/OPTEVFOR/SitePages/01A-Division-OT-Resources.aspx https://flankspeed.sharepoint-mil.us/sites/OPTEVFOR/SitePages/01A-Division-OT-Resources.aspx

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA o Blue & Gold Sheets o Report Letter (IOT&E, FOT&E, OA/EOA, QRA) o SERB/E-SERB Brief o Letter of Observation (LOO) o Assessment of Operational Capability (AOC)

5.5. Overall Task Guidance. To gain efficiencies and synergy in contract execution, the Government encourages the Contractor to consider:

5.5.1. Flexibility in production. OPTEVFOR produces information which may take many forms.

Support teams should be willing to experiment, bring up new ideas or better ways of doing things, but remain willing to execute stated processes as required by the contract and/or task order.

5.5.2. Present concerns about the validity of the information produced, and value to the end customer (Fleet or Program). Remain willing to defend a contrary position until government makes a final decision.

5.5.3. Remain Efficient and Economical in production. Strive to produce information as quickly as possible without sacrificing quality, and with the appropriate level of expertise.

5.5.4. Maintain a broad perspective. Awareness of testing outside of a specific program to seek and recommend testing efficiencies and overall kill chain evaluation opportunities.

5.5.5. Technical competence. Improve T&E community’s ability to understand, analyze, communicate, and ultimately visualize T&E issues from a warfighter and technical perspective.

5.6. Mission Based Test Design

5.6.1. Mission Analysis

5.6.1.1. The Contractor shall perform Mission Based Test Design (MBTD) mission analysis in accordance with the Integrated Evaluation Framework (IEF) Checklist and IEF Template, respectively.

5.6.1.2. The contractor shall analyze source data to perform Mission Based Test Design mission analysis.

5.6.1.3. The contractor shall identify and document the mission areas.

5.6.1.4. The contractor shall identify and document system capabilities.

5.6.1.5. The contractor shall derive and document operator tasks.

5.6.1.6. The contractor shall define and document the operational conditions applicable to the system to be tested.

5.6.1.7. The contractor shall document the mission analysis in the Mission Based Test and Evaluation System (MBTES).

5.6.1.8. The contractor shall document the mission analysis in an IEF document.

5.6.1.9. The contractor shall review the mission analysis during MBTD reviews.

5.6.1.10. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of mission analysis MBTD reviews.

5.6.2. Requirements Analysis

5.6.2.1. The contractor shall perform Mission Based Test Design requirements analysis in accordance with the IEF Checklist and IEF Template, respectively.

5.6.2.2. The contractor shall analyze source data to perform Mission Based Test Design requirements analysis.

5.6.2.3. The contractor shall identify and document measures of system performance, inclusive of critical measures.

5.6.2.4. The contractor shall link measures of system performance to subtasks and document this linkage.

5.6.2.5. The contractor shall identify and document the data required to quantify the measures of system performance.

5.6.2.6. The contractor shall identify and document the data required to quantify the

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA operational conditions applicable to the system to be tested.

5.6.2.7. The contractor shall document requirements analysis in the MBTES.

5.6.2.8. The contractor shall review the requirements analysis during MBTD reviews.

5.6.2.9. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of requirements analysis MBTD reviews.

5.6.3. Test Design

5.6.3.1. The contractor shall perform Mission Based Test Design requirements analysis in accordance with the IEF Checklist and IEF Template, respectively.

5.6.3.2. The contractor shall analyze source data to perform Mission Based Test Design.

5.6.3.3. The contractor shall identify and document the candidate response variables for statistical test design.

5.6.3.4. The contractor shall identify and document factors/conditions (with corresponding levels) associated with each response variable or critical measure used in test design.

5.6.3.5. The contractor shall participate in and document the statistical test design for candidate response variables and/or critical measures.

5.6.3.6. The contractor shall identify and document the test vignettes.

5.6.3.7. The contractor shall determine and document the appropriate test method.

5.6.3.8. The contractor shall determine and document a test schedule supporting execution of the test design.

5.6.3.9. The contractor shall identify and document the expected use of M&S in the execution of the test design.

5.6.3.10. The contractor shall determine and document the resources required to conduct the test.

5.6.3.11. The contractor shall determine and document the limitations to test based on known availability of resources and assets to conduct the test.

5.6.3.12. The contractor shall develop and document data sheets and test instrumentation

(DELIVERABLE A001) to support test execution.

5.6.3.13. The contractor shall document the test design in the MBTES.

5.6.3.14. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of test design MBTD reviews.

5.7. Stakeholder Collaboration Working Integrated Product Team (WIPTs), outreach, etc

5.7.1. The contractor shall make recommendations to the warfare division in accordance with

OPTEVFOR processes found in the OT&E Manual and IEF Checklist

5.7.2. The contractor shall identify and recommend outside organization(s) for participation and inclusion in Test Design activities.

5.7.3. When directed, the contractor shall coordinate briefings/meetings in support of

OPTEVFOR Test Design activities in accordance with IEF Checklist and IEF Template.

5.7.4. When directed, the contractor shall participate in briefings/meetings in support of OPTEVFOR Test Design activities in accordance with IEF Checklist and IEF Template.

5.7.5. When directed, the contractor shall lead briefings/meetings in support of OPTEVFOR Test Design activities in accordance with IEF Checklist and IEF Template.

5.7.6. The contractor shall provide a trip report (DELIVERABLE A002) when the activities occur away from the OPTEVFOR site in Norfolk, VA.

5.8. Document and Brief Preparation. The contractor shall prepare and document briefings (DELIVERABLE A001) to include the IEF, Executive In-Process Review and stakeholder collaboration briefings, as directed.

5.9. Test and Evaluation Master Plan (TEMP) Input Preparation

5.9.1. The contractor shall analyze source data to provide input to the TEMP, to include program schedules, the IEF, applicable policy documents and any other program-provided materials

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA

5.9.2. The contractor shall develop and document input to the TEMP IAW the DOT&E TEMP Guidebook.

5.9.3. The contractor shall develop and document comments to the TEMP related to or affecting

OT&E.

5.9.4. The contractor shall provide a trip report (DELIVERABLE A002) when contract activities occur away from the OPTEVFOR site in Norfolk, VA.

5.10. Test Planning

5.10.1. Plan of Action and Milestones (POA&M)

5.10.1.1. The contractor shall develop and document a POA&M for the Test Planning process in accordance with the Test Planning Checklist.

5.10.1.2. The contractor shall analyze source data to perform Test Planning POA&M development.

5.10.1.3. The contractor shall obtain and maintain access to the program MBTES database to support the test planning process.

5.10.2. Purpose of Test

5.10.2.1. The contractor shall perform the activities associated with development of Purpose of Test in accordance with the Test Planning Checklist.

5.10.2.2. The contractor shall analyze source data to perform Test Planning Purpose of

Test development.

5.10.2.3. The contractor shall update and document the SUT and SoS descriptions from the

IEF for use in the Test Plan.

5.10.2.4. The contractor shall identify and document the SUT test configuration.

5.10.2.5. The contractor shall identify and document the Developmental Test /Integrated

Test contributions to OT&E.

5.10.2.6. The contractor shall identify and document regression test requirements for any

Follow- on OT&E phases, if applicable.

5.10.2.7. The contractor shall identify and document changes to the limitations to test.

5.10.2.8. The contractor shall update and document the Mission Critical Subsystem

Matrix (MCSM) from the IEF.

5.10.2.9. The contractor shall document the Purpose of Test activities in a Test Plan document in accordance with the Test Planning Checklist and the appropriate test plan templates.

5.10.2.10. The contractor shall review the Purpose of Test activities during test plan reviews.

5.10.2.11. The contractor shall identify test equipment/instrumentation needed for test execution.

5.10.2.12. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of Purpose of Test reviews.

5.10.3. Test Schedule

5.10.3.1. The contractor shall perform the activities associated with development of the Test

Schedule in accordance with the Test Planning Checklist.

5.10.3.2. The contractor shall analyze source data to perform Test Planning Test

Schedule development.

5.10.3.3. The contractor shall identify, recommend and document data to be qualified for use in OT&E.

5.10.3.4. The contractor shall develop and document the Test Schedule, to include test events, for each test phase and period.

5.10.3.5. The contractor shall document the Test Schedule data in the MBTES.

5.10.3.6. The contractor shall document the Test Schedule activities in a TP document in accordance with the Test Planning Checklist and the appropriate TP templates.

5.10.3.7. The contractor shall review the Test Schedule activities during TP reviews.

5.10.3.8. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of Test Schedule TP reviews.

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA

5.10.4. Detailed Method of Test (DMOT)

5.10.4.1. The contractor shall perform the activities associated with development of the

DMOT in accordance with the Test Planning Checklist.

5.10.4.2. The contractor shall analyze source data to perform DMOT development.

5.10.4.3. The contractor shall identify and document the data to be collected on each test event.

5.10.4.4. The contractor shall develop and document the start and stop criteria for each test event.

5.10.4.5. The contractor shall develop and document the GO/NO GO criteria for each test event.

5.10.4.6. The contractor shall develop and document the DMOT for each test event.

5.10.4.7. The contractor shall document the DMOT in a TP document in accordance with the Test Planning Checklist and the appropriate TP templates.

5.10.4.8. The contractor shall review the DMOT during TP reviews.

5.10.4.9. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of DMOT plan reviews.

5.10.5. Data Collection Plan (DCP)

5.10.5.1. The contractor shall perform the activities associated with development of the DCP in accordance with the Test Planning Checklist.

5.10.5.2. The contractor shall analyze source data to perform Test planning DCP development.

5.10.5.3. The contractor shall review, update and document, as required, any changes to data requirements.

5.10.5.4. The contractor shall develop and document the DCP.

5.10.5.5. The contractor shall identify test equipment/instrumentation needed for test data collection.

5.10.5.6. The contractor shall identify data repository.

5.10.5.7. The contractor shall create and document data sheets, interview sheets, focus group sheets, and survey sheets in support of the DCP.

5.10.5.8. The contractor shall document the DCP data in the MBTES.

5.10.5.9. The contractor shall document the DCP in a TP document in accordance with the Test Planning

Checklist and the appropriate TP templates.

5.10.5.10. The contractor shall review the DCP during TP reviews.

5.10.5.11. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of DCP TP reviews.

5.10.6. Data Analysis Plan (DAP)

5.10.6.1. The contractor shall perform the activities associated with development of the DAP in accordance with the Test Planning Checklist and appropriate TP templates.

5.10.6.2. The contractor shall analyze source data to perform Test Planning DAP development.

5.10.6.3. The contractor will develop and document the Critical Operational Issue (COI) resolution method.

5.10.6.4. The contractor will develop and document data scoring criteria.

5.10.6.5. The contractor shall identify and document Scoring Board Guidance.

5.10.6.6. The contractor shall provide a data analysis repository with Government concurrence.

5.10.6.7. When directed, the contractor shall automate data analysis with Government concurrence.

5.10.6.8. The contractor shall document the DAP in a TP document in accordance with the Test Planning

Checklist and the appropriate TP templates.

5.10.6.9. The contractor shall review the DAP during TP reviews

5.10.6.10. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of DAP TP reviews.

5.10.7. Stakeholder Collaboration (WIPTs, outreach, etc.)

5.10.7.1. The contractor shall make recommendations to the warfare division in accordance with OPTEVFOR processes found in the Test Planning Checklist and the appropriate TP templates.

5.10.7.2. The contractor shall identify and recommend outside organization(s) for

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA participation and inclusion in Test Planning activities.

5.10.7.3. When directed, the contractor shall coordinate briefings/meetings in support of OPTEVFOR Test Planning activities in accordance with the Test Planning Checklist and the appropriate TP templates.

5.10.7.4. When directed, the contractor shall participate in briefings/meetings in support of OPTEVFOR Test Planning activities in accordance with the Test Planning Checklist and the appropriate TP templates.

5.10.7.5. When directed, the contractor shall lead briefings/meetings in support of OPTEVFOR Test Planning activities in accordance with the Test Planning Checklist and the appropriate TP templates.

5.10.7.6. The contractor shall provide a trip report (DELIVERABLE A002) when the activities occur away from the OPTEVFOR site in Norfolk, VA.

5.10.8. Document and Brief Preparation. The contractor shall prepare and document briefings (DELIVERABLE A001) to include the Test Plan, Data Collection Plan, Concept of Test, Test Plan Review Board and stakeholder collaboration briefings, as directed.

5.11. Modeling and Simulation (M&S) Support

5.11.1. M&S Requirements Development

5.11.1.1. The Contractor shall perform M&S Requirements Development in accordance with references OPTEVFORINST 5000.1 series, M&S Requirements Letter template, and Best Practice series

5.11.1.2. The contractor shall identify and document candidate critical measures/response variables, along with their associated conditions, during the Test Design portion of the MBTD process that will be supplemented by M&S data.

5.11.1.3. The contractor shall develop and document a M&S Requirements letter in accordance with M&S Requirements Letter template and Best Practice series.

5.11.1.4. The contractor shall review the M&S requirements during MBTD reviews.

5.11.1.5. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of M&S Requirements Development.

5.11.2. Develop Accreditation Plans

5.11.2.1. The contractor shall develop an M&S Accreditation Plan in accordance with the Accreditation Plan Development Guide and the Best Practice Series.

5.11.2.2. The contractor shall develop and document the accreditation approach method consistent with COTF policy.

5.11.2.3. The contractor shall develop and document the Accreditation Plan.

5.11.2.4. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of M&S Accreditation Plan Development.

5.11.3. Review Verification &Validation (V&V) Plans and Reports

5.11.3.1. The contractor shall review program M&S V&V plans and reports in accordance with M&S Requirements Letter template and Best Practice Series.

5.11.3.2. The contractor shall review and document disagreements or issues with V&V plan methodologies.

5.11.3.3. The contractor shall provide recommendations for alternative V&V methodologies supporting

5.11.3.4. OPTEVFOR Accreditation Requirements.

5.11.3.5. The contractor shall review program M&S V&V reports and document disagreements or issues with methodologies/results based on comparison with live test results.

5.11.3.6. The contractor shall develop and document the OPTEVFOR V&V Report addendum.

5.11.3.7. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of M&S V&V Plans or

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA

Reports reviews.

5.11.4. Accreditation Support

5.11.4.1. The contractor shall perform M&S Accreditation in accordance with M&S

Requirements Letter template and the Best Practice series.

5.11.4.2. The contractor shall perform and document, in coordination with OPTEVFOR

Test Design analysts, any additional required analysis not provided in the V&V report in support of accreditation.

5.11.4.3. The contractor shall identify and document conclusions and recommendations based on comparison of live test results with M&S test results.

5.11.4.4. The contractor shall identify and document significant M&S limitations that impact the accreditation recommendation.

5.11.4.5. The contractor shall identify and document where the M&S results support the intended use for OT.

5.11.4.6. The contractor shall develop and document an accreditation recommendation in an Accreditation Letter.

5.11.4.7. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of M&S Accreditation activities and reviews.

5.11.5. Stakeholder Collaboration

5.11.5.1. The contractor shall make recommendations to the warfare division in accordance with OPTEVFOR processes found in OPTEVFORINST 5000.1 series and the Best Practice series.

5.11.5.2. The contractor shall identify and recommend outside organization(s) for participation and inclusion in M&S Verification, Validation, and Accreditation (VV&A) activities.

5.11.5.3. When directed, the contractor shall coordinate briefings/meetings in support of OPTEVFOR M&S VV&A activities in accordance with OPTEVFORINST

5000.1 series and the Best Practice series.

5.11.5.4. When directed, the contractor shall participate in briefings/meetings in support of OPTEVFOR M&S VV&A activities in accordance with OPTEVFORINST

5000.1 series and the Best Practice series.

5.11.5.5. When directed, the contractor shall lead briefings/meetings in support of OPTEVFOR M&S VV&A activities in accordance with OPTEVFORINST

5000.1 series and the Best Practice series.

5.11.5.6. The contractor shall provide a trip report (DELIVERABLE A002) when the activities occur away from the OPTEVFOR site in Norfolk, VA.

5.11.6. Document and Brief Preparation. The contractor shall prepare and document briefings (DELIVERABLE

A001) to include the M&S Requirements Letter, Accreditation Plan, Verification and Validation Report Addendum, Accreditation Letter and stakeholder collaboration briefings, as directed.

5.12. Test Execution

5.12.1. Pre-test Coordination

5.12.1.1. The contractor shall identify and schedule required test services (ranges, targets, personnel, etc.) for the test phase.

5.12.1.2. The contractor shall develop and submit asset scheduling requests to OPTEVFOR Fleet Services schedulers in accordance with the OT&E Manual

5.12.1.3. The contractor shall develop and provide the OPTEVFOR pre-test letters for test units and ISICs in accordance with the OT&E Manual

5.12.1.4. The contractor shall develop and provide briefing materials as required to support Operational Test Requirements Reviews (OTRRs) IAW SECNAVINST 5000.2 series.

5.12.2. Pre-test Brief

5.12.2.1. The contractor shall prepare and document the Pre-Test briefing (DELIVERABLE A001).

For Information Warfare (Code 60)

5.12.3. Test Operations

5.12.3.1. The contractor shall collect and document test data using data cards/sheets, interview sheets, focus group sheets, and survey sheets in accordance with OPTEVFOR procedures and the signed TP.

5.12.3.2. The contractor shall distribute text execution materials to all test participants, when directed.

5.12.3.3. The contractor shall develop and document any test deviations resulting from test variances.

5.12.3.4. The contractor shall develop and document a daily summary of test operations email, when directed.

5.12.3.5. The contractor shall coordinate/operate test equipment/instrumentation with Government concurrence.

5.12.4. Post-Test Activities

5.12.4.1. The contractor shall conduct and document post-test interviews for all test events requiring interviews of participants.

5.12.4.2. The contractor shall conduct and document test team debriefs, when directed.

5.12.4.3. The contractor shall conduct and document test-site scoring boards, when directed.

5.13. Post-Test Iterative Process (PTIP) and Reporting

5.13.1. Data Scoring

5.13.1.1. The Contractor shall perform PTIP Data Scoring in accordance with Test Reporting Handbook and the Best Practice series.

5.13.1.2. The contractor shall inventory, compile and document all collected data, comparing data collected to that required by the Test Plan scope, DMOT and

DCP.

5.13.1.3. The contractor shall develop and document a test execution summary.

5.13.1.4. The contractor shall review the data during Data Scoring reviews.

5.13.1.5. The contractor shall develop and document the End of Test message.

5.13.1.6. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of Data Scoring reviews.

5.13.2. COI Evaluation

5.13.2.1. The Contractor shall perform PTIP COI Evaluation in accordance Test

Reporting Handbook and Best Practice Series.

5.13.2.2. The contractor shall develop and document risks and/or deficiencies on

OPTEVFOR Blue and Gold Sheets.

5.13.2.3. The contractor shall develop and document test results in OPTEVFOR test report results paragraph format.

5.13.2.4. The contractor shall document any limitations to test identified during test execution.

5.13.2.5. The contractor shall develop and document COI evaluation via the COI

Evaluation Chart.

5.13.2.6. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of COI Evaluation reviews.

5.13.3. Data Analysis

5.13.3.1. The Contractor shall perform PTIP Data Analysis in accordance with reference

(a) and the Best Practice series.

5.13.3.2. The contractor shall conduct and document an analysis of all measures.

5.13.3.3. The contractor shall develop and conduct a Responds Variable Analysis

Outbrief in coordination with Test Design Analyst support.

5.13.3.4. The contractor shall develop a draft Data Analysis Summary (DAS).

5.13.3.5. The contractor shall review the DAS during analysis working groups, when directed.

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA

5.13.3.6. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of Data Analysis reviews.

5.13.4. Deficiency/Risk Documentation

5.13.4.1. The Contractor shall perform PTIP Deficiency/Risk Documentation IAW reference (a) and the Best Practice series.

5.13.4.2. The contractor shall distribute and collect comments on draft Blue/Gold Sheets.

5.13.4.3. The contractor shall review the Deficiencies/Risks during PTIP reviews.

5.13.4.4. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of Deficiency/Risk reviews.

5.13.5. System Evaluation

5.13.5.1. The Contractor shall perform PTIP System Evaluation in accordance with Test

Reporting Handbook and the Best Practice series.

5.13.5.2. The contractor shall develop and document evaluation reports to support

Command Systems Evaluation Review Board (SERB) / Executive System Evaluation Review Board (E-SERB) PTIP reviews.

5.13.5.3. The contractor shall provide Divisional Summary Briefs.

5.13.5.4. The contractor shall provide post-test Fact Sheet.

5.13.5.5. The contractor shall provide post-test Fleet Brief.

5.13.5.6. The contractor shall provide Lessons Learned.

5.13.5.7. The contractor shall review the system evaluation during PTIP reviews.

5.13.5.8. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of System Evaluation

5.13.6. Stakeholder Collaboration

5.13.6.1. The contractor shall make recommendations to the warfare division in accordance with OPTEVFOR processes found in the Test Reporting Handbook, PTIP Checklist and the Best Practice series.

5.13.6.2. The contractor shall identify and recommend outside organization(s) for participation and inclusion in PTIP activities.

5.13.6.3. When directed, the contractor shall coordinate briefings/meetings in support of OPTEVFOR PTIP activities in accordance with the PTIP Checklist, Best Practice series, and the appropriate test report templates.

5.13.6.4. When directed, the contractor shall participate in briefings/meetings in support of OPTEVFOR PTIP activities in accordance with the PTIP Checklist, Best Practice series, and the appropriate test report templates.

5.13.6.5. When directed, the contractor shall lead briefings/meetings in support of OPTEVFOR PTIP activities in accordance with the PTIP Checklist, Best Practice series, and the appropriate test report templates .

5.13.6.6. The contractor shall provide a trip report (DELIVERABLE A002) when the activities occur away from the OPTEVFOR site in Norfolk, VA.

5.13.7. Document and Brief Preparation

5.13.7.1. The contractor shall prepare and document briefings (DELIVERABLE A001) to include the Evaluation Report, Post-Test Brief, Scoring Board, COI evaluation working group products, SERB/E- SERB Brief, DAS and Blue/Gold Sheets, as directed.

5.13.7.2. The contractor shall prepare the Risk/Deficiency cover letter, the DAS, and the test report per the OT&E Manual, Test Reporting Handbook and the Best Practice series.

5.14. Cyber Security Subject Matter Expert/Operational (SME)

5.14.1. Cyber Security Support.

5.14.1.1. IAW the DoD Cybersecurity Test and Evaluation Guidebook 2.0 Series, the contractor shall utilize

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA paragraph F.2.1.2, Cybersecurity Subject Matter Experts and Figure F-2 RASCI Table to assist the OTD and 01D Exploitation Analyst (EA) in cyber test preparations and coordination.

5.14.2. Cyber Survivability Test Planning.

5.14.2.1. Cyber Survivability Test Planning Research

5.14.2.2. The Contractor shall support Cyber Survivability Test Planning Research in accordance with Cyber

Test Planning Process guidance.

5.14.2.3. The contractor shall review and identify technical documents required to support Cyber

Survivability Test Planning.

5.14.2.4. The contractor shall identify need dates for all required technical documents and provide dates to the OTD/OTC.

5.14.2.5. The contractor shall identify previous and planned Cyber Survivability tests that may contribute to current Cyber Survivability efforts.

5.14.2.6. The contractor shall conduct a Cyber Survivability level of test review with Warfare Division and

Code 01D.

5.14.2.7. The contractor shall coordinate with Fleet Resources and the OTD/OTC to determine asset availability in support of Cyber Security objectives.

5.14.2.8. The contractor shall develop and provide a Cyber Survivability POA&M to support the test planning effort.

5.14.2.9. The contractor shall identify and provide risk management and recertification baseline concerns for communication to the Program Management Office (PMO).

5.14.2.10. The contractor shall document the test planning research in a Cyber Survivability Test Plan document.

5.14.2.11. The contractor shall review the test planning research during Cyber Survivability reviews.

5.14.2.12. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of test planning research during Cyber Survivability Test Planning reviews.

5.14.3. Cyber SUT/SoS and Schedule Development.

5.14.3.1. The Contractor shall support Cyber Survivability SUT/SoS and Schedule Development in accordance with Cyber Test Planning Process guidance.

5.14.3.2. The contractor shall identify and document the system attack surface.

5.14.3.3. The contractor shall identify and document the system attack vectors.

5.14.3.4. The contractor shall develop and document Cyber Survivability Test Plan site visit objectives.

5.14.3.5. The contractor shall develop and document the test execution schedule.

5.14.3.6. The contractor shall document the SUT/SoS and Schedule in a Cyber

Survivability Test Plan document.

5.14.3.7. The contractor shall review the SUT/SoS and Schedule during Cyber

Survivability reviews.

5.14.3.8. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of SUT/SoS and Schedule during Cyber Survivability Test Planning reviews.

5.14.4. Cyber Survivability Test Plan Site Visit.

5.14.4.1. The contractor shall support Cyber Survivability Test Plan Site Visit in accordance with Cyber Test Planning Process guidance.

5.14.4.2. The contractor shall coordinate dates, attendance lists, and documentation requirements with the PMO prior to execution of the site visit.

5.14.4.3. The contractor shall develop and conduct an informal site visit in-brief covering the purpose of the site visit and what the site visit personnel should expect.

5.14.4.4. The contractor shall collect and document information to validate initial elements of the critical components list, system configuration differences from provided documentation, and technical SME support requirements for test execution.

5.14.4.5. The contractor shall provide a post-visit trip report (DELIVERABLE A002) documenting site visit activities and actions when contract activities occur away

For Information Warfare (Code 60) At OPTEVFOR Norfolk VA from the OPTEVFOR site in Norfolk, VA.

5.14.5. Cyber Survivability Final Technical Planning.

5.14.5.1. The Contractor shall support Cyber Survivability Final Technical Planning in accordance with Cyber Test Planning Process guidance.

5.14.5.2. The contractor shall develop and document the preliminary set of anticipated attack surfaces and vectors to be used during test execution.

5.14.5.3. The contractor shall develop and document the final Cyber Survivability Critical

Components list to be used for test execution.

5.14.5.4. The contractor shall develop and document the Cyber Survivability test objectives.

5.14.5.5. The contractor shall develop and document any Cyber Survivability test limitations, with corresponding mitigations.

5.14.5.6. The contractor will collect and incorporate go/no-go criteria from Code 01D test personnel.

5.14.5.7. The contractor shall document the final technical planning in a Cyber

Survivability Test Plan document.

5.14.5.8. The contractor shall review the final technical planning during Cyber

Survivability reviews.

5.14.5.9. The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of final technical planning during Cyber Survivability reviews.

5.14.6. Cyber Survivability Test Plan Finalization.

5.14.6.1. The Contractor shall support Cyber Survivability Test Plan Finalization in accordance with Cyber Test Planning Process guidance.

5.14.6.2. The contractor shall develop and document data requirements for test execution.

5.14.6.3. The contractor shall coordinate with Code 01D to determine test equipment and required resources for execution.

5.14.6.4. The contractor shall document the test plan finalization in a Cyber Survivability

Test Plan document.

5.14.6.5. The contractor shall review the test plan finalization during Cyber Survivability

The contractor shall provide meeting minutes, to include any required comment resolution matrix for issues, to document the results of test plan finalization during Cyber Survivability reviews.

5.15. Government Training

5.15.1. The Contractor is eligible to take OPTEVFOR provided training, including Operational Test Director (OTD) Course (OBC), IEF Course, Test Planning Course, Post Test Iterative Process (PTIP) Course and Reliability, Maintainability, Logistic Supportability and Availability (RMLA) Course, Blue/Gold Sheet Writing Course and Cyber Security Test Planning Course. Contractors are encouraged to stay updated on course updates to ensure training currency. All courses provided by OPTEVFOR are filled on a “first come first serve” basis.

5.15.2. Training class availability and registration can be obtained by contacting the Training Department located in CA-495 the primary points of contact are; Workforce Development Officer or Training Administrator. Training class availability and registration can be found on the OPTEVFOR SharePoint page:

https://flankspeed.sharepoint-mil.us/sites/OPTEVFOR/SitePages/01A-Division-Workforce- Development.aspx.