HC102821R0006 - Draft.pdf

CODE

(Hour)

PAGE(S)

until local t ime

X

A X B X C X D

EX

X

G F 234 - 269

270 - 278 X H 279

RATING PAGE OF PAGES

7. ISSUED BY

(Date)

IMPORTANT - Award will be made on this Form, or on Standard Form 26, or by other authorized official written notice.

Previous Edition is Unusable 33-134 STANDARD FORM 33 (REV. 9-97)

Prescribed by GSA FAR (48 CFR) 53.214(c)

1 279

(If other than Item 7)

15A. NAME 16. NAME AND TITLE OF PERSON AUTHORIZED TO

AND

ADDRESS SIGN OFFER (Type or print)

OF

OFFEROR

AMENDMENT NO. DATE

15B. TELEPHONE NO (Include area code) 17. SIGNATURE15C. CHECK IF REMITTANCE ADDRESS

IS DIFFERENT FROM ABOVE - ENTER

SUCH ADDRESS IN SCHEDULE.

18. OFFER DATE

1. THIS CONTRACT IS A RATED ORDER

UNDER DPAS (15 CFR 700)

2. CONTRACT NO.

HC1028 8. ADDRESS OFFER TO

See Item 7

9. Sealed offers in original and copies for furnishing the supplies or services in the Schedule will be received at the place specified in Item 8, or if handcarried, in the depository located in

CAUTION - LATE Submissions, Modifications, and Withdrawals: See Section L, Provision No. 52.214-7 or 52.215-1. All offers are subject to all terms and conditions contained in this solicitation.

10. FOR INFORMATION

CALL:

A. NAME (NO COLLECT CALLS)

11. TABLE OF CONTENTS

SOLICITATION/ CONTRACT FORM

SUPPLIES OR SERVICES AND PRICES/ COSTS

2 - 14

X I CONTRACT CLAUSES

DESCRIPTION/ SPECS./ WORK STATEMENT X

PACKAGING AND MARKING

15 - 138

J LIST OF ATTACHMENTS

INSPECTION AND ACCEPTANCE

DELIVERIES OR PERFORMANCE

140 - 142

143 - 144 X K REPRESENTATIONS, CERTIFICATIONS AND

OTHER STATEMENTS OF OFFERORS

CONTRACT ADMINISTRATION DATA 145 - 155 X

SPECIAL CONTRACT REQUIREMENTS

OFFER (Must be fully completed by offeror) 156 - 164 X M

L INSTRS., CONDS., AND NOTICES TO OFFERORS

EVALUATION FACTORS FOR AWARD

NOTE: Item 12 does not apply if the solicitation includes the provisions at 52.214-16, Minimum Bid Acceptance Period.

is inserted by the offeror) from the date for receipt of offers specified above, to furnish any or all items upon which prices are offered at the price set opposite each item, delivered at the designated point(s), within the time specified in the schedule.

13. DISCOUNT FOR PROMPT PAYMENT

(See Section I, Clause No. 52.232-8)

14. ACKNOWLEDGMENT OF AMENDMENTS

(The offeror acknowledges receipt of amendments

AMENDMENT NO. DATE

to the SOLICITATION for offerors and related documents numbered and dated):

FACILITY

12. In compliance with the above, the undersigned agrees, if this offer is accepted within calendar days (60 calendar days unless a different period

SOLICITATION, OFFER AND AWARD

X

(X) SEC. DESCRIPTION (X) SEC. DESCRIPTION PAGE(S)

PART I - THE SCHEDULE

26. NAME OF CONTRACTING OFFICER (Type or print) 27. UNITED STATES OF AMERICA 28. AWARD DATE

EMAIL:TEL: (Signature of Contracting Officer)

CODE CODE

B. TELEPHONE (Include area code) C. E-MAIL ADDRESS

AWARD (To be completed by Government)

19. ACCEPTED AS TO ITEMS NUMBERED 20. AMOUNT 21. ACCOUNTING AND APPROPRIATION

22. AUTHORITY FOR USING OTHER THAN FULL AND OPEN COMPETITION:

10 U.S.C. 2304(c)( ) 41 U.S.C. 253(c)( ) (4 copies unless otherwise specified)

23. SUBMIT INVOICES TO ADDRESS SHOWN IN ITEM

24. ADMINISTERED BY (If other than Item 7) CODE 25. PAYMENT WILL BE MADE BY CODE

PART IV - REPRESENTATIO NS AND INSTRUCTIO NS

PART III - LIST O F DO CUMENTS, EXHIBITS AND O THER ATTACHMENTS

165 - 232

PART II - CO NTRACT CLAUSES

DISA/DITCO-SCOTT-PL83

2300 EAST DRIVE, BUILDING 3600

SCOTT AFB IL 62225-5406

FAX:

TEL:

FAX:

TEL:

NOTE: In sealed bid solicitations "offer" and "offeror" mean "bid" and "bidder".

SOLICITATION

6. REQUISITION/PURCHASE NO.5. DATE ISSUED4. TYPE OF SOLICITATION

SEALED BID (IFB)

NEGOTIATED (RFP)

[ X ]

3. SOLICITATION NO.

HC102821R0006

Section B - Supplies or Services and Prices

SECTION B

This solicitation is for full and open competition. A Task Order (TO) is included in this solicitation and is anticipated to be awarded with the base contract. The Minimum Guarantee Task Order is for Department of Defense (DOD) Enterprise Test Framework (DETF).

B.1 MAXIMUM AMOUNTS, INDEFINITE-DELIVERY/INDEFINITE QUANTITY (IDIQ)

a. This TEC II Services contract provides technical solutions for the DOD in support of Test and Evaluation (T&E) services and T&E enabling services. JITC’s service requirements are categorized in the following areas:

T&E Performance Requirements, T&E Capability Management Requirements, T&E Capability Requirements, and Overarching Contract and Task Order (TO) Requirements.

• T&E Performance Requirements. Those service requirements enabling JITC to directly support its customers’ test and evaluation needs, including Developmental Test and Evaluation, Operational T&E, Interoperability T&E, Standards Compliance and Conformance T&E, Cybersecurity T&E, Unified Capabilities (UC) T&E, Other T&E and Subject Matter Expert (SME) Support, and Development Security and Operations. The contractor shall possess the technical competencies and mission expertise to provide full T&E lifecycle support to include T&E planning, execution, data collection, analysis, and reporting.

• T&E Capability Management Requirements. Those service requirements enabling JITC to maintain and enhance its core organizational T&E policies, processes, and capabilities.

• T&E Capability Requirements. Those service requirements enabling JITC to identify, research, respond to, and adopt emerging technologies and innovative approaches.

• Overarching Contract and TO Requirements. Those service requirements pertaining to the overall administration of the JITC contract and TOs.

TEC Services are provided to DoD Combatant Commanders/Services/Agencies, federal and state agencies, coalition partners, and commercial entities to aid in the design, development, acquisition, integration, deployment, operational use, and sustainment of new and enhanced Information Technology/National Security System capabilities.

b. This is an IDIQ contract with Fixed-Price (FP), Cost-Reimbursement (CR). Time and Material (T&M), and Labor Hour (LH) type TOs in accordance with FAR Subpart 16.5 - Indefinite- Delivery Contracts. The guaranteed minimum amount is $10,000.00 for the overall IDIQ contract, including all option periods. The minimum guaranteed amount will be satisfied via the TO issued at contract award. The total amount of all orders placed against all contracts resulting from this solicitation shall not exceed $960,000,000.00 over a nine and one half year period: a four-year base period of performance, one (1) five-year option period, and one (1) six-month option period.

c. The Government anticipates a single contract award resulting from this unrestricted competitive solicitation.

B.2 LABOR RATE TABLES

The labor rate tables represent the offeror’s proposed fully-burdened hourly labor rates for each skill classification for work performed at Government sites and work performed at Contractor sites. Labor rates proposed for the TOs cannot exceed these labor rates.

B.3 INCORPORATION OF PROPOSED PRICING

The offeror’s proposed pricing, labor rate tables, test menus submitted in response to the solicitation shall be incorporated into the contract as attachments to Section B.

Pricing for all cost-reimbursement contract actions (including, but not limited, to proposal and billing of future TOs and modifications) shall be restricted to the labor categories and fully burdened labor rates established in the CPFF Pricing Scenario tab of the Pricing Workbook Template, found in Section J.

Pricing for all FFP contract actions (including, but not limited to, proposal and billing of future TOs and modifications) shall be restricted to the labor categories and fully burdened labor rates established in the FFP Pricing Scenario tab of the Pricing Workbook Template found in Section J.

B.4 CONTRACT DATA REQIREMENTS LIST (CDRL’s) Contractors are advised that data items listed on the DD Form 1423, CDRL's, shall not be separately priced.

B.5 COST/PRICE RESTRICTIONS

The Government will not reimburse the contractor for proposal preparation costs in response to this solicitation nor for future proposals prepared in response to future contract actions under this IDIQ contract.

B.5.1 OTHER DIRECT COSTS (ODCs) Profit and/or fixed fee are not allowed on ODCs. ODCs shall be authorized by the Contracting Officer’s Representative (COR), in writing, prior to performance and travel will be paid IAW FAR 31.205-46 unless otherwise specified. Relocation expenses shall not be reimbursed as an ODC unless the Government directs existing contractor personnel to move due to changing Government requirements.

For cost-reimbursable ODC estimates the contractor may be provided a Government estimate in future TO Request for Proposals for travel, materials, equipment, and OCONUS allowances (as defined by the Department of State guidelines) that are incidental to the services being performed.

The cost of general-purpose items required for the conduct of the contractor’s normal business operations will not be considered an allowable ODC in the performance of this contract. These EXACT amounts include all markups and burdens allowed by the contract and shall not be adjusted based upon independent decisions regarding actual anticipated costs. Should there be additional ODCs related to labor necessary to execute to the contract requirements, these costs must be SEPARATELY identified and shall be included in the Offeror's total evaluated price proposal as "additional ODCs" in future task order proposals. The proposed additional ODCs must be broken out in sufficient detail to identify the methodology and supporting documents employed to derive the proposed costs. Additional ODCs that are not proposed, or otherwise not adequately substantiated, will be considered unallowable costs and under no circumstance be reimbursed to the contractor on future task orders.

B.5.2 PROGRAM MANAGEMENT SUPPORT COSTS AND OTHER INDIRECT COSTS

Contract-level program management support costs shall be included within the fully-burdened labor category rates and encompass support for contract-level management, reporting requirements, and related travel and meeting attendance costs associated with the contractor's program management staff, as it relates to overall management of TEC II. As a result, these program management support costs are allocated as an indirect expense and may not be charged directly to any task order.

These “program management” support costs are differentiated from individual task order “Program Manager” and “Project Manager” support costs, which are billed as hourly labor rates against individual task orders for direct support to the effort performed under those task orders. This will result in direct billings at the task order level for labor hours in the “Program Manager” and “Project Manager” category, to specifically support program or project management for the task order.

Indirect costs shall consist of incurred costs identified with two or more cost objectives but not identified specifically with any final cost objective. These include, but are not limited to, costs such as fringe (bonuses), General and Administrative (G&A), overhead, and executive salaries.

B.6 SCHEDULE OF RATES (added at award)

ITEM NO SUPPLIES/SERVICES QUANTITY UNIT UNIT PRICE AMOUNT

TEC Services - Firm Fixed Price

FFP

Firm Fixed Price (FFP) All Variations - This CLIN applies to all variations of contract types under fixed price.

Period of Performance: Four (4) year base period - Contract Years 1 through 4.

FOB: Destination

NET AMT

TEC Services - Cost

COST

Cost Reimbursable (CR) All Variations - This CLIN applies to all variations of contract types under cost reimbursable.

ESTIMATED COST

TEC Services - Time and Materials

T&M Time and Material (T&M) - This CLIN applies to all variations of contract types under T&M.

TOT ESTIMATED PRICE

CEILING PRICE

ITEM NO SUPPLIES/SERVICES ESTIMATED

QUANTITY

UNIT UNIT PRICE AMOUNT

TEC Services - Labor Hours

LH

Labor Hours (LH) - This CLIN applies to all variations of contract types under

LH.

Other Direct Costs (ODCs)

COST

Cost Reimbursable ODCs

TEC Services - UC TEST MENU

FFP

Requirements shall be completed in accordance with the Performance Work Statement, contract clauses, and contract requirements.

Requirements will be awarded through the issuance of individual task orders that will indicate the specific CLIN, quantity, duration, price, invoicing instructions.

TEC Services - RTF TEST MENU

FFP

Requirements shall be completed in accordance with the Performance Work Statement, contract clauses, and contract requirements.

Requirements will be awarded through the issuance of individual task orders that will indicate the specific CLIN, quantity, duration, price, invoicing instructions.

OPTION TEC Services - Firm Fixed Price

FFP

Firm Fixed Price (FFP) All Variations - This CLIN applies to all variations of contract types under fixed price.

Period of Performance: Five (5) year option period - Contract Years 5 through 9.

OPTION TEC Services - Cost

COST

Cost Reimbursable (CR) All Variations - This CLIN applies to all variations of contract types under cost reimbursable.

OPTION TEC Services - Time and Materials

T&M Time and Material (T&M) - This CLIN applies to all variations of contract types under T&M.

UNIT UNIT PRICE AMOUNT

OPTION TEC Services - Labor Hours

LH

Labor Hours (LH) - This CLIN applies to all variations of contract types under

LH.

OPTION Other Direct Costs (ODCs)

COST

Cost Reimbursable ODCs

OPTION TEC Services - UC TEST MENU

FFP

Requirements shall be completed in accordance with the Performance Work Statement, contract clauses, and contract requirements.

Requirements will be awarded through the issuance of individual task orders that will indicate the specific CLIN, quantity, duration, price, invoicing instructions.

OPTION TEC Services - RTF TEST MENU

FFP

Requirements shall be completed in accordance with the Performance Work Statement, contract clauses, and contract requirements.

Requirements will be awarded through the issuance of individual task orders that will indicate the specific CLIN, quantity, duration, price, invoicing instructions.

OPTION TEC Services - Firm Fixed Price

FFP

Firm Fixed Price (FFP) All Variations - This CLIN applies to all variations of contract types under fixed price.

Period of Performance: six (6) month option period - Contract Period 10.

OPTION TEC Services - Cost

COST

Cost Reimbursable (CR) All Variations - This CLIN applies to all variations of contract types under cost reimbursable.

Period of Performance: Six (6) month option period - Contract Period 10.

OPTION TEC Services - Time and Materials

T&M Time and Material (T&M) - This CLIN applies to all variations of contract types under T&M.

UNIT UNIT PRICE AMOUNT

OPTION TEC Services - Labor Hours

LH

Labor Hours (LH) - This CLIN applies to all variations of contract types under

LH.

OPTION Other Direct Costs (ODCs)

COST

Cost Reimbursable ODCs

Period of Performance: Six (6) month option period - Contract Period10.

OPTION TEC Services - UC TEST MENU

FFP

Requirements shall be completed in accordance with the Performance Work Statement, contract clauses, and contract requirements.

Requirements will be awarded through the issuance of individual task orders that will indicate the specific CLIN, quantity, duration, price, invoicing instructions.

OPTION TEC Services - RTF TEST MENU

FFP

Requirements shall be completed in accordance with the Performance Work Statement, contract clauses, and contract requirements.

Requirements will be awarded through the issuance of individual task orders that will indicate the specific CLIN, quantity, duration, price, invoicing instructions.

CONTRACT MINIMUM/MAXIMUM QUANTITY AND CONTRACT VALUE

The minimum quantity and contract value for all orders issued against this contract shall not be less than the minimum quantity and contract value stated in the following table. The maximum quantity and contract value for all orders issued against this contract shall not exceed the maximum quantity and contract value stated in the following table.

MINIMUM

QUANTITY

MINIMUM

AMOUNT

MAXIMUM

QUANTITY

MAXIMUM

AMOUNT

1.00 $10,000.00 1,000.00 $960,000,000.00

DELIVERY/TASK ORDER MINIMUM/MAXIMUM QUANTITY AND ORDER VALUE

The minimum quantity and order value for each Delivery/Task Order issued shall not be less than the minimum quantity and order value stated in the following table. The maximum quantity and order value for each Delivery/Task Order issued shall not exceed the maximum quantity and order value stated in the following table.

MINIMUM

QUANTITY

MINIMUM

AMOUNT

MAXIMUM

QUANTITY

MAXIMUM

1.00 $500.00 1,000.00 $20,000,000.00

CLIN DELIVERY/TASK ORDER MINIMUM/MAXIMUM QUANTITY AND CLIN ORDER VALUE

The minimum quantity and order value for the given Delivery/Task Order issued for this CLIN shall not be less than the minimum quantity and order value stated in the following table. The maximum quantity and order value for the given Delivery/Task Order issued for this CLIN shall not exceed the maximum quantity and order value stated in the following table.

CLIN

MINIMUM

QUANTITY

MINIMUM

AMOUNT

MAXIMUM

QUANTITY

MAXIMUM

Section C - Descriptions and Specifications

C.1. - PWS

PERFORMANCE WORK STATEMENT (PWS)

Solicitation/Amendment Effective Version Date Solicitation Date of Solicitation Issuance

Contract Number: (completed by the KO at time of contract or TO award) Task Order (TO) Number: (completed by the KO at time of TO award or N/A) Contractor Name (completed by the KO at time of contract or TO award) Tracking Number: 832016230 Follow-on to Previous Contract Number:

HC102817D0006

1. Contracting Officer’s Representative (COR).

a. Primary COR.

Name: (completed by the KO at time of contract or TO award) Organization:

DODAAC:

Address:

Phone Number:

Fax Number:

E-Mail Address:

b. Alternate COR.

Department of Defense Activity Address Code (DODAAC):

Address:

Phone Number:

c. Alternate COR.

DODAAC:

Address:

Phone Number:

d. Property Administrator

DODAAC:

Address:

Phone Number:

2. Contract Title. Test, Evaluation, and Certification (TEC) II Services (TEC II)

3. Background.

The Defense Information Systems Agency (DISA)/ Joint Interoperability Test Command (JITC) TEC II Services is a follow-on to the DISA/JITC TEC Services contract, valued at $480,000,000.00. The DISA/JITC TEC Services contract was awarded using full and open competition. It is a single award Indefinite-Delivery/Indefinite-Quantity (ID/IQ) contract with Firm-Fixed-Price and Cost-Plus-Fixed-Fee TOs.

This PWS addresses the Test and Evaluation (T&E) and T&E Enabling Service Requirements of the DISA, JITC.

JITC is the Department of Defenses (DoD's) sole Joint Interoperability Certifier and only non-Service Operational Test Agency (OTA) for Information Technology/National Security Systems (IT/NSS). JITC provides its T&E services to the DoD Combatant Commands/Services/Agencies (CC/S/A); federal, state, and local Governmental agencies; United States (US) Coalition Partners; and commercial vendors and service providers. JITC provides these services to aid in mitigating the risks associated with designing, developing, acquiring, integrating, deploying, using, and sustaining new and enhanced IT/NSS capabilities. JITC’s T&E services help to ensure that joint warfighting Information Technology (IT) capabilities are interoperable and enable mission needs by:

• Determining to what extent applicable requirements and standards have been satisfied

• Identifying and helping to reduce technical and operational risks

• Baselining and characterizing levels of performance

• Documenting operational capabilities and limitations

• Inform acquisition, deployment, and operational use milestone decisions

General information concerning DISA’s and JITC’s missions, functions, organizational structures, operational capabilities, and service offerings are available through their respective public websites, as are regulations, directives, instructions, manuals, policies, and guidelines referenced in this PWS. The public website addresses are:

• DISA: http://www.disa.mil

• JITC: http://jitc.fhu.disa.mil

4. Objectives. The objective of the TEC II Services contract is for JITC to carry out unique T&E roles and responsibilities for which it is assigned within the DoD:

• Joint Interoperability Certifier – JITC has sole responsibility for certifying joint and combined interoperability of all DoD IT and NSS. JITC performs critical interoperability and standards conformance T&E to reduce the risks faced by warfighters in the field. JITC often provides evaluation and certification services based in part or in whole on testing conducted by other organizations.

• OTA – JITC is the OTA for DISA-managed programs. JITC also serves as the OTA for other DoD agencies such as the Defense Logistics Agency, the Defense Finance and Accounting Service (DFAS), the Defense Commissary Agency, the Defense Health Agency, the United States Special Operations Command, and the National Security Agency. As an OTA, JITC conducts Operational Test and Evaluation (OT&E) to assess system operational effectiveness, suitability, interoperability, and cybersecurity. JITC is the sole non-Service OTA recognized by the DoD Director, Operational Test and Evaluation (DOT&E).

http://www.disa.mil/ http://jitc.fhu.disa.mil/

• Warfighter Support – JITC aids the CC/S/As during exercises and contingency operations, helping the warfighter with on-the-spot evaluations of problem areas and viable mission-oriented solutions to include providing 24x7-hotline response.

• Major Range and Test Facility Base (MRTFB) – JITC is the only non-Service member of the 23-element MRTFB. As such, JITC is considered a national asset focused on testing IT/NSS. Under MRTFB guidelines, JITC must account for all direct costs at the job order level. MRTFB services provided include T&E capabilities, infrastructure, and resources to respond to DoD acquisition programs and other Governmental and commercial customers on a reimbursable basis.

• Developmental Test and Evaluation (DT&E) – JITC provides a full range of DT&E services. Through these services, JITC responds to DISA and other customers in their delivery of verified, reliable, fully functional, and secure technical solutions.

The TEC II services contract will provide JITC with T&E products and services that reflect a higher level of quality, innovation, and technical sophistication than JITC has required in the past. This greater emphasis on quality, innovation, and technical sophistication will raise the trust that customers have in JITC’s capabilities, with the end goal of increasing the demand for JITC’s value-added TEC services. To achieve this goal, JITC requires a contract that is staffed with a highly qualified, technically competent, professionalized workforce possessing knowledge and experience in many diverse technical areas.

5. Scope. The scope of the TEC II Services requirements includes T&E planning, test conducting, data collection and analysis, and results reporting for a wide range of functional areas and technologies to include: critical Command, Control, Communications, Computers, Combat Systems, Intelligence, Surveillance and Reconnaissance (C5ISR); defense enterprise business, logistics, and medical systems; emerging network, mobility, cloud, and Department of Defense Information Network (DoDIN) capabilities; cybersecurity, tactical, strategic, and national intelligence systems; and other IT/NSS capabilities. IT/NSS capabilities include infrastructure and environments such as the DoD Joint Information Environment which use both traditional waterfall development and deployment processes, as well as more modern Agile/Development Security Operations (DevSecOps) processes. JITC’s TEC II Services needs also include contractor-provided maintenance of the underlying policies, processes, and capabilities that enable JITC to satisfy its customers’ T&E needs in a responsive and cost effective manner. In addressing these needs, JITC develops and applies proven methods, tools, instrumentation, automation, modeling, simulation, statistical and analytical methods, and other advanced technologies and techniques to ensure the T&E rigor needed is in place to mitigate the risks and uncertainty its customers face in developing, fielding, operating, and sustaining IT/NSS capabilities. Surge support may be required and will be determined at the TO level.

In addition to this TEC II Services contract, DISA has two other large contracts, which provide T&E services: the System Engineering, Technology, and Innovation (SETI) contract and the Encore III contract. By design, neither the SETI nor Encore III contracts provide DISA OT&E or DoD Joint Interoperability Certification services. Both types of these T&E services are the exclusive responsibility of JITC to perform, and in turn, the TEC II Services contract to perform. The TEC II Services contract will complement the T&E services available through the SETI and Encore III contracts. All three contracts encourage the use of integrated testing, the reuse and sharing of T&E data, and the coordination of test events that enable the collection of data to efficiently address multiple T&E objectives.

Evolving DoD priorities and missions, coupled with continuous technological advancements, result in ever-evolving T&E requirements and challenges. Given this uncertainty and the rapid pace of change, it is important that the TEC II Services contract provide the depth, breadth, and flexibility of services required by JITC to remain relevant and responsive in satisfying its customers’ individual and unique T&E needs throughout the life of the contract.

In 2009, DoD OTA Commanders signed a memorandum of agreement committing to the use of Design of Experiments (DOE) in OT&E. In 2010, the DOT&E issued a memorandum requiring the OTAs to use scientific and statistical methods in developing rigorous and defensible test plans and in thoroughly evaluating resulting test data. The TEC II Services Contractor will be required to implement the 'Operational Test Agencies Six Core Test Principles' memo from DOT&E dated 31 May 2019.

The Deputy Assistant Secretary of Defense, Developmental Test and Evaluation (DASD(DT&E)) in turn established the term “Scientific Test and Analysis Techniques (STAT)” to refer to the use of scientific and statistical methods for the purposes of developing efficient, rigorous test strategies that yield highly accurate and repeatable results. As an OTA and provider of both DT&E and OT&E services, JITC has embraced the use of these DOT&E and DASD (DT&E) endorsed approaches. Online sources for information pertaining to Science Based Test Design (SBTD) concepts and approaches include the DOT&E website, http://www.dote.osd.mil/; the DASD(DT&E) sponsored STAT T&E Center of Excellence, www.afit.edu/STAT/; and the DOT&E sponsored Test Science Knowledge Center, www.testscience.org.

6. Performance Requirements.

To operate in an effective, efficient, and responsive manner, JITC requires a coordinated unity of effort across the full spectrum of T&E services it provides its customers. In satisfying JITC’s customers’ T&E needs, the scope of the TEC II Services contract must address requirements in the following distinct but interrelated performance areas:

• T&E Performance Requirements – Those service requirements enabling JITC to directly fulfill its customers’ T&E needs.

• T&E Capability Management Requirements – Those service requirements enabling JITC to maintain and enhance its core organizational T&E policies, processes, and capabilities.

• T&E Capability Requirements – Those service requirements enabling JITC to identify, research, respond to, and adopt emerging technologies and innovative approaches.

• Overarching Contract and TO Requirements – Those service requirements pertaining to the overall administration of the JITC contract and TOs.

To ensure JITC and its customers realize maximum benefit from the finite resources available to perform T&E requirements, the Contractor must possess the knowledge, skills, and abilities to seamlessly and efficiently perform all performance areas. The specific form, format, and content for the deliverables associated with each of the performance areas will be specified in individual TOs. The general requirements associated with these performance areas are as follows.

6.1 T&E Performance Requirements.

The following performance requirement areas identify the services JITC requires to ensure “unity of effort” in meeting its customers’ T&E needs:

• Performance Area 1.1: DT&E

• Performance Area 1.2: OT&E

• Performance Area 1.3: Interoperability T&E

• Performance Area 1.4: Standards Compliance and Conformance T&E

• Performance Area 1.5: Cybersecurity T&E

• Performance Area 1.6: Unified Capabilities (UC) T&E

• Performance Area 1.7: Other T&E and Subject Matter Expert Support

• Performance Area 1.8: DevSecOps

The DoD Chief Information Officer (CIO), Joint Staff (JS), DOT&E, Defense, Deputy Assistant Secretary of Defense for Developmental Test & Evaluation, CC/S/A level Milestone Decision Authorities, Component Acquisition Executives, and Program Managers (PMs) for DoD Acquisition Category (ACAT) and non-ACAT programs are all reliant on JITC and its T&E services to perform various program oversight functions. JITC’s T&E results and findings serve as critical input to program acquisition and deployment decisions and to particular program review board processes such as Network Readiness Review Boards and Acquisition Review Boards.

Because of the importance of the information JITC provides to leaders providing program oversight and decision processes, it is vital that the Contractor is thorough, objective, and fully responsive in providing the T&E services required in this PWS. It is equally vital that the products and deliverables the Contractor produces in providing http://www.afit.edu/STAT/ http://www.testscience.org/ these T&E services be of high quality, free of defects and inaccuracies, and fully compliant with all DoD, DISA, JITC, and other applicable policies, instructions, guidelines, and reporting and formatting requirements.

The Contractor shall possess the technical competencies and mission expertise to provide full T&E lifecycle services to include T&E planning, execution, data collection, analysis, and reporting:

• T&E Planning – As part of T&E planning, the Contractor shall review and comment on system/program requirements; recommend specific T&E scope and objectives; perform in-depth analyses and risk assessments to identify specific T&E requirements and data sources; review testing conducted by other elements outside of JITC to help determine what data can be re-used and identify gaps to help focus JITC efforts; establish effective strategies based on SBTD approaches, automation, and instrumented data collection methods to reliably achieve T&E objectives; develop T&E documentation; and plan for and coordinate the activities needed to carry out these strategies and plans in a rigorous, disciplined, and cost efficient fashion.

• T&E Execution and Data Collection – In T&E execution, the contractor shall conduct the test and ensure the required data is collected according to approved T&E strategies and plans, including the effective implementation of manual and automated data collection processes, and the performance of those activities required to manage, validate, and reduce collected data. Throughout T&E execution, the Contractor shall also identify and mitigate any limitations with the potential to affect the accomplishment of T&E objectives.

• T&E Analysis – The Contractor shall apply systematic and rigorous statistical methods and analytical tools to thoroughly evaluate collected data to efficiently establish T&E objectives and ensure the resulting conclusions are at the confidence levels targeted during the T&E planning phase. In analyzing the collected data, the Contractor shall focus on documenting the operational significance and mission impact of any capabilities and limitations associated with the System Under Test (SUT).

• T&E Reporting – In performing T&E reporting functions, the Contractor shall apply appropriate methods to clearly, concisely, and objectively document T&E results and findings in a timely and responsive manner, delivering high quality test reports that comply with JITC-approved processes and satisfy end-customer information needs. Reporting of T&E results should emphasize the operational significance of test findings, and when possible, provide actionable value-added recommendations.

The Contractor shall have extensive knowledge of the functional and operational requirements, intended users, mission needs, and environmental conditions that the systems, technologies, and devices undergoing T&E are intended to fulfill. The Contractor shall possess a thorough understanding of the Tactics, Techniques, and Procedures, Concepts of Operation, Standard Operating Procedures (SOPs), Military and Commercial Standards, and operational issues pertaining to each SUT. The Contractor shall apply this understanding to help specify and establish the operationally representative test environments and conditions needed to effectively determine to what extent each SUT satisfies operational mission needs. The Contractor shall apply techniques such as mission thread analysis and decomposition, message traffic generation, models and simulation, stress and load testing, and other techniques to provide operational realism and mission context in the conduct of testing.

In providing T&E services, the Contractor shall identify and apply innovative methods and technologies to improve the overall efficiency and cost effectiveness by which JITC satisfies its customers’ T&E needs. In executing specific T&E objectives, the Contractor shall consider all valid data sources, which includes data collected by JITC or other Governmental and non-Governmental organizations through dedicated test events, technology demonstrations, exercises, actual system operations, or other relevant sources of information. To the extent possible, the Contractor is encouraged to propose and apply automated and instrumented capabilities that will both lead to affordable and accurate results and establish processes and capabilities that can be reused to satisfy similar T&E requirements in the future. To improve overall T&E efficiency, the Contractor should exploit all opportunities for integrated test events led by JITC or other testing organizations that allow multiple T&E objectives to be concurrently and cost effectively achieved.

To ensure accurate and defensible T&E outcomes, and to comply with DOT&E Assistant Secretary of Defense for Research & Engineering mandates for increased T&E rigor and efficiency, the Contractor shall adopt SBTD methods as part of their execution of JITC’s TEC II Services requirements. Methods such as regression analysis, DOE, Analysis of Variance and other statistical approaches, combinatorial test techniques, and test automation and instrumentation should be considered as means to accurately and efficiently evaluate the performance characteristics of systems or products undergoing test, over the full range of required operating conditions. The Contractor shall help define formal analysis structures and evaluation frameworks as the basis for a program’s T&E, to ensure that T&E is performed in a thorough, systematic, and structured manner. In T&E planning, the Contractor shall make use of SBTD approaches to determine the relevant combination of factors to be systematically controlled, varied, or simply recorded to achieve efficient test designs. For each test design, the Contractor must justify test data requirements by citing statistical measures of merit (applicable statistical models, measures of orthogonality, prediction variance, power, confidence, etc.). Each test design must also account for the statistical methods of analysis to be used to determine if correlation exists between changes in test conditions to measured variations in performance. In cases where non-statistical methods are used to achieve particular T&E objectives, the Contractor shall justify such test design choices and explain how sufficient statistical confidence will be achieved, based on the available data. To mitigate physical or resource constraints, the Contractor shall make use of computer models and simulations, virtualized test environments, and other innovative valid approaches for cost effectively satisfying particular T&E objectives, where appropriate. To provide accurate and well-informed post-test analysis and reporting, the Contractor shall determine the statistical uncertainty and levels of confidence associated with test findings. The Contractor shall document SBTD activities, results, and findings in a manner that is clear, concise, and easily understood by JITC customers who may not possess an in-depth knowledge of technical, scientific, or statistical concepts.

The TEC II Service contract T&E requirements are divided into eight performance areas. Many JITC customers require T&E services in more than one of these performance areas. In these cases, the Contractor shall satisfy the requirements in the most efficient and cost effective manner possible, using methods such as combined or integrated test events, where appropriate. The eight T&E requirement performance areas follow.

6.1.1 Performance Area 1.1: DT&E.

The Contractor shall perform planning, conducting, and reporting on DT&E. The Contractor shall verify the design, development, integration, and technical characteristics of new or enhanced IT/NSS capabilities prior to their acceptance and consideration for operational use. These DT&E services will be provided for capabilities being delivered by developers using both conventional development methods, as well as advanced methods such as Agile/SCRUM and Test Driven Design approaches. The Contractor shall apply SBTD methods such as combinatorial test methods, instrumentation and automation, and other advanced approaches, where appropriate, to improve the overall efficiency, effectiveness, rigor, and timeliness of JITC’s DT&E activities. The Contractor may provide DT&E services to DISA programs pertaining to enterprise services, cloud computing, mobility, UC, and other emerging technologies being adopted by DoD. As part of this service, the Contractor shall attend and provide subject matter expertise to advise Working-level Integrated Product Teams (WIPTs) to facilitate program T&E planning and provide DT&E input to programs’ Test and Evaluation Master Plan (TEMP). The DT&E services the Contractor shall provide includes:

6.1.1.1 Unit, Module, Device, Subsystem, and System Testing.

The Contractor shall perform DT&E planning, execution, data collection and analysis, and reporting to verify that software and hardware modules, devices, subsystems, and systems undergoing development satisfy their stated requirements for functionality, interoperability, standards conformance, performance, capacity, scalability, reliability, availability, maintainability, cybersecurity, and other required technical characteristics as specified in applicable requirements sources and design specifications. The Contractor shall perform risk assessments to help focus DT&E resources on areas of highest technical risk. The Contractor shall conduct regression testing to verify that previously evaluated system characteristics are not adversely affected by subsequent configuration changes.

The Contractor shall provide services that include:

• Develop DT&E strategies and methodologies

• Develop Risk Assessment plans and reports

• Develop Requirements Traceability Matrices (RTMs)

• Develop use cases, test cases, detailed test procedures, and test scripts

• Develop DT&E Test Plans

• Develop Product Deficiency Reports

• Develop DT&E Test Reports

• Provide input to the TEMP and/or Test and Evaluation Strategy (TES)

6.1.1.2 Integration and Qualification Testing.

The Contractor shall perform T&E planning, execution, data collection and analysis, and reporting to verify that software and hardware modules, devices, subsystems, and systems perform as intended when integrated together in an approved system design or architecture. The Contractor shall perform risk assessments to help focus T&E resources on areas of highest technical and operational risk. The Contractor shall perform T&E to determine the degree to which the integrated system satisfies applicable design requirements for functionality, interoperability, standards conformance, performance, capacity, scalability, reliability, availability, maintainability, cybersecurity, and other required technical characteristics as specified in applicable requirements sources and design specifications.

The Contractor shall conduct regression testing to verify that previously evaluated system characteristics are not adversely affected by subsequent configuration changes. The Contractor shall provide services that include:

• Develop Integration/Qualification strategies and methodologies

• Develop risk assessment plans and reports

• Develop RTMs

• Develop use cases, test cases, detailed test procedures, and test scripts

• Develop and present Test Concept Briefs (TCBs)

• Develop Integration/Qualification Test Plans

• Develop Product Deficiency Reports

• Develop Integration/Qualification Test Reports

• Provide input to the TEMP and/or TES

6.1.1.3 Acceptance Testing.

The Contractor shall perform T&E planning, execution, data collection and analysis, and reporting to verify that the developer has satisfied applicable functional requirements. Such testing may take the form of a software/system acceptance test, government acceptance test, or user acceptance test; normally involves test participation by representatives of the system’s intended user community; and precedes any Government commitment to limited deployment and formal OT&E. The Contractor shall conduct regression testing to verify that previously evaluated system characteristics are not adversely affected by subsequent configuration changes. The Contractor shall provide services that include:

• Develop Integration/Qualification strategies and methodologies

• Develop Risk Assessment plans and reports

• Develop RTMs

• Develop TCBs

• Develop use cases, test cases, detailed test procedures, and test scripts

• Develop Acceptance Test Plans

• Develop Product Deficiency Reports

• Develop Acceptance Test Reports

• Provide input to the TEMP and/or TES

6.1.2 Performance Area 1.2: OT&E.

The Contractor shall perform OT&E planning, conduct, data collection and analysis, and reporting for purposes of assessing or determining the operational effectiveness, suitability, and survivability of the IT/NSS SUT (OT&E survivability requirements are addressed in PWS section 6.1.5.4, OT&E Cyber Survivability). JITC performs such OT&E activities to inform program acquisition decisions, both for programs relying on traditional waterfall development approaches as well as those utilizing Agile/DevSecOps development methods. The Contractor shall provide these services IAW: USC Title 10, Section 139 requirements, latest DOT&E guidance and directives (https://www.dote.osd.mil/), and per the processes set forth in the JITC OT&E Guidebook (http://jitc.fhu.disa.mil/).

In providing these services, the contractor shall apply the 'Operational Test Agencies Six Core Test Principles' presented in the OTA Commanders’ memo to DOT&E, 31 May 2019. Where appropriate, the Contractor shall apply SBTD approaches, T&E automation, instrumentation, simulation, and stimulation methods to ensure efficient collection of quantifiable, operationally representative, and highly accurate test data that feeds into OT&E objectives with increased confidence and mission relevance. OT&E activities may take the form of either Operational Assessments (OAs), to include Field Assessments (FAs) and Limited User Tests (LUTs); or Operational Tests (OTs), to include Initial Operational Test and Evaluation or Follow-On Operational Test and Evaluation. Whether an OA or an OT, OT&E necessitates evaluation based on typical trained users and maintainers exercising an operationally representative system configuration, operated under realistic conditions, to determine to what extent the SUT satisfies operational mission needs. The Contractor shall provide services that include:

• Identify operational requirements through review of program documentation such as Joint Capabilities Integration and Development System (JCIDS) documents and other requirements sources.

• Perform mission analysis through review of applicable Joint Capability Areas (JCAs), Universal Joint Task Lists, Mission Essential Task Lists, Mission Essential Tasks, mission threads, use cases, etc.

• Conduct risk assessments to determine appropriate levels of OT&E.

• Develop OT&E concepts and strategies.

• Document OT&E strategies within TCBs and TEMPs.

• Attend and provide input to T&E, User Working Groups, and WIPTs with stakeholders (PMs, DOT&E representatives, user representatives, T&E participants, etc.) to plan OT&E activities and resource requirements.

• Develop OT&E Analysis Structures composed of an evaluation framework and a data source matrix.

• Capitalize on existing relevant test or operational data, and opportunities for integrated and/or combined testing where DT&E, Cyber, or other relevant T&E events can be leveraged to feed OT&E objectives, thus potentially reducing the scope, duration, expense, and delay associated with dedicated phases of OT&E.

• Construct user surveys, questionnaires, interview forms, and other qualitative and quantitative data collection instruments.

• Employ SBTD methods, automation, and instrumentation as means to carry out OT&E objectives in an efficient and rigorous manner.

• Develop detailed OT&E test plans.

• Attend and provide input to Operational Test Readiness Reviews (OTRR).

• Conduct pilot tests, not for the OT&E of the SUT, but to verify and ‘dry run’ test and data collection procedures, etc., prior to the start of formal phases of testing.

• Carry out OT&E execution and data collection in accordance with approved test plans.

• Provide input to ‘hot washes,’ test coordination meetings, and test status reporting throughout the conduct of OT&E events.

• Attend and provide input to Data Authentication Group (DAG) functions performed to authenticate and validate test data.

• Analyze collected data to arrive at an operational effectiveness, suitability, and cyber survivability determination.

• Prepare final reports and briefings to convey OT&E results and findings to Senior Executive Service (SES) and/or General Level officials.

• Develop OT&E TCBs

• Provide OT&E input to program TEMPs

• Develop Analysis Structures

• Develop Risk Assessments Plans and Reports

• Develop OT&E Test Plans

• Develop OTRR Reporting

• Develop Operational Quick Look Reports

• Develop OT&E Reports

• Develop OT&E Results Briefings

• Develop DAG Reports

6.1.2.1 Traditional OT&E.

The Contractor shall perform T&E planning, execution, data collection and analysis, and reporting to address the objectives of assessing or determining operational effectiveness, suitability, and cyber survivability. JITC conducts such testing for major program acquisition decisions. OT&E activities may take the form of either OAs, to include FAs and LUTs; or OTs, to include Initial Operational Test and Evaluation or Follow-On Operational Test and Evaluation. Whether an OA or an OT, OT&E necessitates evaluation based on typical trained users and maintainers exercising an operationally representative system configuration, operated under realistic conditions, to determine to what extent the SUT satisfies operational mission needs. In providing services to OT&E activities, the Contractor shall follow the processes set forth in the JITC OT&E Guidebook, accessible via the JITC public website at http://jitc.fhu.disa.mil/. The Contractor shall also be responsive to the latest guidance from DOT&E, accessible at http://www.dote.osd.mil/guidance.html. The Contractor shall provide services that include:

• Identify operational requirements through review of program documentation such as JCIDS documents and other requirements sources.

• Perform mission analysis through review of applicable JCAs, Universal Joint Task Lists, Mission Essential Task Lists, Mission Essential Tasks, mission threads, use cases, etc.

• Conduct risk assessments to determine appropriate levels of OT&E.

• Develop OT&E concepts and strategies.

• Document OT&E strategies within TCBs and TEMPs.

• Attend and provide input to T&E, User Working Groups, and WIPTs with stakeholders (PMs, DOT&E representatives, user representatives, T&E participants, etc.) to plan OT&E activities and resource requirements.

• Develop OT&E Analysis Structures composed of an evaluation framework and a data source matrix.

• Capitalize on existing relevant test or operational data, and opportunities for integrated and/or combined testing where DT&E, Cyber survivability, or other relevant T&E events can be leveraged to address OT&E objectives, thus potentially reducing the scope, duration, expense, and delay associated with dedicated phases of OT&E.

• Construct user surveys, questionnaires, interview forms, and other qualitative and quantitative data collection instruments.

• Employ SBTD methods, automation, and instrumentation as means to carry out OT&E objectives in an efficient and rigorous manner.

• Develop detailed OT&E test plans.

• Attend and provide input to OTRR.

• Conduct pilot tests, not for the OT&E of the SUT, but to verify and ‘dry run’ test and data collection procedures, etc., prior to the start of formal phases of testing.

• Carry out OT&E execution and data collection in accordance with approved test plans.

• Provide input to ‘hot washes,’ test coordination meetings, and test status reporting throughout the conduct of OT&E events.

• Attend and provide input to DAG functions performed to authenticate and validate test data.

• Analyze collected data to arrive at an operational effectiveness, suitability, and cyber survivability determination.

• Prepare final reports and briefings to convey OT&E results and findings to SES and/or General Officer

Level officials.

6.1.2.2 Capability Assessments.

The…