TEC_Draft_RFP_-_2_June.pdf

CODE

(Hour)

PAGE(S)

until local t ime

X

A X B X C X D

EX

X

G F 146 - 159

160 - 185 X H 186 - 193

RATING PAGE OF PAGES

7. ISSUED BY

(Date)

IMPORTANT - Award will be made on this Form, or on Standard Form 26, or by other authorized official written notice.

Previous Edition is Unusable 33-134 STANDARD FORM 33 (REV. 9-97)

Prescribed by GSA FAR (48 CFR) 53.214(c)

1 193

(If other than Item 7)

15A. NAME 16. NAME AND TITLE OF PERSON AUTHORIZED TO

AND

ADDRESS SIGN OFFER (Type or print)

OF

OFFEROR

AMENDMENT NO. DATE

15B. TELEPHONE NO (Include area code) 17. SIGNATURE15C. CHECK IF REMITTANCE ADDRESS

IS DIFFERENT FROM ABOVE - ENTER

SUCH ADDRESS IN SCHEDULE.

18. OFFER DATE

1. THIS CONTRACT IS A RATED ORDER

UNDER DPAS (15 CFR 700)

2. CONTRACT NO.

HC1028 8. ADDRESS OFFER TO

See Item 7

9. Sealed offers in original and copies for furnishing the supplies or services in the Schedule will be received at the place specified in Item 8, or if handcarried, in the depository located in

CAUTION - LATE Submissions, Modifications, and Withdrawals: See Section L, Provision No. 52.214-7 or 52.215-1. All offers are subject to all terms and conditions contained in this solicitation.

10. FOR INFORMATION

CALL:

A. NAME (NO COLLECT CALLS)

11. TABLE OF CONTENTS

SOLICITATION/ CONTRACT FORM

SUPPLIES OR SERVICES AND PRICES/ COSTS

2 - 9

X I CONTRACT CLAUSES

DESCRIPTION/ SPECS./ WORK STATEMENT X

PACKAGING AND MARKING

10 - 88

J LIST OF ATTACHMENTS

INSPECTION AND ACCEPTANCE

DELIVERIES OR PERFORMANCE

90 - 91

92 - 93 X K REPRESENTATIONS, CERTIFICATIONS AND

OTHER STATEMENTS OF OFFERORS

CONTRACT ADMINISTRATION DATA 94 - 97 X

SPECIAL CONTRACT REQUIREMENTS

OFFER (Must be fully completed by offeror) 98 - 104 X M

L INSTRS., CONDS., AND NOTICES TO OFFERORS

EVALUATION FACTORS FOR AWARD

NOTE: Item 12 does not apply if the solicitation includes the provisions at 52.214-16, Minimum Bid Acceptance Period.

is inserted by the offeror) from the date for receipt of offers specified above, to furnish any or all items upon which prices are offered at the price set opposite each item, delivered at the designated point(s), within the time specified in the schedule.

13. DISCOUNT FOR PROMPT PAYMENT

(See Section I, Clause No. 52.232-8)

14. ACKNOWLEDGMENT OF AMENDMENTS

(The offeror acknowledges receipt of amendments

AMENDMENT NO. DATE

to the SOLICITATION for offerors and related documents numbered and dated):

FACILITY

12. In compliance with the above, the undersigned agrees, if this offer is accepted within calendar days (60 calendar days unless a different period

SOLICITATION, OFFER AND AWARD

X

(X) SEC. DESCRIPTION (X) SEC. DESCRIPTION PAGE(S)

PART I - THE SCHEDULE

26. NAME OF CONTRACTING OFFICER (Type or print) 27. UNITED STATES OF AMERICA 28. AWARD DATE

EMAIL:TEL: (Signature of Contracting Officer)

CODE CODE

B. TELEPHONE (Include area code) C. E-MAIL ADDRESS

AWARD (To be completed by Government)

19. ACCEPTED AS TO ITEMS NUMBERED 20. AMOUNT 21. ACCOUNTING AND APPROPRIATION

22. AUTHORITY FOR USING OTHER THAN FULL AND OPEN COMPETITION:

10 U.S.C. 2304(c)( ) 41 U.S.C. 253(c)( ) (4 copies unless otherwise specified)

23. SUBMIT INVOICES TO ADDRESS SHOWN IN ITEM

24. ADMINISTERED BY (If other than Item 7) CODE 25. PAYMENT WILL BE MADE BY CODE

PART IV - REPRESENTATIO NS AND INSTRUCTIO NS

PART III - LIST O F DO CUMENTS, EXHIBITS AND O THER ATTACHMENTS

105 - 143

PART II - CO NTRACT CLAUSES

DISA/DITCO-SCOTT-PL83

2300 EAST DRIVE

SCOTT AFB IL 62225-5406

FAX:

TEL:

FAX:

TEL:

NOTE: In sealed bid solicitations "offer" and "offeror" mean "bid" and "bidder".

SOLICITATION

144 - 145

6. REQUISITION/PURCHASE NO.5. DATE ISSUED

02 Jun 2016

4. TYPE OF SOLICITATION

SEALED BID (IFB)

NEGOTIATED (RFP)

[ X ]

3. SOLICITATION NO.

HC1028-16-R-0007

Section B - Supplies or Services and Prices

ITEM NO SUPPLIES/SERVICES QUANTITY UNIT UNIT PRICE AMOUNT

0001 1 Lot TEC Services - Fixed Price

FFP

Fixed Price (FP) All Variations - This CLIN applies to all variations of contract types under fixed price. See Schedule B labor rate tables.

Period of Performance: Two (2) year base period - Contract Years 1 through 2.

FOB: Destination

NET AMT

0002 Lot TEC Services - Cost

COST

Cost Reimbursable (CR) All Variations - This CLIN applies to all variations of contract types under cost reimbursable.

ESTIMATED COST

0003 Lot Other Direct Costs (ODCs)

COST

Cost Reimbursable ODCs - IAW Section H.10.

1001 1 Lot OPTION TEC Services - Fixed Price

FFP

Fixed Price (FP) All Variations - This CLIN applies to all variations of contract types under fixed price. See Schedule B labor rate tables.

Period of Performance: Option Year 1 - Contract Year 3.

1002 Lot OPTION TEC Services - Cost

COST

Cost Reimbursable (CR) All Variations - This CLIN applies to all variations of contract types under cost reimbursable.

1003 Lot OPTION Other Direct Costs (ODCs)

COST

Cost Reimbursable ODCs - IAW Section H.10.

2001 1 Lot OPTION TEC Services - Fixed Price

FFP

Fixed Price (FP) All Variations - This CLIN applies to all variations of contract types under fixed price. See Schedule B labor rate tables.

Period of Performance: Option Year 2 - Contract Year 4.

2002 Lot OPTION TEC Services - Cost

COST

Cost Reimbursable (CR) All Variations - This CLIN applies to all variations of contract types under cost reimbursable.

2003 Lot OPTION Other Direct Costs (ODCs)

COST

Cost Reimbursable ODCs - IAW Section H.10.

3001 1 Lot OPTION TEC Services - Fixed Price

FFP

Fixed Price (FP) All Variations - This CLIN applies to all variations of contract types under fixed price. See Schedule B labor rate tables.

Period of Performance: Option Year 3 - Contract Year 5.

3002 Lot OPTION TEC Services - Cost

COST

Cost Reimbursable (CR) All Variations - This CLIN applies to all variations of contract types under cost reimbursable.

3003 Lot OPTION Other Direct Costs (ODCs)

COST

Cost Reimbursable ODCs - IAW Section H.10.

4001 1 Lot OPTION TEC Services - Fixed Price

FFP

Fixed Price (FP) All Variations - This CLIN applies to all variations of contract types under fixed price. See Schedule B labor rate tables.

Period of Performance: Option Year 4 - Contract Year 6.

4002 Lot OPTION TEC Services - Cost

COST

Cost Reimbursable (CR) All Variations - This CLIN applies to all variations of contract types under cost reimbursable.

4003 Lot OPTION Other Direct Costs (ODCs)

COST

Cost Reimbursable ODCs - IAW Section H.10.

SECTION B

B.1 MAXIMUM AMOUNTS, INDEFINITE-DELIVERY/INDEFINITE QUANTITY (ID/IQ)

a. This TEC Services contract provides technical solutions for the Department of Defense (DoD) in support of: (1) T&E services including Interoperability (IOP), Operational, Developmental, Standards Compliance and Conformance, Cybersecurity, Enterprise Service, and other T&E and Subject Matter Expert Support services, and

(2) T&E enabling services such as DoD IOP status development and management, management of TEC policies and processes, advanced technologies research, knowledge management, requirements review and documentation management, customer relationship management, test automation and instrumentation, testbed management, and laboratory support. TEC Services will be provided to DoD Combatant Commanders/Services/Agencies, federal and state agencies, coalition partners, and commercial entities to aid in the design, development, acquisition, integration, deployment, operational use, and sustainment of new and enhanced Information Technology/National Security System (IT/NSS) capabilities.

b. This is an Indefinite-Delivery/Indefinite-Quantity (ID/IQ) contract against which it is planned to award Fixed Price (FP) and Cost-Reimbursement (CR) type task orders in accordance with FAR Subpart 16.5 - Indefinite- Delivery Contracts. The total amount of all orders placed against all contracts resulting from this solicitation shall not exceed $480,000,000.00 over a five and one half year period: two (2) year base period of performance, three (3) one-year option periods and one six (6) month option period.

c. The Government anticipates a single contract award resulting from this unrestricted competitive solicitation.

B.2 LABOR RATE TABLE

The Labor Rate Table represents the offeror’s proposed fully-burdened hourly labor rates for each skill classification for work performed at Government sites and work performed at Contractor sites. At time of award, contractor rate tables will be incorporated in Section J.

B.3 INCORPORATION OF PROPOSED PRICING

The offeror’s proposed pricing and labor rate tables shall be incorporated into the contract as attachments to Section B provided in Section J, Attachments 6, 7 and 8.

Pricing for all cost-type contract actions (including, but not limited to proposal and billing of future task orders, modifications) will be restricted to the labor categories established in Attachment 7, CPFF Pricing Template, found in Section J. The schedules of labor rates proposed in the TO 0001 (FFP) Pricing Template will be utilized as test of reasonableness in the evaluation and analysis process for all cost-type task orders.

Pricing for all FFP contract actions (including, but not limited to proposal and billing of future task orders, modifications) will be restricted to the labor categories and fully burdened labor rates established in Attachment 8 Pricing Template, found in Section J.

Section C - Descriptions and Specifications

SECTION C

Performance Work Statement (PWS) Joint Interoperability Test Command (JITC) – Test, Evaluation, and Certification (TEC) Services

Contract Number: TBD Task Order Number: (completed by the KO at time of TO award or N/A) Tracking Number: HC1028-16-R-0007 Follow-on to Previous Contract and Task Order Number:

DISA Test and Evaluation (T&E) Mission Support Services (MSS) contract:

1. Alion Science & Technology (Alion), HC1028-12-D-0023

2. TASC, Inc. (acquired by Engility), HC1028-12-D-0024

3. ManTech Telecommunications & Information Systems Corporation (ManTech), HC1028-12-D-0026

4. Lockheed Martin Global Training & Logistics, HC1028-12-D-0028

5. Oberon, HC1028-12-D-0029

6. AMERICAN SYSTEMS, HC1028-12-D-0027

7. Science Applications International Corporation (SAIC)*, HC1028-12-D-0025

* Effective 28 April 2013, AMERICAN SYSTEMS purchased the testing arm of SAIC.

1 Contracting Officer’s Representative (COR).

a. Primary COR.

Name: Marty Peterson Organization: DISA JITC Department of Defense Activity Address Code (DODAAC): H91257

Address: 2001 Brainard Rd., Building 57305 Fort Huachuca, AZ 85613-7051

Phone Number: (520) 538-5440 Fax Number: (520) 538-5068 E-Mail Address: Marty.J.Peterson.civ@mail.mil

b. Alternate COR.

Name: Susan Fenwick Organization: DISA, JITC

DODAAC: H98287

Address: 101 Strauss Avenue, Building 900

Indian Head, MD 20640-5035 Phone Number: (301) 743-4123 Fax Number: (301) 743-4361 E-Mail Address: Susan.C.Fenwick2.civ@mail.mil

2 Contract Title. Test, Evaluation, and Certification (TEC) Services.

3 Background.

This Performance Work Statement (PWS) addresses the Test and Evaluation (T&E) and T&E Enabling Service Requirements of the Defense Information Systems Agency (DISA), Joint Interoperability Test Command (JITC).

JITC is the Department of Defense’s (DoD's) sole Joint Interoperability Certifier and only non-Service Operational Test Agency (OTA) for Information Technology (IT) and National Security Systems (NSS). JITC provides its T&E services to the DoD Combatant Commanders/Services/ Agencies (CC/S/A), federal and state agencies, Coalition Partners, and commercial entities to aid in their design, development, acquisition, integration, deployment, operational use, and sustainment of new and enhanced IT/NSS capabilities. JITC’s risk-based T&E services, tools, and environments ensure joint warfighting IT capabilities are interoperable and support mission needs by:

a. Determining to what extent applicable requirements and standards have been satisfied

b. Identifying and helping to mitigate technical and operational risks

c. Baselining and characterizing levels of performance

d. Documenting operational capabilities and limitations

e. Supporting acquisition, deployment, and operational use milestone decisions

General information concerning DISA’s and JITC’s missions, functions, organizational structures, operational capabilities, and service offerings are available through their respective public websites, as are regulations, directives, instructions, manuals, policies, and guidelines referenced in this PWS. The public website addresses are:

a. DISA: http://www.disa.mil/

b. JITC: http://jitc.fhu.disa.mil/ http://www.disa.mil/ http://jitc.fhu.disa.mil/

4 Objectives.

The objectives of the TEC Services contract are to support JITC in carrying out the unique T&E roles and responsibilities for which it is assigned within the DoD:

a. Joint Interoperability Certifier – JITC has sole responsibility for certifying joint and combined interoperability of all DoD IT and NSS. JITC performs critical interoperability and standards conformance T&E to reduce the risks faced by warfighters in the field. JITC often provides evaluation and certification services based in part or in whole on testing conducted by other organizations.

b. OTA – JITC is the OTA for DISA-managed programs. JITC also serves as the OTA for other DoD agencies such as the Defense Logistics Agency (DLA), the Defense Finance and Accounting Service (DFAS), the Defense Commissary Agency (DCA), the Defense Health Agency (DHA), the United States Special Operations Command (USSOCOM), and the National Security Agency (NSA). As an OTA, JITC conducts Operational Test and Evaluation (OT&E) to assess system operational effectiveness, suitability, interoperability, and cybersecurity.

c. Warfighter Support – JITC supports the Combatant Commanders during exercises and contingency operations, supporting the warfighter with on-the-spot evaluations of problem areas and viable mission-oriented solutions to include providing 24x7 hotline support.

d. Major Range and Test Facility Base (MRTFB) – JITC is the only non-Service member of the 23-element MRTFB. As such, JITC is considered a national asset focused on testing of IT/NSS. Under MRTFB guidelines, JITC must account for all direct costs at the job order level. MRTFB services provided include T&E capabilities, infrastructure, and resources to support DoD acquisition programs and other Governmental and commercial customers on a reimbursable basis.

e. Developmental Test and Evaluation (DT&E) – JITC is increasing its capabilities to provide a full range of DT&E services. Through these services, JITC will support DISA and other customers in their delivery of verified, reliable, fully functional, and secure technical solutions.

5 Scope.

The scope of JITC’s TEC Service requirements includes support for T&E planning, test conduct, data collection and analysis, and results reporting. JITC performs these T&E functions in the support of a wide range of critical Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance (C4ISR), combat, and defense business and health affairs systems; in addition, JITC evaluates emerging network, mobility, cloud, enterprise, cybersecurity, and other IT/NSS capabilities that include infrastructure, and environments such as the DoD Joint Information Environment (JIE). JITC’s TEC needs also include contractor-provided services for establishing and sustaining the underlying policies, processes, and capabilities that enable JITC to satisfy its customers’ T&E needs in a responsive and cost effective manner. In addressing these needs, JITC develops and applies proven methods, tools, instrumentation, automation, modeling, simulation, statistical and analytical methods, and other advanced technologies and techniques to ensure the T&E rigor needed to mitigate the risks and uncertainty its customers face in developing, fielding, operating, and sustaining IT/NSS capabilities.

JITC provides T&E support to numerous functional areas and technologies to include enterprise and cloud services;

tactical, strategic, and national intelligence systems; logistics, business, and medical applications; command and control; battlespace awareness; force application; intelligence; communications and collaboration; weapon system platforms; Unified Capabilities (UC); mobility; and cybersecurity. The customers that JITC supports through its T&E services include the DoD Combatant Commands, Services, and Agencies; federal, state and local Governmental agencies; U.S. Coalition Partners; and commercial vendors and service providers.

To avoid a conflict of interest as defined in United States (U.S.) Code (USC) Title 10, a DoD system’s Program Manager (PM) and its support contractors are prohibited from conducting OT&E for the purposes of supporting a milestone decision brief to a DoD Acquisition Executive. In accordance with USC Title 10, OT&E must be conducted by an independent OTA. JITC is the sole Joint OTA recognized by the DoD Director, Operational Test and Evaluation (DOT&E).

In addition to this TEC Services contract, DISA is also awarding two large contracts which provide T&E services, the System Engineering, Technology, and Innovation (SETI) contract and the Encore III contract. By design, neither the SETI nor Encore III contracts will provide DISA OT&E or DoD Joint Interoperability Certification services. Both types of T&E services are the exclusive responsibility of JITC to perform, and in turn, the DISA TEC Services contract to support. The TEC contract will complement and support the T&E services available through the SETI and Encore III contracts. All three contracts encourage the use of integrated testing, the reuse and sharing of T&E data, and the coordination of test events that enable the collection of data to efficiently address multiple T&E objectives.

Evolving DoD priorities and missions, coupled with continuous technological advancements, result in ever-evolving T&E requirements and challenges. Given this uncertainty and the rapid pace of change, it is important that the TEC Services contract provide the depth, breadth, and flexibility of support required by JITC to remain relevant and responsive in satisfying its customers’ individual and unique T&E needs over the full life of the contract.

In 2009, the OTA Commanders, to include the then commander of JITC, Colonel Stephens, signed a memorandum of agreement committing to the use of Design of Experiments (DOE) in OT&E. In 2010, the DOT&E, Dr. Gilmore, issued a memorandum requiring the OTAs to use scientific and statistical methods in developing rigorous and defensible test plans and in evaluating resulting test data. The Deputy Assistant Secretary of Defense, Developmental Test and Evaluation (DASD(DT&E)) in turn established the concept of Scientific Test and Analysis Techniques (STAT) to refer to the use of scientific and statistical methods and processes to enable the development of efficient, rigorous test strategies that yield defensible results. As an OTA and provider of DT&E services, JITC must comply with these DOT&E and DASD(DT&E) memorandums and concepts. Accordingly, for purposes of this PWS the term Science Based Test Design (SBTD) refers to the use of STAT and other scientific, statistical, and analytical methods as means of achieving T&E objectives through rigorous, systematic, and defensible approaches.

Two online resources for information pertaining to SBTD concepts and approaches are the DASD(DT&E) sponsored STAT T&E Center of Excellence at www.afit.edu/STAT/ and the DOT&E sponsored Test Science Knowledge Center at www.testscience.org.

6 Performance Requirements.

To operate in an effective, efficient, and responsive manner, JITC requires a coordinated unity of effort across the full spectrum of T&E services it provides its customers. In satisfying JITC’s customers’ T&E needs, the scope of the TEC Services contract must address requirements in three distinct but interrelated performance areas:

a. T&E Requirements – Those service requirements enabling JITC to directly support its customers’ T&E needs.

b. T&E Enabling Capability Requirements – Those service requirements enabling JITC to support its customers’ needs through the development and sustainment of core organizational T&E policies, processes, strategies, and guidance, in addition to MRTFB IT infrastructure and testbed modernization, automation, instrumentation, services, and capabilities.

c. Overarching Contract and Task Order (TO) Requirements – Those services requirements pertaining to the overall administration of the TEC Services contract and its individual TOs.

Tasks and subtasks within these performance areas will be specified at the level of individual TOs to be issued by the Government following contract award. To ensure JITC and its customers realize maximum benefit of the finite T&E resources at their disposal, the Contractor must possess the knowledge, skills, and abilities needed to seamlessly support all three performance areas.

In the Contractor’s proposal submitted in response to this RFP, the Contractor is required to provide a thorough explanation to demonstrate its understanding, experience, unique abilities, and proposed approaches for satisfying each of the requirements contained within this RFP. The Contractor’s proposal should cite specific, recent, and relevant examples of how they have or are currently supporting similar customer requirements.

The specific form, format, and content for the deliverables associated with each of the performance area requirements that follow will be specified in individual TOs.

6.1 T&E Performance Requirements.

The following seven performance requirement areas identify the services JITC requires to ensure “unity of effort” in meeting its external customers’ TEC needs:

• Performance Area 1.1: Interoperability T&E

• Performance Area 1.2: Standards Compliance and Conformance T&E

• Performance Area 1.3: Developmental Test and Evaluation (DT&E)

• Performance Area 1.4: OT&E

• Performance Area 1.5: Cybersecurity T&E

• Performance Area 1.6: Enterprise Service T&E

• Performance Area 1.7: Other T&E and Subject Matter Expert (SME) Support

In support of specific TOs issued for these seven performance areas, the Contractor shall possess the technical competencies and mission expertise to provide full T&E lifecycle support to include T&E planning, execution, data collection, analysis, and reporting:

a. T&E Planning – As part of T&E planning, the Contractor shall review and comment on system/program requirements; recommend specific T&E scope and objectives; perform in-depth analyses and risk assessments to identify specific T&E requirements and data sources; review testing conducted by other http://www.afit.edu/STAT/ http://www.testscience.org/ elements outside of JITC to help determine what data can be re-used and identify gaps to help focus JITC efforts; establish effective strategies based on SBTD approaches and instrumented data collection methods to reliably achieve T&E objectives; develop supporting T&E documentation; and plan for and coordinate the activities needed to carry out these strategies and plans in a rigorous, disciplined, and cost efficient fashion.

b. T&E Execution and Data Collection – In performing T&E execution, the Contractor shall ensure that required data is collected according to approved T&E strategies and plans, including the effective implementation of manual and automated data collection processes, and the performance of those activities required to manage, validate, and reduce collected data. Throughout T&E execution, the Contractor shall also help to identify and mitigate any limitations with the potential to affect the accomplishment of T&E objectives.

c. T&E Analysis – The Contractor shall apply systematic and rigorous statistical methods and analytical tools to thoroughly evaluate collected data to support T&E objectives and to ensure that resulting conclusions are at the confidence levels targeted during the T&E planning phase. In analyzing the collected data, the Contractor shall focus on determining the operational significance and mission impact of identified capabilities and limitations associated with the System Under Test (SUT).

d. T&E Reporting – In performing T&E reporting functions, the Contractor shall apply appropriate methods to clearly, concisely, and objectively document T&E results and findings in a timely and responsive manner, delivering high quality test reports that comply with JITC-approved processes and satisfy end-customer information needs.

Effective support of JITC’s T&E performance area typically requires the Contractor to have extensive knowledge of the functional and operational requirements, intended users, mission needs, and environmental conditions that the systems, technologies, and devices undergoing T&E are intended to support. The Contractor must often acquire a thorough understanding of the Tactics, Techniques, and Procedures (TTPs), Concepts of Operation, Standard Operating Procedures (SOPs), Military and Commercial Standards, and operational issues pertaining to each SUT.

The Contractor shall apply this understanding to help specify and establish the operationally representative test environments and conditions needed to effectively determine to what extent each SUT satisfies operational mission needs. The Contractor shall apply techniques such as mission thread analysis and decomposition, message traffic generation, models and simulation, stress and load testing, and other techniques to provide operational realism and mission context in the conduct of testing.

In providing T&E support, the Contractor shall identify and apply innovative methods and technologies to improve the overall efficiency and cost effectiveness by which JITC satisfies its customers’ T&E needs. In support of specific T&E objectives, the Contractor is to consider all valid data sources, which include data collected by JITC or other Governmental and non-Governmental organizations through dedicated test events, technology demonstrations, exercises, actual system operations, or other relevant sources of information. To the extent possible, the Contractor is encouraged to propose and apply automated and instrumented capabilities to achieve affordable, repeatable, and statistically relevant sample sizes, and a reusable means of satisfying specific T&E requirements. To improve overall T&E efficiency, the Contractor should exploit all opportunities for integrated test events led by JITC or other testing organizations that allow multiple T&E objectives to be concurrently and cost effectively achieved. The Contractor shall make use of (as appropriate) operationally representative, instrumented, and configuration-managed test environments to ensure test findings are relevant, repeatable, and quantifiable.

To ensure accurate and defensible T&E outcomes, and to comply with DOT&E and DASD(DT&E) mandates for increased T&E rigor and efficiency, the Contractor should consider the use of quantitative and qualitative SBTD methods, where appropriate. Such methods as regression analysis, DOE, Analysis of Variance (ANOVA), and other statistical approaches, combinatorial test techniques, and test automation and instrumentation should be considered as means to accurately and efficiently evaluate the performance characteristics of systems or products undergoing test, over the full spectrum of their intended operating environments. The Contractor should also consider the use of formal, well-researched analysis structures/evaluation frameworks as the basis for a program’s T&E, to ensure that T&E activities are performed in a thorough, systematic, and structured manner. In T&E planning, the Contractor should make use of SBTD approaches to determine the relevant combination of factors to be controlled, systematically varied, or left uncontrolled and recorded, to achieve efficient test designs. For each test design, the Contractor must justify both the number of proposed trials, and the placement of each test point within the operational environment, by citing statistical measures of merit (statistical models supported, measures of orthogonality, prediction variance, power, confidence, etc.). Each test design should also account for the statistical methods(s) of analysis to be used to determine if correlation exists between changes in test conditions to measured variations in performance. In cases where non-statistical methods are used to support particular T&E objectives, the contract shall justify such test design choices and explain how sufficient statistical confidence will be achieved. To mitigate physical or resource constraints, the Contractor shall consider the use of computer models and simulations, virtualized test environments, cloud computing, and other similar innovative and validated approaches for cost effectively satisfying particular T&E objectives. To provide accurate and complete post-test analysis, the Contractor shall report the statistical certainty and confidence intervals associated with test findings. The Contractor shall document SBTD activities, results, and findings in a manner that is clear, concise, and easily understood by JITC customers who many not possess an in-depth knowledge of technical, scientific, or statistical concepts.

The DoD Chief Information Officer (CIO), Joint Staff (JS), DOT&E, Defense, Deputy Assistant Secretary of Defense for Developmental Test & Evaluation (DDT&E), CC/S/A level Milestone Decision Authorities (MDAs), Component Acquisition Executives (CAEs), and PMs for DoD Major Automated Information Systems (MAIS) Acquisition Category (ACAT) and non-ACAT programs are all reliant on JITC and its T&E products and services to support various program oversight functions. JITC’s T&E results and findings serve as critical input to program acquisition and deployment decisions and to particular program review board processes such as Network Readiness Review Boards (NRRBs) and Acquisition Review Boards (ARBs). Because of the importance of the information JITC provides to support such program oversight and decision processes, it is vital that the Contractor be thorough, objective, and fully responsive in providing the T&E services required in this PWS. It is equally vital that the products and deliverables the Contractor produces in providing these T&E services be of high quality, free of defects and inaccuracies, and fully compliant with all DoD, DISA, JITC, and other applicable policies, instructions, guidelines, and reporting and formatting requirements.

All equipment, hardware, software, capabilities, and devices acquired in support of T&E performance requirements shall become the property of JITC, unless specified otherwise at the time of acquisition.

A description of the requirements associated with the seven TEC performance areas follows. It is important to note that customers seeking JITC T&E support for their programs, projects, systems, or products may request some combination of one or more of the following T&E services. In cases where multiple T&E services are required to support the same customer, the Contractor shall provide these services in the form of integrated or combined T&E activities to satisfy the customer’s overall needs in the most efficient and cost effective manner as possible.

6.1.1 Performance Area 1.1: Interoperability T&E.

6.1.1.1 Interoperability Certification.

The Contractor shall perform T&E planning, execution, data collection and analysis, and reporting to support the specific objectives of Joint Interoperability Certification and assessment as outlined in DoD Instruction (DoDI) 8330.01, “Interoperability of Information Technology (IT), Including National Security Systems (NSS),” the JITC Interoperability Process Guide (IPG), the JITC Net Ready Key Performance Parameter (NR KPP) Evaluation Guidebook, and JITC Instruction 380-50-02, “Interoperability and Standards Conformance Test and Evaluation (T&E) Certification Instruction.” Additional guidance, instructions, and policies pertaining to DoD interoperability roles, responsibilities, and requirements are documented in a number of United States Codes (USC), DoDIs, Chairman, Joint Chief of Staff Instructions (CJCSIs), and DoD guides and manuals to include: USC Title 10-Armed Forces, Sections 2222 and 2223; DoDI 5000.02; CJCSI 5123.01G; CJCSI 3170.01I; JS Joint Capabilities Integration and Development System (JCIDS) Manual; and the JS Net-Ready Key Performance Parameter Manual. These materials and other information pertinent to DoD interoperability certification and assessment are available through the JITC public website at http://jitc.fhu.disa.mil/ In cases where JITC will rely on testing conducted by other organizations to make a certification determination, the Contractor shall collaborate with other organizations to ensure appropriate data is collected and analyzed. In cases where JITC’s customers may not require or be ready for Joint Interoperability Certification, JITC may perform an interoperability assessment or component-level interoperability certification instead. In support of JITC customers’ interoperability certification or assessment needs, the Contractor shall perform the following activities per the processes outlined in the JITC IPG:

a. Assist programs with Net-Ready Key Performance Parameter (NR KPP), JCIDS, and Business Capability Lifecycle (BCL) Defense Business Systems (DBS) product reviews and development to ensure valid and testable requirements.

b. Participate in program T&E working groups and Working Integrated Process Teams (WIPTs) to coordinate test data requirements and interoperability T&E activities with PMs, users, OTAs and other test agencies.

c. Provide interoperability T&E input to program Test and Evaluation Master Plans (TEMPs).

d. Assess the levels of risk associated with a program’s specific interoperability-related requirements (e.g., interfaces, information exchanges, and standards).

e. Review and provide feedback on DoD Architecture Framework (DoDAF) viewpoints.

f. Develop accurate Integrated Architecture Traceability Matrices (IATMs) that account for any gaps or omissions within DoDAF viewpoints.

g. Design T&E strategies to enable program Joint Interoperability Certification.

h. Identify methodologies that employ SBTD approaches, automation, and instrumentation to evaluate Joint

Interoperability Certification in an efficient and rigorous manner.

i. Create Data Management and Analysis Plans (DMAPs) to manage data, products, and programs artifacts supporting Joint Interoperability Certification.

j. Prepare Joint Interoperability Evaluation Plans (JIEPs) based on IATMs, T&E strategies, evaluation methodologies, and DMAPs.

k. Issue Interoperability Statements as input to Operational Test Readiness Reviews (OTRRs).

l. Collect and analyze interoperability certification data per approved JIEPs.

m. Report interoperability T&E findings.

n. Prepare certification products.

o. Provide program specific interoperability and T&E status, results, findings, and recommendations in support of NRRBs, ARBs, and other program review boards and oversight functions.

Deliverables (as specified in individual TOs may include, but are not limited to):

a. Comment Resolution Matrices (CRMs)

b. Test Concept Briefs (TCBs)

c. IATMs

d. DMAPs

e. JIEPs

f. OTRR Interoperability Statement

g. Quick Look Reports

h. Test Results Briefings

i. Joint Interoperability Assessment/Certification Memorandums

j. Joint Interoperability Assessment/Certification Reports

6.1.1.2 Other Interoperability T&E.

For interoperability T&E not requiring certification under DoD Instruction 8330.01, the Contractor shall perform T&E planning, execution, data collection and analysis, and reporting to support the specific interoperability T&E objectives of the program. Examples of such support includes Mission Partner Environment testing and laboratory support in accordance with CJCSI 5128.01.

Deliverables (as specified in individual TOs may include, but are not limited to):

a. IATMs

b. TCBs

c. Interoperability Evaluation Plans

d. T&E Readiness Reviews and Briefings

e. DMAPs

f. Quick Look Reports

g. Test Results Briefings

h. Interoperability Assessment Reports

6.1.2 Performance Area 1.2: Standards Compliance and Conformance T&E.

6.1.2.1 Standards Conformance and Compliance Verification.

The Contractor shall support JITC in planning, execution, data collection and analysis, and reporting to determine system, product, or device compliance and conformance with particular standards and formal specifications. JITC provides standards conformance/compliance verification in support of a broad range of different types of standards and regulations to include, but not limited to: UC, North Atlantic Treaty Organization Standardization Agreements, tactical data links, electronic messaging formats, imagery, communication waveforms, finance and accounting, health and medical affairs, records management, and other standards conformance and compliance areas as listed in section 6.3.5.s of this PWS, “Technical Skills, Training, Credentials, and Experience.” In providing standards conformance/compliance support, the Contractor shall:

a. Develop standards compliance and conformance concepts, methods, and detailed test plans and procedures as outlined in JITC Instruction 380-50-02, Section 8.

b. Apply SBTD methods and instrumented and automated data collection and system stimulation, where appropriate, as means to provide rigorous, highly accurate, repeatable, and cost effective determination of standards compliance/conformance status.

c. Conduct standards compliance/conformance assessment and verification activities per approved plans.

d. Prepare standards compliance/conformance assessment and certification reports.

Deliverables (as specified in individual TOs may include, but are not limited to):

a. Compliance/Conformance plans and procedures

b. Standards Conformance/Compliance Assessment/Certification Reports

6.1.2.2 Standards Compliance Program Administration.

The Contractor shall perform research and administrative functions to maintain and improve the policies, procedures, and capabilities supporting the compliance/conformance certification of existing standards. The Contractor shall also perform research and administrative functions to support JITC’s efforts to establish the policies, procedures, and capabilities required to address the evaluation and conformance/compliance certification of new standards. The Contractor shall maintain standards conformance reference profiles and Approved Product List (APL) information for particular standards compliance programs under JITC’s oversight. Within the scope of this performance area, the Contractor shall also provide technical support to JITC as the SMEs for particular standard programs and provide standards input to the JITC Risk Assessment Database (JRAD) tool.

Deliverables (as specified in individual TOs may include, but are not limited to):

a. TEC policies and procedures

b. Standards Conformance Reference Profiles

c. APL updates

6.1.3 Performance Area 1.3: DT&E.

In support of U.S. Code Title 10, Section 139b requirements, and to increase JITC’s ability to verify the design, development, integration, and technical characteristics of new or enhanced capabilities prior to their acceptance and consideration for operational use, the Contractor shall support JITC in the planning and preparation for, conduct of, and reporting on DT&E. These DT&E services will be provided in support of capabilities being delivered by developers using both conventional development methods, as well as advanced methods such agile and Test Driven Design (TDD) approaches. The Contractor shall make use of SBTD methods such as combinatorial test methods, instrumentation and automation, and other advanced approaches, where appropriate, to improve the overall efficiency, effectiveness, rigor, and timeliness of JITC’s DT&E support. The types of systems that the Contractor may be requested to provide DT&E support of include DISA programs pertaining to enterprise services, cloud computing, mobility and UC, and other emerging technologies being adopted by DoD. The Contractor shall provide DT&E support that includes, but is not limited to:

6.1.3.1 Unit, Module, Device, Subsystem, and System Testing.

The Contractor shall perform T&E planning, execution, data collection and analysis, and reporting to verify that software and hardware modules, devices, subsystems, and systems undergoing development satisfy the stated requirements for functionality, interoperability, standards conformance, performance, capacity, scalability, reliability, availability, maintainability, cybersecurity, and other required technical characteristics as specified in applicable requirements sources and design specifications. The Contractor shall perform risk assessments to help focus T&E resources on areas of highest technical risk. The Contractor shall conduct regression testing to verify that previously evaluated item characteristics are not adversely affected by subsequent configuration changes.

Deliverables (as specified in individual TOs may include, but are not limited to):

a. DT&E strategies and methodologies

b. Risk Assessment plans and reports

c. Requirements Traceability Matrices (RTMs)

d. Use cases, test cases, detailed test procedures, and test scripts

e. Test Plans

f. Product Deficiency Reports

g. Test Reports

6.1.3.2 Integration and Qualification Testing.

The Contractor shall perform T&E planning, execution, data collection and analysis, and reporting to verify that software and hardware modules, devices, subsystems, and systems perform as intended when integrated together in an approved system design or architecture. The Contractor shall perform risk assessments to help focus T&E resources on areas of highest technical and operational risk. The Contractor shall perform T&E to determine the degree to which the integrated system satisfies applicable design requirements for functionality, interoperability, standards conformance, performance, capacity, scalability, reliability, availability, maintainability, cybersecurity, and other required technical characteristics as specified in applicable requirements sources and design specifications.

The Contractor shall conduct regression testing to verify that previously evaluated system characteristics are not adversely affected by subsequent configuration changes.

Deliverables (as specified in individual TOs may include, but are not limited to):

a. DT&E strategies and methodologies

b. Risk Assessment plans and reports

c. RTMs

d. Use cases, test cases, detailed test procedures, and test scripts

e. TCBs

f. Test Plans

g. Product Deficiency Reports

h. Test Reports

6.1.3.3 Acceptance Testing.

The Contractor shall perform T&E planning, execution, data collection and analysis, and reporting to verify that the developer has satisfied applicable functional requirements. Such testing may take the form of a Software/System Acceptance Test, Government Acceptance Test, or User Acceptance Test; normally involves test participation by representatives of the system’s intended user community; and precedes any Government commitment to limited deployment and formal OT&E. The Contractor shall conduct regression testing to verify that previously evaluated system characteristics are not adversely affected by subsequent configuration changes.

Deliverables (as specified in individual TOs may include, but are not limited to):

a. RTMs

b. TCBs

c. Use cases, test cases, detailed test procedures, and test scripts

d. Test Plans

e. Product Deficiency Reports

f. Test Reports

6.1.4 Performance Area 1.4: OT&E.

In support of U.S. Code Title 10, Section 139 requirements, the Contractor shall plan, conduct, and report on OT&E activities to assess operational capabilities and limitations and to determine operational effectiveness, suitability, interoperability, and cybersecurity under realistic conditions and representative mission support test scenarios. In providing such support, the Contractor shall make use of SBTD approaches, and T&E automation, instrumentation, simulation, and stimulation methods, where appropriate, as means to efficiently collect quantifiable, operationally representative, and highly accurate test data to provide increased confidence and mission relevance to OT&E results and findings. The Contractor shall provide support to the two following forms of OT&E:

6.1.4.1 Traditional OT&E.

The Contractor shall perform T&E planning, execution, data collection and analysis, and reporting to support the objectives of assessing or determining operational effectiveness, suitability, interoperability, and cybersecurity.

JITC conducts such testing in support of major program acquisition decisions as outlined in DoDI 5000.02, “Operation of the Defense Acquisition System.” OT&E activities may take the form of either Operational Assessments (OAs), to include Field Assessments (FAs) and Limited User Tests (LUTs), or Operational Tests (OTs), to include Initial Operational Test and Evaluation (IOT&E) or Follow-On Operational Test and Evaluation (FOT&E). Whether an OA or an OT, OT&E necessitate evaluation based on typical trained users and maintainers exercising an operationally representative system configuration, operated under realistic conditions, to determine to what extent the SUT satisfies operational mission needs. In providing support to OT&E activities, the Contractor shall follow the processes set forth in the JITC OT&E Guidebook, accessible via the JITC public website at http://jitc.fhu.disa.mil/. In providing OT&E support, the Contractor shall also be responsive to the latest guidance from DOT&E, accessible at http://www.dote.osd.mil/guidance.html. The Contractor shall provide support to OT&E activities that include, but are not limited to:

a. Identify operational requirements through review of program documentation such as JCIDS documents and other requirements sources.

b. Perform mission analysis through review of applicable Joint Capability Areas (JCAs), Universal Joint Task Lists, Mission Essential Task Lists, Mission Essential Tasks, mission threads, use cases, etc.

c. Conduct risk assessments to determine appropriate levels of OT&E.

d. Develop OT&E concepts and strategies.

http://www.dote.osd.mil/guidance.html

e. Document OT&E strategies within TCBs and TEMPs.

f. Participate in T&E and User Working Groups and WIPTs with stakeholders (PMs, DOT&E representatives, user representatives, T&E participants, etc.) to plan OT&E activities and resource requirements.

g. Develop OT&E Analysis Structures.

h. Capitalize on existing relevant test or operational data and opportunities for integrated and combined testing where DT&E, Interoperability, Cybersecurity, or other relevant T&E events that can be leveraged to support OT&E objectives, thus potentially reducing the scope, duration, expense, and delay associated with dedicated phases of OT&E.

i. Construct user surveys, questionnaires, interview forms, and other qualitative and quantitative data collection instruments.

j. Employ SBTD methods, automation, and instrumentation as means to carry out OT&E objectives in an efficient and rigorous manner.

k. Develop detailed OT&E plans.

l. Participate in OTRRs.

m. Conduct pilot tests, not for the OT&E of the SUT, but to verify and ‘dry run’ test and data collection procedures, etc., prior to the start of formal phases of testing.

n. Carryout OT&E execution and data collection in accordance with approved test plans.

o. Conduct ‘hot washes,’ lead or participate in test coordination meetings, and report test status throughout the conduct of OT&E events.

p. Chair Data Authentication Group (DAG) meetings to authenticate and validate collected data.

q. Analyze collected data to arrive at an operational effectiveness, suitability, interoperability, and cybersecurity determination.

r. Prepare final reports and briefings to convey OT&E results and findings.

Deliverables (as specified in individual TOs may include, but are not limited to):

a. TCBs

b. TEMPs

c. Analysis Structures

d. Risk Assessments Plans and Reports

e. OT&E Plans

f. OTRR Reporting

g. Quick Look Reports

h. OT&E Reports

i. OT&E Results Briefings

j. DAG Reports

6.1.4.2 Capability Assessments.

The Contractor shall perform T&E planning, execution, data collection and analysis, and reporting in support of non-traditional OT&E objectives and data collection opportunities. In non-traditional OT&E, select OT&E techniques are tailored to provide an operational perspective during system evaluation, without the associated structure, rigor, or formality normally associated with traditional OT&E. Examples of non-traditional OT&E include observational studies performed as part of normal system operations; abbreviated field assessments to identify operational capabilities and limitations; limited assessments conducted as part of military exercises, proofs of concept, and technology demonstrations; and assessments which reach or infer operational conclusions from purely developmental test data (e.g., component reliability effects on overall system reliability). Another example of non-traditional OT&E is the application of the JITC Mission Thread Analytic Framework (MTAF) Assessment Handbook approach.

Deliverables (as specified in individual TOs may include, but are not limited to):

a. TCBs

b. Assessment Plans

c. Quick Look Reports

d. Assessment Reports

e. Results Briefings

6.1.5 Performance Area 1.5: Cybersecurity T&E.

The Contractor shall perform cybersecurity T&E to assess system capabilities that support the protection of system data, detection of unauthorized system activities, reaction to system compromises, restoration of system capabilities, and continuous monitoring for system threats. JITC conducts cybersecurity T&E across the full system acquisition lifecycle, supporting both Defensive Cyber Operations (DCO) and DoD mission systems. Cybersecurity testing may be conducted as part of DT&E, OT&E, Enterprise Service T&E, interoperability T&E, Combatant Command exercises, and Commercial Product evaluations. JITC provides cybersecurity T&E services to a range of customers, including DOD, Federal, State and commercial organizations.