36C10B25R0003 0003.pdf

5. PROJECT NUMBER (if applicable)

CODE 7. ADMINISTERED BY

2. AMENDMENT/MODIFICATION NUMBER

CODE

6. ISSUED BY

8. NAME AND ADDRESS OF CONTRACTOR

4. REQUISITION/PURCHASE REQ. NUMBER 3. EFFECTIVE DATE

9A. AMENDMENT OF SOLICITATION NUMBER

9B. DATED

PAGE OF PAGES

10A. MODIFICATION OF CONTRACT/ORDER NUMBER

10B. DATED

BPA NO. 1. CONTRACT ID CODE

FACILITY CODE CODE

Offers must acknowledge receipt of this amendment prior to the hour and date specified in the solicitation or as amended, by one of the following methods:

The above numbered solicitation is amended as set forth in Item 14. The hour and date specified for receipt of Offers

E. IMPORTANT:

is extended,

(a) By completing Items 8 and 15, and returning __________ copies of the amendment; (b) By acknowledging receipt of this amendment on each copy of the offer submitted; or (c) By separate letter or electronic communication which includes a reference to the solicitation and amendment numbers. FAILURE OF YOUR

ACKNOWLEDGMENT TO BE RECEIVED AT THE PLACE DESIGNATED FOR THE RECEIPT OF OFFERS PRIOR TO THE HOUR AND DATE SPECIFIED MAY is not extended.

12. ACCOUNTING AND APPROPRIATION DATA

(REV. 11/2016)

is required to sign this document and return ___________ copies to the issuing office. is not, A. THIS CHANGE ORDER IS ISSUED PURSUANT TO: (Specify authority) THE CHANGES SET FORTH IN ITEM 14 ARE MADE IN THE CONTRACT ORDER NO. IN ITEM 10A.

15C. DATE SIGNED

B. THE ABOVE NUMBERED CONTRACT/ORDER IS MODIFIED TO REFLECT THE ADMINISTRATIVE CHANGES

SET FORTH IN ITEM 14, PURSUANT TO THE AUTHORITY OF FAR 43.103(b).

RESULT IN REJECTION OF YOUR OFFER. If by virtue of this amendment you desire to change an offer already submitted, such change may be made by letter or electronic communication, provided each letter or electronic communication makes reference to the solicitation and this amendment, and is received prior to the opening hour and date specified.

C. THIS SUPPLEMENTAL AGREEMENT IS ENTERED INTO PURSUANT TO AUTHORITY OF:

D. OTHER

Contractor

16C. DATE SIGNED

14. DESCRIPTION OF AMENDMENT/MODIFICATION

16B. UNITED STATES OF AMERICA

Except as provided herein, all terms and conditions of the document referenced in Item 9A or 10A, as heretofore changed, remains unchanged and in full force and effect.

15A. NAME AND TITLE OF SIGNER 16A. NAME AND TITLE OF CONTRACTING OFFICER

15B. CONTRACTOR/OFFEROR

STANDARD FORM 30 PREVIOUS EDITION NOT USABLE

Prescribed by GSA - FAR (48 CFR) 53.243

(Type or print) (Type or print)

(Organized by UCF section headings, including solicitation/contract subject matter where feasible.)

(Number, street, county, State and ZIP Code)

(If other than Item 6)

(Specify type of modification and authority)

(such as changes in paying office, appropriation date, etc.)

(If required)

(SEE ITEM 11)

(SEE ITEM 13)

(X)

CHECK

ONE

13. THIS ITEM APPLIES ONLY TO MODIFICATIONS OF CONTRACTS/ORDERS,

IT MODIFIES THE CONTRACT/ORDER NO. AS DESCRIBED IN ITEM 14.

11. THIS ITEM ONLY APPLIES TO AMENDMENTS OF SOLICITATIONS

AMENDMENT OF SOLICITATION/MODIFICATION OF CONTRACT

(Signature of person authorized to sign) (Signature of Contracting Officer)

1 119

0003 12-2-2024

VA-24-00050415

Department of Veterans Affairs

Technology Acquisition Center

23 Christopher Way

Eatontown NJ 07724

Department of Veterans Affairs

Technology Acquisition Center

23 Christopher Way

Eatontown NJ 07724

To all Offerors/Bidders

36C10B25R0003

X

X X

***December 9, 2024 12:00PM EST

X 1

***Please see Continuation Page

Jason King

Contracting Officer

36C10B25R0003 0003

Continuation Page:

The purpose of Amendment 0003 to Request for Proposal (RFP) 36C10B25P0003 entitled

“Independent Enterprise Testing and Support Services (IETSS)” is as follows:

1. To revise Sections B.5 Performance Work Statement, Section C – Contract Causes, and

Section E – Solicitation Provisions. Updates in track changes below.

2. To incorporate revised Attachment 0001 – IETSS Price Evaluation Spreadsheet_v2 and reserve Attachment 0004 at Section D – Contract Documents, Exhibits, or Attachments.

3. All other terms and conditions remain in full force and effect.

B.5 PERFORMANCE WORK STATEMENT

PERFORMANCE WORK STATEMENT (PWS)

DEPARTMENT OF VETERANS AFFAIRS (VA)

Office of Information and Technology (OIT)

Electronic Health Record Modernization Integration Office (EHRM-IO)

Independent Enterprise Testing and Support Services (IETSS)

Date: November 20December 2, 2024

TAC-24-00050415

PWS Version Number: 3.23

1.0 BACKGROUND

The mission of Department of Veterans Affairs (VA), Office of Information & Technology

(OIT) is to collaborate with business partners to create the best experience for Veterans. In meeting these goals, OIT strives to provide high quality, effective, and efficient Information

Technology (IT) services to those who provide care to Veterans at point-of-care as well as throughout all points of each Veteran’s health care, in an effective, timely and compassionate manner. VA depends on Information Management/Information Technology (IM/IT) systems to meet mission goals.

OIT is the steward of VA’s IT resources and is responsible for ensuring the efficient and effective operation of VA’s IT Management System to meet mission requirements defined by the

VA Secretary and other key VA officials. OIT partners with other VA business units to deliver available, adaptable, secure, and cost-effective technology services to VA. Technology services are introduced into VA via two primary approaches: in-house development or acquisition of

Commercial Off The Shelf (COTS) and Government Off The Shelf (GOTS). These approaches follow industry, Government, and VA standards for technology, including but not limited to the phases of Software Development LifeCycle (SDLC) for development and Software Acquisition, Configuration, and Implementation (SACI) for COTS and GOTS.

The EHRM-IO Program is the program executive office charged with the preparation, deployment, and transition to a new electronic health record (EHR) within VA. The program will ensure that the process of deploying the new EHR is done in a manner that meets VA needs and supports seamless healthcare to Veterans and qualified beneficiaries. The Integrated Testing program within the EHRM-IO manages and oversees the execution of the test and evaluation activities across the EHRM program and its products. The testing community that makes up the

EHRM-IO Integrated Testing Program is comprised of three main organizations: Integrated

Testing Enterprise Test (IntT-ET), Office of Functional Champion, Functional Test & Evaluation

(OFC Test/OFCt), and the Oracle Health Testing & Quality Assurance (T&QA). The resources delivering these services are a mix of EHRM-IO IntT VA resources and IntT-ET contractor services through this IETSS contract; OFC Test VA; and OIT Test Management and Operations

(TMO) and Test Center (TC) VA resources.

The IETSS Contract will provide services in support of EHRM-IO IntT and OFC Test and OIT

TMO and Test Center (TC) VA resources. IETSS resources will work closely and collaboratively with the Oracle Health T&QA team to ensure an integrated approach and schedule for the EHRM-IO Test and Evaluation lifecycle, the IETSS Contractor shall coordinate services with OIT and EHRM-IO. The IETSS Contract will provide services in support of

EHRM-IO as well as VA OIT current legacy and future major programs requiring test service support.

2.0 APPLICABLE DOCUMENTS

In the performance of the tasks associated with this Performance Work Statement, the Contractor shall comply with the most current version of the following:

1. “Federal Information Security Modernization Act of 2014”

2. Federal Information Processing Standards (FIPS) Publication 140-3, “Security

Requirements for Cryptographic Modules”, March 22, 2019

3. FIPS Pub 199. “Standards for Security Categorization of Federal Information and

Information Systems,” February 2004

4. FIPS Pub 200, “Minimum Security Requirements for Federal Information and

Information Systems,” March 2006

5. FIPS Pub 201-3, “Personal Identity Verification of Federal Employees and

Contractors,” January 2022

6. 10 U.S.C. § 2224, "Defense Information Assurance Program"

7. 5 U.S.C. § 552a, as amended, “The Privacy Act of 1974”

8. Public Law 109-461 (P.L. 109-461), Veterans Benefits, Health Care, and

Information Technology Act of 2006, Title IX, Information Security Matters

9. 42 U.S.C. § 2000d “Title VI of the Civil Rights Act of 1964”

10. VA Directive 0710, “Personnel Security and Suitability Program,” June 4, 2010, https://www.va.gov/vapubs/index.cfm

11. VA Handbook 0710, “Personnel Security and Suitability Program,” May 2, 2016, https://www.va.gov/vapubs/index.cfm

12. VA Directive and Handbook 6102, “Internet/Intranet Services,” August 5, 2019

13. 36 C.F.R. Part 1194 “Information and Communication Technology Standards and

Guidelines,” January 18, 2017

14. Office of Management and Budget (OMB) Circular A-130, “Managing Federal

Information as a Strategic Resource,” July 28, 2016

15. 32 C.F.R. Part 199, “Civilian Health and Medical Program of the Uniformed

Services (CHAMPUS)”

16. NIST SP 800-66 Rev. 1, “An Introductory Resource Guide for Implementing the

Health Insurance Portability and Accountability Act (HIPAA) Security Rule,”

October 2008

17. Sections 504 and 508 of the Rehabilitation Act (29 U.S.C. § 794d), as amended, January 18, 2017

18. Homeland Security Presidential Directive (12) (HSPD-12), August 27, 2004

19. VA Directive 6500, “VA Cybersecurity Program,” February 24, 2021

20. VA Handbook 6500, “Risk Management Framework for VA Information Systems

VA Information Security Program,” February 24, 2021

21. VA Handbook 6500.2, “Management of Breaches Involving Sensitive Personal

Information (SPI),” June 30, 2023

22. VA Handbook 6500.5, “Incorporating Security and Privacy into the System

Development Lifecycle,” March 22, 2010

23. VA Handbook 6500.6, “Contract Security,” March 12, 2010 (Appendix C updated

April 22, 2024

24. VA Handbook 6500.8, “Information System Contingency Planning,” April 6, 2011

25. VA Handbook 6500.10, “Mobile Device Security Policy,” February 15, 2018

26. VA Handbook 6500.11, “VA Firewall Configuration,” August 22, 2017

27. OIT Process Asset Library (PAL), https://www.va.gov/process/ . Reference Process

Maps at https://www.va.gov/process/maps.asp and Artifact templates at https://www.va.gov/process/artifacts.asp

28. One-VA Technical Reference Model (TRM) (reference at https://www.va.gov/trm/TRMHomePage.aspx)

29. VA Directive 6508, “Implementation of Privacy Threshold Analysis and Privacy

Impact Assessment,” October 15, 2014

30. VA Handbook 6510, “VA Identity and Access Management,” January 15, 2016

31. VA Directive and Handbook 6513, “Secure External Connections,” October 12,

32. VA Directive 6300, “Records and Information Management,” September 21, 2018 https://www.va.gov/vapubs/index.cfm https://www.va.gov/vapubs/index.cfm http://www.va.gov/vapubs http://www.va.gov/vapubs https://www.va.gov/process/ https://www.va.gov/process/maps.asp https://www.va.gov/process/artifacts.asp https://www.va.gov/trm/TRMHomePage.aspx

33. VA Handbook, 6300.1, “Records Management Procedures, “March 24, 2010

34. NIST SP 800-37 Rev 2, “Risk Management Framework for Information Systems and

Organizations: A System Life Cycle Approach for Security and Privacy,” December

35. NIST SP 800-53 Rev. 5, “Security and Privacy Controls for Federal Information

Systems and Organizations,” September 23, 2020 (includes updates as of

12/10/2020)

36. VA Directive 0735, “Homeland Security Presidential Directive 12 (HSPD-12)

Program,” October 26, 2015

37. VA Handbook 0735, “Homeland Security Presidential Directive 12 (HSPD-12)

Program,” March 24, 2014

38. OMB Memorandum 05-24, “Implementation of Homeland Security Presidential

Directive (HSPD) 12 – Policy for a Common Identification Standard for Federal

Employees and Contractors,” August 5, 2005

39. OMB Memorandum M-19-17, “Enabling Mission Delivery Through Improved

Identity, Credential, and Access Management,” May 21, 2019

40. OMB Memorandum, “Guidance for Homeland Security Presidential Directive

(HSPD) 12 Implementation,” May 23, 2008

41. Federal Identity, Credential, and Access Management (FICAM) Architecture, June

30, 2023 (https://arch.idmanagement.gov/#what-is-the-ficam-architecture)

42. NIST SP 800-116 Rev 1, “Guidelines for the Use of Personal Identity Verification

(PIV) Credentials in Facility Access, “June 2018

43. NIST SP 800-63-3, 800-63A, 800-63B, 800-63C, “Digital Identity Guidelines,” updated March 02, 2020

44. NIST SP 800-157, “Guidelines for Derived PIV Credentials,” December 2014

45. NIST SP 800-164, “Guidelines on Hardware-Rooted Security in Mobile Devices

(Draft),” October 2012

46. Draft National Institute of Standards and Technology Interagency Report (NISTIR)

7981, “Mobile, PIV, and Authentication,” March 2014

47. VA Memorandum, VAIQ #7100147, “Continued Implementation of Homeland

Security Presidential Directive 12 (HSPD-12),” April 29, 2011 (reference https://www.voa.va.gov/documentlistpublic.aspx?NodeID=514)

48. IAM Identity Management Business Requirements Guidance document, May 2013, (reference Enterprise Architecture Section, PIV/IAM (reference https://www.voa.va.gov/documentlistpublic.aspx?NodeID=514)

49. VA Memorandum “Personal Identity Verification (PIV) Logical Access Policy

Clarification,” July 17, 2019, https://www.voa.va.gov/DocumentView.aspx?DocumentID=4896

50. Trusted Internet Connections (TIC) 3.0 Core Guidance Documents, https://www.cisa.gov/publication/tic-30-core-guidance-documents

51. OMB Memorandum M-19-26, “Update to the Trusted Internet Connections (TIC)

Initiative,” September 12, 2019

52. OMB Memorandum M-08-23, “Securing the Federal Government’s Domain Name

System Infrastructure,” August 22, 2008

53. Sections 524 and 525 of the Energy Independence and Security Act of 2007, (P.L.

110–140), December 19, 2007

54. Section 104 of the Energy Policy Act of 2005, (P.L. 109–58), August 8, 2005

55. Executive Order 14057, “Catalyzing Clean Energy Industries and Jobs through

Federal Sustainability, 12/8/2021 https://arch.idmanagement.gov/#what-is-the-ficam-architecture https://www.voa.va.gov/documentlistpublic.aspx?NodeID=514 https://www.voa.va.gov/documentlistpublic.aspx?NodeID=514 https://www.voa.va.gov/DocumentView.aspx?DocumentID=4896 https://www.cisa.gov/publication/tic-30-core-guidance-documents

56. Executive Order 13221, “Energy-Efficient Standby Power Devices,” August 2, 2001

57. VA Directive 0057, “VA Environmental Management Program,” October 25, 2022

58. Office of Information Security (OIS) VAIQ #7424808 Memorandum, “Remote

Access,” January 15, 2014, https://www.voa.va.gov/DocumentListPublic.aspx?NodeId=28

59. Clinger-Cohen Act of 1996, 40 U.S.C. §11101 and §11103

60. “Veteran Focused Integration Process (VIP) Guide 4.0,” January 2021, https://www.voa.va.gov/DocumentView.aspx?DocumentID=4371

61. VA Memorandum “Proper Use of Email and Other Messaging Services,” January 2, 2018, https://www.voa.va.gov/DocumentListPublic.aspx?NodeId=28

62. “Product Line Management Transformation Playbook” version 3.1, August 2023, https://www.voa.va.gov/DocumentView.aspx?DocumentID=4946

63. NIST SP 500-267B Revision 1, “USGv6 Profile,” November 2020

64. OMB Memorandum M-21-07, “Completing the Transition to Internet Protocol

Version 6 (IPv6),” November 19, 2020

65. Social Security Number (SSN) Fraud Prevention Act of 2017

66. Section 240 of the Consolidated Appropriations Act (CAA) 2018, March 23, 2018

67. “VIP Release Process Guide”, Version 1.4, May 2016, https://www.voa.va.gov/DocumentView.aspx?DocumentID=4411

68. “POLARIS User Guide”, Version 1.9, March 2017, https://www.voa.va.gov/DocumentView.aspx?DocumentID=4412

69. EHRM-IO Joint Master Test Plan (JMTP)

3.0 SCOPE OF WORK

The Contractor shall provide support in the areas of test and evaluation of software, infrastructure, environments, and operations of the Independent Verification and Validation

(IV&V) TC, operations for TC environments hosted in VA Enterprise Cloud, and continuous quality process and program management support.

OIT and major program offices each require specific services during the period of performance to support all areas of daily operations. This work includes test and evaluation of existing legacy solutions, and VA applications in support of VA modernization initiatives. These applications include but are not limited to EHRM solutions and all systems integration to the EHR, Financial

Management Business Transformation (FMBT), future modernization programs, and other

GOTS and COTS applications. The required services also involve testing-related activities associated with Legacy Veterans Health Information Systems and Technology Architecture

(VistA) (Mumps, Delphi), modernized applications (Java/Oracle, Microsoft.Net / Structured

Query Language (SQL)), and other middleware and custom interface and data exchange systems.

While OIT TMO and TC may cover any application in this arena, primary focus is on modernization initiatives, data migration, major system interoperability, and data exchange applications while concurrently supporting testing activities during the transition from VA legacy systems.

3.1 CONTRACT TYPE

The effort shall be a hybrid Firm-Fixed-Price (FFP) and Time-and-Materials (T&M) contract.

https://www.voa.va.gov/DocumentListPublic.aspx?NodeId=28 https://www.voa.va.gov/DocumentView.aspx?DocumentID=4371 https://www.voa.va.gov/DocumentListPublic.aspx?NodeId=28 https://www.voa.va.gov/DocumentView.aspx?DocumentID=4946 https://www.voa.va.gov/DocumentView.aspx?DocumentID=4411 https://www.voa.va.gov/DocumentView.aspx?DocumentID=4412

4.0 PERFORMANCE DETAILS

4.1 PERFORMANCE PERIOD

The Period of Performance (PoP) shall be 12 months from date of award with four 12-month option periods. The contract includes one optional task to provide Transition Support. If exercised, the period of performance of the optional task will be a total of 180 days. The period of performance for the optional task will not exceed 180 days past the expiration date of the period in which it is exercised.

Any work at a Government site shall not take place on Federal holidays or weekends unless directed by the Contracting Officer (CO). (If required, the CO may designate the Contractor to work during holidays and weekends)

There are 11 Federal holidays set by law (USC Title 5 Section 6103) that VA follows:

Under current definitions, five are set by date:

New Year's Day January 1

Juneteenth June 19

Independence Day July 4

Veterans Day November 11

Christmas Day December 25

If any of the above falls on a Saturday, then Friday shall be observed as a holiday. Similarly, if one falls on a Sunday, then Monday shall be observed as a holiday.

The other six are set by a day of the week and month:

Martin Luther King's Birthday Third Monday in January

Washington's Birthday Third Monday in February

Memorial Day Last Monday in May

Labor Day First Monday in September

Columbus Day Second Monday in October

Thanksgiving Fourth Thursday in November

4.2 PLACE OF PERFORMANCE

Tasks under this PWS shall be performed at Contractor facilities.

All support will be within the Continental United States (inclusive of Alaska, Hawaii, and US

Territories). Some tasks under this PWS require the on-site presence of contractor resources and shall be performed at a VA location designated by the VA PM.

4.3 TRAVEL (T&M)

The Government anticipates the need for travel events to perform tasks associated with the effort, and to attend program-related meetings or conferences throughout the PoP. Below are travel estimates for the base period and each option period; however, the number, location and duration of travel events listed below are subject to change and will be based on operational requirements.

Travel is on a T&M, no fee basis and shall be in accordance with the Federal Travel Regulations

(FTR) and requires advance concurrence by the Contracting Officer Representative (COR). The

Contractor shall provide cost estimates with each travel request to the COR. Each Contractor invoice must include copies of all requested, applicable, and required receipts (FTR §301-11.25:

required for lodging, regardless of amount, and a receipt for every authorized expense over $75) that support the travel costs claimed in the invoice.

Contractor travel within the local commuting area of 50-miles, as well as contractor participation in virtual meetings, will not be reimbursed. Travel performed for personal convenience and daily travel to and from work at the Contractor’s facility will not be reimbursed. Contractor travel within the local commuting area will not be reimbursed.

Travel will be performed in accordance with Federal Travel Regulations and will be billed after performance.

Estimated Travel per Year

EHRM

OIT

Type of Meeting Location Number of Trips

Number of

Persons

Number of

Days

EHRM Testing at

Deployment Site

(6 events)

TBD pending operational requirements 6 3 5

EHRM Integrated

Testing Team

Meetings Washington DC 2 12 5

OIT IV&V Strategic

Planning Austin, TX 1 9 5

5.0 SPECIFIC TASKS AND DELIVERABLES

The Contractor shall perform the following in accordance with VA processes to be provided to the contractor.

5.1 PROJECT MANAGEMENT (FFP)

5.1.1 Contractor Project Management Plan

The Contractor shall deliver a Contractor Project Management Plan (CPMP) that lays out the

Contractor’s approach, timeline, and tools to be used in execution of the contract. The CPMP shall take the form of both a narrative and graphic format that displays the schedule, milestones, risks, and resource support. The CPMP shall also include how the Contractor shall coordinate and execute planned, routine, and ad hoc data collection reporting requests as identified within the PWS. The initial baseline CPMP shall be concurred upon and updated monthly thereafter.

The Contractor shall update and maintain the VA-approved CPMP throughout the period of performance.

Deliverable:

Contractor Project Management Plan

5.1.2 Contract Key Personnel

The Contractor shall assign a single individual to serve as the Contract Program/Project Manager to support the contract, as well as any other key personnel deemed necessary for support of the contract. Any change of key personnel, defined as Contract Program/Project Manager, Project

Manager, and Division Leads shall require the, CO, and COR and IETSS VA Senior

Management notification and a transition plan to ensure minimum disruption to services delivered via the IETSS contract.

Any personnel the Contractor offers as substitutes shall have capabilities and qualifications at least equal to the key personnel being replaced, meaning that candidates shall have experience in program and project management, leadership, and testing services at an equivalent or greater level to the key personnel being replaced. If any change to a key personnel position becomes necessary, the Contractor shall immediately notify the VA PM in writing. Whenever possible the

Contractor shall notify the VA PM of substitutions of personnel in writing 30 calendar days prior to making any change in key personnel. The notification should provide a detailed explanation of the circumstances necessitating the proposed substitution and shall demonstrate that the proposed replacement is of at least substantially equal ability and qualifications as the individual originally proposed for that position.

The Contractor agrees that it has a contractual obligation to mitigate the consequences of the loss of key personnel and shall promptly secure any necessary replacements in accordance with this

PWS section. Failure to replace key personnel without a break in performance of the labor category at issue shall be considered a condition endangering contract performance and may provide grounds for termination.

5.1.3 Reporting Requirements & Management Meetings

The Contractor shall provide the COR with Monthly Progress Reports in electronic form in

Microsoft Word or Excel formats. The report shall include detailed explanations for each required area and ensure that data is accurate and consistent. These reports shall reflect data as of the last day of the preceding month.

The Contractor shall conduct weekly management meetings that summarizes services delivered for the past week, active or open services, issues resolved and issues still under consideration that are relevant for VA management visibility and related to the requirements review. Weekly management meetings are attended by VA and Contractor management and team leads with meeting notes generated as an artifact and distributed to the Government.

The Monthly Progress Reports shall cover all work completed during the reporting period and work planned for the subsequent reporting period. The report shall also identify any problems that arose and a description of how the problems were resolved. If problems have not been completely resolved, the Contractor shall provide an explanation including its plan and timeframe for resolving the issue. The Contractor shall monitor performance against the CPMP and report any deviations. It is expected that the Contractor will keep in communication with VA accordingly so that when issues arise, they are visible to both parties, to enable solutions to be developed without escalation. The Monthly Progress Report shall organize the report into the

EHRM-IO Integration Testing Section & OIT Section.

A. Monthly Progress Report

5.1.4 Testing IndependenceRESERVED

Under the terms of this contract/order, the Contractor or its subcontractors shall not test VA major IT program projects if they are acting as prime or subcontractor on a VA contract that provides IT project management, software development services as defined in Section C.14. To maximize transparency, the Government and Contractor will take an active role in ensuring that independence is established prior to contract award and maintained throughout the life of the contract.

The Contractor shall submit an annual Memorandum of Independence Confirmation that will enable VA to verify and monitor independence of Contractor and their subcontractors.

Additionally, throughout the period of performance of this contract, the Contractor shall immediately notify the CO, in writing, if it or its subcontractors intends on pursuing VA business opportunities for software development or IT project management. Thereafter, the Contractor agrees to comply with the directions of the CO that, in his or her discretion, deems necessary in order to ensure its independence. The ultimate responsibility for complying with this independence requirement remains with the Contractor. A CO’s response, or lack of response, to any notification made pursuant to this clause is not a representation of compliance with the independence requirement. Noncompliance with the independence requirement may lead to possible termination of this contract. These procedures will increase contract management effectiveness and will demonstrate Government due diligence.

A. Memorandum of Independence Confirmation

5.2 TEST AND EVALUATION SUPPORT (T&M)

The Contractor shall execute the following tasks in support of EHRM-IO Integrated Testing &

OIT. The Contractor shall support EHRM-IO Integrated Testing & OIT and execute required testing and testing-related activities. The EHRM-IO program services required from TMO and

TC cover the full scope of test and evaluation services needed for the successful implementation of the EHR COTS solutions across all waves of EHRM-IO deployment. In addition, the EHRM-

IO services required from TMO cover the large-scale data migration test and evaluation of legacy production data into the new EHR solutions.

5.2.1 Work Product Reviews

The Contractor shall conduct reviews of work products generated by organizations external to

TMO, TC and EHRM-IO Integrated Testing. Documents to be reviewed include task orders, test plans and other documents or products directly or indirectly related to testing activities. The

Contractor shall provide a Work Product Review Analysis Report that provides a summary of the product reviewed, highlighting positive areas as well as areas of concern. For EHRM-IO products tracked within the EHRM-IO deliverables review system, the Contractor shall conduct reviews of the assigned work products with the review comments being logged within the

EHRM-IO deliverables review system.

For informational purposes only, the current average monthly deliverable review workload is 55 artifacts, which is expected to ramp up significantly over the period of performance.

A. Work Product Review Analysis Report

5.2.2 Information Gathering/Requirements Analysis

The Contractor shall participate as a member of requirements reviews as requested by TMO and

TC in coordination with the project team responsible for the system under test (SUT). The

Contractor shall engage with a project team’s requirement management resources to understand the requirements management process for SUT, to determine processes for establishing traceability between test cases and SUT requirements and to provide input and feedback on the testability of the product software requirements under review. The Contractor shall use a specific approach agreed to between TMO and the project team PM prior to beginning work. The

Contractor shall coordinate the testability review of the requirements with other members of the review team.

Historically, there have been approximately 139 product requirement reviews per 12-month period.

5.2.3 Testing Intake Assessment/Criticality Analysis and Risk Assessment

The Contractor shall perform the Testing Intake Assessment (TIA) and Criticality Analysis and

Risk Assessment (CARA) processes on products and/or workflows for major programs utilizing the predefined processes. The Contractor shall provide detailed risk analyses, documenting the various aspects of competing risks and provide findings to IETSS management.

The Contractor shall:

a. Perform TIA process by accessing the project’s required repositories, such as GitHub, or other VA requirements sources such as contract task orders or performance work statements, or exports of requirements from Vendor requirements repository for COTS products.

b. Collaborate with the project team to resolve any issues with the data gathered via the

TIA. Review available documentation (e.g., Epics, User Stories) for initial completeness;

work with the project team to clarify any questions or issues pertaining to the documentation.

c. Generate Product/Project Assessment Summary (PAS) as applicable from TIA processes.

The Contractor shall use Government provided templates for the PAS.

d. Generate Build Assessment Summary (BAS) as applicable from TIA processes. The

Contractor shall use Government provided templates for the BAS.

e. Perform CARA process to evaluate, score and assess for criticality and risk of requirements and quality of the SUT documentation.

f. Conduct an Analysis Committee Meeting or Analysis Committee Review Briefing

(ACM/ACRB) with diverse expertise representation to consider the risk scores and other factors (e.g., availability of testing environment, quality of project documentation, resource availability) to determine the testing services recommended for a specific SUT.

g. Generate Risk Analysis Summary (RAS) as applicable as the output of the TIA, CARA and ACM/ACRB processes. The Contractor shall use Government provided templates for the RAS.

h. Perform a Quality Assurance Review of the RAS by submitting the RAS into the Change

Management tool and executing the QA Review Process for each project/product in the approved project artifact repository.

i. File all RAS documents in the approved project artifact repository.

Historical numbers of TIAs, CARAs, and RAS for the 12-month period are shown in the table below. Actual numbers will vary per requirements. CARA services in support of EHRM are based on historical workflow estimates with each workflow undergoing a CARA process. RAS produced in support of EHRM-IO IntT on EHRM workflows historically have been compiled as batches, thus the delta between CARA performed and RAS produced.

Table 5.2.3-1: Historical 12-month estimate for TIAs, CARAs, and RAS

OIT EHRM

TIA/Workflows 492 81

CARA 135 81

RAS 135 40

A. Risk Analysis Summary

5.2.4 Test Plan Summary

The Contractor shall document SUT requirements identified for test services and the overall approach to testing by creating one Test Plan Summary (TPS) for each SUT. The TPS utilizes the RAS, SUT application documentation, as well as collaboration with the SUT project team with regards to the testing services and approach as inputs. The Contractor shall ensure that the

TPS is completed to a level of detail agreed to by the VA Test and Evaluation manager using

TMO standards and templates, prior to the start of testing. For EHRM-Integration Office

Integrated Testing, the TPS is created and maintained within the Application Lifecycle

Management (ALM) Test Plan module.

a. Collaborate with the SUT project team to clarify any questions and resolve any issues with SUT documentation and clarify any scope changes.

b. Perform analysis of the project team documentation (e.g., Epics, User Stories, Requirements Traceability Matrices, or System Design Documents), as well as the RAS to reach agreement on specific features or functions of project requirements identified for test services.

c. Document TPS with requirements identified for testing services. These requirements, features, and functions shall also be managed and maintained in approved Government test management tools such as Micro Focus Application Lifecycle Management. The TPS shall include:

• requirements, features, and functions to be tested

• features and/or functions not to be tested

• testing services to be delivered as listed in section 5.2

• test methodology

• test criteria

• test deliverables

• test schedule

• test environments

• risks and constraints

• a communication matrix identifying points of contact

• testing/defects meetings approach

d. For OIT Projects, perform a TPS Quality Assurance Review by submitting the TPS into the Change Management tool and executing the artifact QA Review Process.

e. File all TPS documents in the approved project artifact repository.

f. For EHRM Integration Office Integrated Testing services, follow the EHRM Integrated

Testing Test Plan review process by which an integrated TPS is created across all

EHRM-IO testing teams. Participation in process includes activities such as documenting the test cases planned for Enterprise test execution and participating in test plan reviews.

For EHRM-IO Integrated testing services, the Test Plan for the SUT is created and maintained using the test plan modules within the ALM tool suite.

A. Test Plan Summary

5.2.5 Creation of Test Cases & Test Scripts, Test Data

The Contractor shall create multiple test cases and scripts for each assigned SUT that adheres to business and technical requirements provided by the project team responsible for the SUT. Test

Cases shall include items such as: setup steps, user provision requirements, pre-conditions, input data, user interaction, expected and actual results, and the type of test or technique being performed. Test Case creation and maintenance includes test cases that are used within the Test

Automation Services & tool suite,

a. Collaborate with the SUT project team to clarify any questions and resolve any issues related to creation of test cases, test scripts and test data.

b. Ensure all Test Scripts and Test Cases are completed for requirements to be tested using

Government provided testing management platforms and tools which presently include but may change and are not limited to: Micro Focus ALM, MF Test Tools, Eggplant Test

Automation tool suite.

c. Maintain all Test Scripts and Test Cases in Government provided test management tools.

Based on historical workloads, there have been 256 (236 EHRM, 20 OIT) SUT for a 12-month period.

5.2.5.1 Execute Test Plan, Analyze Results, & Report Findings

The Contractor shall execute the TPS once the Government has approved all testing documentation and create a Test Analysis Summary (TAS) for each SUT after execution of services. If multiple services (e.g., performance testing, interoperability testing) are executed for a SUT, the individual analysis of findings may be combined into a single TAS. The types, levels, and intensity of testing services are based on the RAS and TPS creation processes. Delivery and acceptance dates for the TPS and TAS are defined in the project schedule for the SUT. The project schedule for the SUT is developed by the Contractor and approved by the Government.

Test Executions will take place in the test environments which host the SUT. These environments may be a mix of Government and vendor test environments. For example, for

EHRM-IO testing, the test environment is an Oracle Health test environment that integrates with both Vendor COTS systems and VA test environments.

For informational purposes only, Government anticipates requiring 256 (236 EHRM, 20 OIT)

SUT for the 12-month period estimated from historical data and projections of EHRM system integrations and EHRM project workstreams (subject to change based on requirements).

Collaborate with the SUT project team to clarify any questions and resolve any issues related to executing the TPS and delivering test services.

Create and maintain project schedule within the VA approved schedule tool for each service provided for SUT.

Execute the TPS, leveraging tools/techniques such as continuous testing in DevSecOps model, automating testing as appropriate.

Log, track, manage all Findings/Incidents/Defects in Government test management repository tools.

Collaborate with the SUT project team and other project stakeholders to reconcile findings/defects and conduct retests as applicable.

Document all findings, test incidences, and results for each type of testing executed in a TAS.

Perform a Quality Assurance Review by submitting TAS into the Government Change

Management tool and executing the Government QA Review Process.

File all TAS documents in the approved project artifact repository.

A. Test Analysis Summary

5.2.5.2 Requirements Validation Testing

The Contractor shall execute Requirements Validation Testing services specified and documented in the TPS for each SUT.

a. Include all requirements listed in the requirements validation section of the TPS for verification through testing, modeling, demonstration, or other means.

b. Incorporate automation tools for development and execution of scripts as needed/appropriate.

c. Produce a Requirements Validation analysis (one per SUT cycle) as part of the TAS, documenting the results.

d. File all Requirements Validation analysis artifacts in the approved Government artifact repository.

5.2.5.3 Performance Testing

The Contractor shall execute Performance Testing services specified and documented in the TPS for each SUT.

a. Execute performance testing, which includes but is not limited to ramp-up tests, stress tests, spike tests, benchmarking, endurance, burst and end-to-end tests.

b. Analyze the test results.

c. Develop and produce Performance Point Reports prior to completion of the planned full test cycle when significant incidents or product performance requirement failures are encountered during testing.

d. Develop and produce a Performance Analysis (one per SUT cycle) as part of the TAS, documenting the test results and analysis of findings at the completion or termination of the planned test cycle, as defined in the SUT project schedule.

e. File all Performance Points and Performance Analysis work products in the approved

Government artifact repository.

5.2.5.4 Integration/Interoperability Testing

The Contractor shall execute Integration/Interoperability Testing services specified and documented in the TPS for each SUT for both OIT and major programs such as EHRM.

There are over 236 legacy VA systems (spans various topics such as clinical, financial, Veterans benefits) targeted for integration with Electronic Health Record at deployment sites. Additional integrations are identified when site deployments contain new capabilities that require a VA legacy integration. EHRM-IO IntT estimates 50% (118) of systems will require EHRM-IO IntT services during a 12-month period. It is anticipated that some system interoperability sustainment responsibilities would be transitioning to OIT in the future.

a. Execute integration/interoperability tests.

b. Incorporate automation tools for development and execution of scripts as needed/appropriate.

c. Analyze the test results.

d. Produce an Integration/Interoperability Analysis (one per SUT cycle) as part of the TAS documenting incidents and findings.

e. File all Integration/Interoperability Analysis artifacts in the approved Government artifact

5.2.5.5 Test Observation and Validation

The Contractor shall perform Test Observation and Validation (TOV) services specified and documented in the TPS for each SUT. The TOV may be face-to-face, via web conferencing, or hybrid of both as appropriate. Tools such as Microsoft Teams and WebEx can be used in situations where there is time, budget, and staffing constraints.

a. Coordinate and schedule the TOV session with the project team and all necessary stakeholders.

b. Prepare observation templates.

c. Accomplish TOV through observation of the test execution by the project team on selected requirements, functions, and features.

d. Complete the observation templates and notes during the test execution.

e. Document results in a TOV Analysis (one per SUT cycle) as part of the TAS.

f. File all TOV Analysis artifacts in the approved Government artifact repository.

5.2.5.6 Software Code Quality Check Scanning

The Contractor shall execute Software Code Quality Check (SCQC) services specified and documented in the TPS for each SUT. The Contractor shall execute SCQC services using both automated tools and manual walk-through of the software. SCQC shall include one or more of the following types of analysis:

• Static Analysis/Static Security Analysis - analysis of computer software and related documentation that is performed without executing programs built from the software to detect and report weaknesses that can lead to security vulnerabilities.

• Dynamic Program Analysis/Dynamic Security Analysis - analysis of computer software and related documentation that is performed by executing programs built from that software on a real or virtual processor to detect and report weaknesses that can lead to security vulnerabilities.

• Architectural Analysis - analysis of computer software and related documentation usually performed by manual walk-through of documentation and visual inspection of the code, may be supported by automated tools.

a. Execute combination of agreed upon SCQC services on SUT.

b. Analyze the SUT for compliance with VA quality and security standards.

c. Scan the source code/executables and inspect artifacts to assess that the SUT satisfies the stated performance, maintainability, and security requirements.

d. Document SCQC analysis results (one per SUT cycle) as part of the TAS.

e. File all SCQC artifacts in the approved Government artifact repository.

5.2.5.7 Automated Testing

The Contractor shall utilize automation tools where appropriate for efficiency and effectiveness of the testing.

a. Automate the test preconditions as applicable

b. Create automation scripts for both Oracle Health and VA side test cases

c. Maintain the automated scripts for relevancy

d. Expand automated tests for block upgrades, regression tests, domain refreshes, etc.

e. Ensure automated tests are domain and site agnostic to the maximum extent possible

5.2.6 Patient Safety Issue Testing

The Contractor shall execute Patient Safety Issue (PSI) testing on patches by project teams that require a Patient Safety Patch test, or as designated by a major program testing manager. This testing applies either to code patches or to software configuration as classified by the project team, the major program test manager, or the Office of Patient Safety as having potential patient safety concerns.

a. Review the available documentation, including but not limited to the patch description, the service request, and Service Now reports or their equivalent.

b. Examine the code or software configuration and perform functional testing on all affected routines, to include pre- and post-installation checks

c. Document results in a PSI Testing Report.

d. File all Patch or Configuration Analysis artifacts in the approved project artifact

A. Patient Safety Issue Testing Report

5.2.7 Section 508 Compliance Support Services

The Contractor shall provide Section 508/Accessibility support services for Section 508 test events and test findings as requested for the SUT.

a. Create and maintain the EHRM-IO Accessibility Test repository & configuration in support of the EHRM-IO IntT Section 508 workstreams.

b. Perform quality management activities designed to improve accessibility, customer service activities, or other business procedures related to Section 508 requirements.

c. In support of OIT projects, when requested by OIT, the contractor shall perform quality assurance and testing activities in support of Section 508 Compliance, on specified OIT products and systems.

5.2.8 Exploratory Testing

The Contractor shall execute exploratory testing services documented in the TPS for each SUT.

Exploratory testing may include stability analysis of computer software, systems and related documentation that is performed by applying massive amounts of random, invalid or unexpected data to the SUT and monitoring the SUT reaction and responses.

a. Define the scope and duration of the test effort in the Test Plan, including objectives and planned approaches to be used, including time-boxing of specific test events.

b. Design and execute tests in parallel (formal documentation of test conditions, test cases, and test scripts is not required).

c. Log results as tests are executed to document key aspects of what is tested, any defects found, and recommendations for further testing.

d. Document results in an Exploratory Testing summary (one per SUT cycle) as part of the

TAS.

e. File all Exploratory Testing artifacts in the approved Government artifact repository.

5.2.9 Mobile and Portable Device Application Testing

The Contractor shall execute testing on mobile and portable platforms as applicable for each

SUT. This testing may be automated or manual and cross multiple platforms.

a. Test the application software on a mobile device (examples include but are not limited to notepads, smartphones, touchscreen devices, etc.) for a SUT for its functionality, usability and consistency to include utilizing portions or subsets of the above listed test types.

b. Analyze the test results.

c. Document results in a Mobile and Portable Device Application Testing summary (one per

SUT cycle) as part of the TAS.

d. File all Mobile and Portable Device Application Testing artifacts in the approved

Government artifact repository.

5.2.10 Test and Evaluation Management Support

The Contractor shall provide test management support with test planning and preparation, test execution, and test closeout and documentation support for SUTs and by updating the weekly status update. This support also includes the support for test events for SUT by OIT and major program user/stakeholder communities such as EHRM-IO OFC. All testing services shall be conducted under a formal project management structure.

Specific project methodologies (e.g., waterfall, Agile, etc.) utilized for test engagements may vary as appropriate to specific efforts, but all must include:

a. requirements management

b. resource management

c. schedule management

d. risk/issue management

e. change management

f. knowledge management

g. communication management

The methodologies and processes must follow those supported by the Test Quality and Process

Management division of TMO as described in section 5.5.

5.3 TESTING AND TECHNOLOGY SUPPORT (T&M)

Testing Technology Support (TTS) provides support for OIT, EHRM-IO IntT and major programs. Services include architecture, interoperability, VistA Data Service (VDS), creation & maintenance of automated processes to support data migration validation, and service virtualization functionality. TTS provides test environment logical requirements documentation, architecture diagrams, required test data, virtual service components, and configuration of test environments for TMO testing projects for both OIT and ERHM-IO IntT.

5.3.1 Service Virtualization

a. Create and maintain Virtual Services Configuration Management and Change

Management Plans that define, document, control and implement required changes to virtualized services.

b. Develop and maintain a Virtual Service creation plan that includes the deployment process of virtualized services to the testing community.

c. Create and maintain the library of virtualized services available to the testing community to integrate with the SUT for an end-to-end test environment availability.

5.3.2 VistA Data Service-Test Data

Test data requirements will include collaboration with multiple test communities such as VA

OIT, Department of Defense, Oracle Health, EHRM-IO Office of Functional Champion for creation and maintenance of test patient identities, establishment of patient correlations between patient identity systems, clinical and financial/revenue cycle activity data.

a. Develop a Test Data Creation Process and Strategy that enables data to be prepared and manipulated to support various test scenarios in support of testing requirements.

b. Develop and maintain a test data intake process by which requests for test data creation are submitted and processed.

c. Create and maintain test data needed for execution of Test Scripts and Test Cases on the

SUT within the database(s) in the testing environment in support of EHRM-IO Integrated

Testing and OIT projects as needed or requested. Examples of test data include patient demographics, clinical orders, order results, patient insurance or other financial information, Veterans eligibility and benefits, revenue cycle accounting etc.

d. Incorporate automation tools for test data creation as needed.

e. Develop test data management plan addressing the creation, maintenance, assignment/portioning of test data when multiple teams are…