Draft PWS - DMM EHRM.pdf

DRAFT PERFORMANCE WORK STATEMENT

REQUEST FOR INFORMATION (RFI)

DATA MIGRATION MANAGEMENT (DMM) SUPPORT SERVICES for

ELECTRONIC HEALTH RECORD MANAGEMENT (EHRM)

FEBRARY 28, 2024

1.0 BACKGROUND

The mission of the Department of Veterans Affairs (VA), Office of Information and Technology (OIT), Electronic Health Record Modernization Integration Office (EHRM-IO) and Data Analytic (DA) is to provide high quality, effective, and efficient Information Technology IT and data management services to those responsible for providing care to the Veterans at the point-of-care as well as throughout all the points of the Veterans’ health care in an effective, timely and compassionate manner.

VA depends on Information Management/Information Technology (IM/IT) systems to meet mission goals. VA has been aggregating data nationwide for more than 14 years. The software to perform this activity is called VX130 (Veterans Health Information Systems and Technology Architecture (VistA) extract for 130 VAMC’s), this software has been modified extensively to assist in VA modernization.

VA announced its decision in 2018 to replace their current Electronic Health Record (EHR) system, the Veterans Information Systems and Technology Architecture (VistA), with the Oracle Health Millennium solution being deployed by the Department of Defense (DoD). The goal of this initiative is to leverage an existing commercial solution to achieve interoperability within VA, DoD and with community care providers. VA’s adoption of the same EHR system as DoD will ultimately result in all patient data residing in one common system. This will enable seamless care between the Departments without the necessity of manual and electronic exchange of information and without the need for reconciliation of data between two separate systems. To accomplish this, all relevant VA/VistA data must be continuously moved into the Oracle Health environment, specifically within the HealtheIntent platform. This is being accomplished by using the VX130 software, a process VA currently uses to load data into the VX130 Cache Object Database and pushed into the Corporate Data Warehouse (CDW). Data must be constantly streamed and synchronized with the Oracle Health Data Center system.

This remote system is a replica of the aggregated VX130 extracted VistA data in Intersystems Cache Object Database format. There can be other data pulls that go directly into Oracle Health via queries that are run then ingested into the Oracle Health Millennium EHR.

VX130 is used to deliver VistA data to Oracle Health via a historic load of selected data domains and then utilize a transactional feed of updates for additional data domains. After an initial historic load of past data is accomplished, the VX130 transactional feed will “funnel” data from the entire VistA nationwide system into a single point in near real time. As data is being accumulated, a Caché/IRIS “Mirror” will channel the data into a secondary server that will be housed in the Oracle Health Data Center. This process will continue until all VistA data is fully migrated to the Oracle Health HealtheIntent platform.

As the VA migrates to the Oracle Health EHR (Millennium), data is written back to the VA. This process is now referred to as Data Syndication. Data is expected to be coming back to VA in a variety of formats such as; but not limited to Text files, FHIR, HL7 and Relational Database structures. The syndicated data will be merged with VistA and other VA data so that enterprise level reports can still be run. The data from Oracle Health is in the Millennium (Oracle Health EHR) data structures and are not the same as VistA. This incongruity requires work to make the data homogenous where possible and to map and list fields that have no counterparts. The combined data is then dispersed as needed to other VA applications and can be merged with other data sources. Some or all of the current architecture could be moved to the cloud. Performance and storage will need to be maintained at acceptable levels as any move takes place and will represent tasks that need to be performed as part of this work. Syndication of data is an essential requirement for continuity of Veterans Health Administration (VHA) reporting, analytics, and decision support. The primary strategy for EHRM data syndication is extraction of new data out of Oracle Health, delivery of data to a central location, and distribution of data to necessary VA systems for reporting, applications, and analytics.

2.0 KEY PERSONNEL

The Contractor shall provide qualified personnel commensurate with this PWS, in terms of necessary skills at the requisite level of knowledge and experience. As part of the requirement, the Contractor shall include the following labor categories with specified level of knowledge and experience.

2.1 DATABASE ENGINEER (AZURE CLOUD DATABRICKS)

The Database engineer shall demonstrate knowledge of and experience with Spark Architecture, managing Databricks, and Databricks workspace. Two (2) years hands on experience with data extraction, (e.g. experience with ELT (Extract, Load, Transfer), corrupt record handling, parallel architecture data coding, user defined functions, join optimizations, structured streaming, managing Databricks cluster is required. Experience required with SQL server SSIS, Visual Studio, Azure data factory, Python, PySpark, spark SQL and execution hierarchy.

2.2 SYSTEM ENGINEER, SENIOR (SAN ENGINEER) (SE)The SAN Engineer must be at an expert level with proficiency with NetApp OnTap, HP Primera 20850 :

HP Lefthand Centralized Management Console Lefthand, HP Storage Essentials;

HP SN8700 Fiber Channel director switch with at least five (5) years’ experience.

The SE must also provide expert level troubleshooting of storage connectivity, performance, and availability issues across the VA Enterprise and in the Azure Cloud. Other required skills are SAN interaction with Commvault backups, Snapshots and Snapclones.

2.3 SENIOR NETWORK ENGINEER

This individuals’ skills must include designing, analyzing, evaluating, configuring, troubleshooting, and implementing network solutions. Experience with Cisco products is required. Senior network engineers have extensive knowledge of coding languages, protocols, hardware components (F5 Load Balancers, Palo Alto Firewalls, Cisco IP switches), and networking technologies. Security protocols and measures to protect the networks must be maintained and managed. This individual must have expertise in the above functions. The Senior Network Engineer shall demonstrate strong problem-solving strategies, excellent communication skills, and a willingness to collaborate with and lead junior engineers.

3.0 RESERVED

4.0 PERFORMANCE DETAILS

4.1 PERIOD OF PERFORMANCE

The Period of Performance (PoP) for this effort shall be a 12-month base period and four (4) 12-month option periods. The Option Periods will consist of the same tasks required in the Base Period.

4.2 PLACE OF PERFORMANCE

Efforts under this TO shall be performed at both contractor and Government facilities.

Data Center activity will take place at Austin Information Technology Center (AITC) - 1615 Woodward Street, Austin, TX 78772, and VACO - 810 Vermont Avenue, Washington DC, and remote/telework. The VA Program Manager (PM) shall identify the Contractor’s place of performance in the Task Execution Plan submission.

5.0 SCOPE OF WORK AND SPECIFIC DUTIES

5.1 GENERAL

The Contractor shall:

a) Architect, implement and maintain VX130 Cache Object classes for migration from VistA to Oracle Health data domains.

b) Develop, implement, and maintain data enterprise architecture and web-based software to monitor the VX130 as it provides enterprise processing. The software and enhancements shall track messages to all Intersystems Cache Mirrors and Oracle Health Data Center via Wide Area Network (WAN).

c) Provide support for the following: Windows system administration; Linux System Administration, Intersystems Cache System Administration; .NET and Cache Development, Data Syndication, Data Architecture SQL Script Development;

Web Development, Database administration; Storage Management for very large databases, LAN (Local Area Network) and WAN (Wide Area Network) monitoring, Business and Technical; Project Management. The contractor shall implement application programming interfaces, database or other techniques to make data available inside the VA that is returned from commercial EHR’s that are being used in the VA.

5.2 EHRM DATA SYNDICATION ENTERPRISE ARCHITECTURE AND DATA

SERVICES ANDSUPPORT

The Contractor shall assess the current EHRM data syndication requirements and create a strategy to meet the requirements. The Contractor shall document and update the strategy in the EHRM Data Syndication Plan as deemed necessary by the Government PM.

The Contractor shall develop and implement an enterprise data syndication architecture to meet the EHRM data syndication requirements. The Contractor shall document and update the Enterprise Architecture Design Document as needed.

The Contractor shall enhance and maintain the Enterprise Architecture Design Document that outlines implementation of data syndication for the VA with consideration of potential data coming from DoD and other sources. The Contractor shall update the Enterprise Architecture Design Document as changes affecting the document arise.

The Contractor shall also provide sustainment and updates to the EHRM Data Syndication Plan and shall provide guidance to VA in implementing the enterprise architecture. The architecture design shall incorporate cloud services, migrations, database transitions and support for cloud-based applications.

The Contractor shall contribute to the design of a new EHR focused data model sourced from the VistA based VX-130 platform and the Oracle Health Millennium data that is being transferred back to the VA through Data Syndication efforts. The Contractor shall update the EHR data model in the EHR Model Design Document.

The data syndication work will require ETL (Extract Transform and Load) to put Oracle Health data into a variety of possible target formats (text, SQL, Parquet, CSV, MDF, IRIS). New data tables shall be modeled and be made both practical and usable within the evolving data syndication database architecture, which is currently SQL and Azure Databricks but may evolve.

Currently the Data Syndication models are in Microsoft SQL Server and Databricks while Oracle Health data is in Oracle format.

The primary responsibilities for the Contractor shall include the following:

1) Design the logical and physical schemas needed to support an enterprise dimensional model

2) Track meta data related to all database objects

3) Evaluate commercial modeling tools against internal VA offerings to determine best fit and capabilities

4) Evaluate commercial data cataloging platforms with possible implementation

5) Support deployment and change management efforts

6) Provide input on customer access security models

The Contractor shall update the following:

a. Customer Access Security Document

b. Data Syndication Monthly Enhancement Document

c. Data Syndication Architecture Monthly Design Document

d. Data Syndication Implementation Plan and list fields that have no counterparts. The combined data is then dispersed as needed to other VA applications and can be merged with other data sources. Some or all of the current architecture could be moved to the cloud. Performance and storage will need to be maintained at acceptable levels as any move takes place and will represent tasks that need to be performed as part of this work.

Syndication of data is an essential requirement for continuity of Veterans Health Administration (VHA) reporting, analytics, and decision support. The primary strategy for EHRM data syndication is extraction of new data out of Oracle Health, delivery of data to a central location, and distribution of data to necessary VA systems for reporting, applications, and analytics.

Deliverables:

A. Weekly Progress Report

5.3. EHRM DATA SYNDICATION ENTERPRISE ARCHITECTURE SERVICES AND

SUPPORT

The Contractor shall assess the current EHRM data syndication requirements and create a strategy to meet the requirements.

The Contractor shall implement, configure, operate and support Data Syndication databases and environments such as IRIS, Oracle, NoSQL databases or others as required by VA PM. The Contractor shall enhance and maintain the Enterprise Architecture Design Document that outlines implementation of data syndication for the VA with consideration of potential data coming from DoD and other sources. The Contractor shall update the Enterprise Architecture Design Document as changes affecting the document arise.

The Contractor shall contribute to the design of a new EHR data model sourced from the VistA based VX-130 platform (VIDA – Veterans Information for Data Analytics) and the Oracle Health Millennium data that is being transferred back to the VA through Data

Syndication efforts. The Contractor shall update the EHR data model in the EHR Model Design Document and Wiki. The data syndication work will require ETL (Extract Transform and Load) to put OH (Oracle Health) data into a variety of possible target formats (text, SQL, Parquet, CSV, MDF, IRIS). New data tables must be modeled and be made both practical and usable within the evolving data syndication database architecture, which is currently SQL and Azure Databricks but may evolve.

Currently the Data Syndication models are in Microsoft SQL Server and Databricks while Oracle Health data is in Oracle format.

The Contractor shall also provide sustainment and updates to the EHRM Data Syndication Plan, and provide guidance to VA in implementing the enterprise architecture. The architecture design shall incorporate cloud services, migrations, database transitions and support for cloud-based applications.

The Contractor shall sustain, update, develop and implement an enterprise data syndication architecture to meet the EHRM data syndication requirements.

The Contractor shall provide:

1. Data Syndication Strategic Plan shall be provided with details on a future road map and enhancements and how it will affect architecture.

2. Data Syndication Architecture Design Document that outlines how-to implement flexible data syndication on the size and scale of VA.

3. Data Syndication Implementation Plan updates and shall provide guidance to VA in implementing the enterprise architecture.

4. Design the logical and physical schemas needed to support an enterprise dimensional model

5. Track meta data related to all database objects

6. Support deployment and change management efforts

7. Provide input on customer access security models

Deliverables:

A. Updated Data Syndication Strategic Plan B. Updated Data Syndication Architecture Design Document C. Updated Data Syndication Implementation Plan

5.4 EHRM DATA MIGRATION ENTERPRISE ARCHITECTURE SERVICES &

SUPPORT

The Contractor shall assess the current EHRM data migration requirements and create a strategy to meet changing requirements and request as they come in. The Contractor shall update the EHRM Data Migration Strategic Plan as needed.

The Contractor shall develop, implement and enhance the enterprise data migration architecture to meet the EHRM data migration requirements. This architecture shall contain elements of CPU, Storage, Network and Cloud options and be built from the current VX130 and Data Syndication infrastructure. The Contractor shall update the Data Migration Architecture Design Document as needed.

The Contractor shall update the Enterprise Architecture Design Document with any enhancements to keep the systems running and performing. The Contractor shall also update the EHRM Data Migration Implementation Plan and provide guidance to VA in implementing the enterprise architecture and aligned with all current security practices in achieving and maintaining an Authority to Operate (ATO).

5.5 EHRM DATA INTEGRATION ENTERPRISE ARCHITECTURE AND DATA

SERVICES & SUPPORT

As the VA migrates to Oracle Health EHR (Millennium) there will be a need to have data written back to the VA. This process is now referred to as Data Syndication. Data is expected to be coming back to VA in a variety of formats such as; but not limited to Text, FHIR (Fast Healthcare Interoperability Resources), HL7 and Relational Database structures. This data will then be dispersed as needed by other VA applications and can be merged with other data sources and prepared for use in other analytic tools.

The Contractor shall assess the current EHRM VistA, OH HealtheIntent, OH Enterprise Data Warehouse (EDW), Millennium and data integration requirements and create an enhanced strategy to meet the data and architecture requirements. The Contractor shall update the Data Integration Strategic Plan as needed.

The Contractor shall develop, enhance and implement an EHRM data integration architecture to meet the EHRM VistA and OH data integration requirements. This architecture shall be able to accept data in a variety of formats such as text, relational databases, FHIR, HL7 and other structures. Data will then be made accessible from API’s (Application Programmer Interfaces), by queries or other mechanisms for consumption and use by analytic tools, or custom code. The Contractor shall document, update and provide their architecture in the Data Integration Architecture Design Document as needed.

The Contractor shall provide updates and enhancements to Enterprise Architecture Design Documents that outline how-to implement changes in data integration on the size and scale of VA. The Contractor shall also update the EHRM Data Syndication Data Integration Plan, with Data Models as work is completed and shall provide guidance to VA in implementing the enterprise architecture.

The Contractor shall:

1. Implement as required, data integration and exchange using but not limited to:

FHIR, HL7, CSV files etc. for parsing and incorporation into databases that might include but are not limited to MS SQL Server, Oracle, Mongo, Azure Databricks, Varigence BimlStudio, Azure Data Lake, Azure Data Factory, Spark etc.

2. Troubleshoot, monitor performance and manage security on Oracle Active Data Guard or other mirroring technology.

3. Create Oracle Health to CDW, VistA, Millennium or Cloud Database data mappings for potential data migrations – OH has provided many of these mappings already.

4. Evaluate and integrate data from multiple sources, which requires data mapping from one data source to another minimizing any data loss.

5. Integrate data from Oracle Health and other data sources; this will involve use of common data standards and terminology including but not limited to: Digital Imaging and Communications (DICOM), Health Level 7 (HL7), SNOMED Clinical Terms (SNOMED CT), LOINC (Laboratory Data), RxNorm (Pharmacy), ICD9, ICD10, Current Procedural Terminology (CPT), United States Core Data for Interoperability (USCDI), International Statistical Classification of Diseases and Related Health Problems (ICD).

6. Write SQL queries for customers as well as to test database effectiveness and the performance of queries. Develop and update Conceptual Data Models for EHRM that support applicable VA enterprise data standards and the ability for VA to adopt emerging industry data standards.

7. Create or enhance database architectures and designs to meet data integration needs as they arise.

8. Develop or enhance high availability and Disaster Recovery Architectures

9. Conform to security best practices for code as well as system environment.

10. Monitor applications and system logs and take corrective action to achieve high system uptime.

11. Create code with Microsoft Visual Studio, SQL Server Integration Services

(SSIS), Varigence BimlStudio, ERWIN or similar tools to automate, monitoring, design or data mapping functions.

12. Create Powershell or bash scripts to assist in daily tasks and other automation.

13. Interpret Oracle Health’s Data model for ingestion, migration and addition into a new EHRM data model that could be used for the construction of API’s to be consumed by internal or external applications.

14. Interpret Oracle Health’s Data model and CCL (Oracle Health Command Language) for the construction queries and reports to be consumed by internal or external applications or users.

15. Create user and technical documentation for any programs, architectures or database work that is performed.

The Contractor shall document the data integration weekly activities in the Weekly Progress Report per PWS Section 5.2.

5.6 DATA INTEGRATION WEEKLY ACTIVITY REPORT EHRM ENTERPRISE

SYSTEM ARCHITECTURE SERVICES & SUPPORT

The Contractor shall assess the current EHRM System VA side Data Management, Data Migration, and Data Syndication requirements and create a strategy to meet the requirements. The Contractor shall update the System Strategic Plan as needed by the Government PM.

The Contractor shall develop and implement an EHRM System Architecture to meet the VA side Data Management, Data Migration, and Data Syndication requirements.

These shall include migration and syndication strategies on using new architectures that are a hybrid of on premise and cloud services. The Contractor shall update the architecture in the System Architecture Design Document as needed by the Government PM.

The Contractor shall update the Enterprise Architecture Design Document that outlines how-to implement a system architecture on the size and scale of EHRM. The Contractor shall also update the EHRM System Implementation Plan and provide guidance to VA in implementing the enterprise architecture as required to keep up with any changes that occur in the system.

Deliverables:

A. Updated System Strategic Plan

B. Updated System Architecture Design Document

C. Updated System Implementation Plan

5.7 EHRM DATA MAPPING AND CURATION ON PREMISES AND IN CLOUD

The Contractor shall define and refine mapping logic needed to transform data from both VistA and OH Millennium EHR systems into a new combined ERHM data model.

The current working name for this is VIDA (Veterans Integrated Data for Analytics). This work will require coordinating efforts with clinical staff, domain experts and other technical teams from the VA and Oracle Health to identify and implement the necessary mapping rules to populate a single converged model. This model should support the potential of being a data distribution point for other analytic models on premise and in the cloud.

The Contractor shall provide Domain Mapping Documentation detailing the data model.

Working with the Oracle Health Millennium data will require a basic understanding of the Oracle Health Command Language (CCL). Many of the mapping efforts will originate from discussions with Oracle Health experts where the output will be a series of CCL scripts. Those CCL scripts will need to be converted to transformation logic either in T- SQL or other standard formats. As data from VistA, Oracle Health and other sources such as community care is ingested any failures must be tracked, reported and remediated where applicable. The contractor shall perform data analysis as required to map and document source and destination data fields. In addition, the contractor shall develop custom scripts to validate clinical data across multiple systems and/or platforms and update and maintain Data Dictionaries, Wiki pages and other technical documentation.

The Contractor shall provide a Test Plan with pass/fail percentages for the ingested data. Other work will require extensive ETL, ELT processing to populate and test the new data models. The overall architecture is in development and this work will contribute to the end goal of a curated new EHRM data model as seen in “curated zone” in right bottom part of the Data Migration and Data Syndication Notional Architecture Diagram below. CDWWork represents VistA data, CDWWork2 represents Millennium Data and CDWWork3 represent the combination of Millennium and VistA data.

Figure 1: Current Data Migration Management Architecture

These databases the Contractor shall:

1. Write T-SQL scripts

2. Write code using Azure Databricks

3. Write code using Azure Data Factory

4. Create/load/Optimize Delta tables

5. Maintain/administrate Delta tables

6. Work with large datasets

7. Document mapping logic for customer consumption

8. Perform test case development to validate mapping logic

On Premises the Contractor shall:

1 Monitor ingestion of Millennium data 2 Monitor daily CDWWork3 ETL processes 3 Perform SSIS development for new Data Syndication tables 4 Test SSIS packages 5 Deploy SSIS projects and database objects 6 Develop CDWWork3 ETL processes 7 Test CDWWork3 ETL processes 8 Deploy CDWWork3 (SSIS projects and database objects)

In Cloud the Contractor shall:

1. Monitor various data pipelines (Vista, Millennium, VIDA) across different

Azure resources (Databricks, Azure Data Lake Storage, Azure Data Factory, SQL Server)

2. Test and validate data pipelines

3. Perform ongoing data quality development

4. Develop Databricks notebooks for processing/loading data into the data lake

5. Create ADF pipelines that orchestrate specific data ingestion tasks and/or call the appropriate Databricks notebooks

6. Design and develop updates to the meta data configuration database that drives the various data pipelines

7. Deploy Databricks, ADF, SQL and other Azure objects

8. Create automated deployment CI/CD pipelines

9. Create automated tests that validate the new code being promoted from one environment to another

A. Domain Mapping Documentation including Daily File Reporting of Oracle File transmissions.

B. Test Plan and Testing artifacts with pass/fail percentages.

5.8 DATA MIGRATION IRIS SYSTEM MANGEMENT OPERATIONAL TASKS

Currently, there are 62 Cache/IRIS systems that comprise the VX130 data migration system. The process VX130 uses is to read journal files and then insert them using a data filer queue into a Cache database. The Contractor shall monitor and check the VX130 data migration system and provide Daily Error/Log/Performance Reports to the Project Manager or designee. The Contractor shall:

1. Perform, analyze, and fix errors reported by the IRIS utility DATACHECK. The DATACHECK utility is used to check that a shadowed or mirrored database is the same as the source database. DATACHECK must be run on any Mirror servers on a regular basis of at least twice per month or as designated by the Project Manager. The results of all DATACHECKS shall be maintained in a database that can be easily queried and reported on.

2. Troubleshoot Data Quality Problems

3. Run Government off-the-shelf (GOTS) developed software for Quality Control that checks data between the source VistA systems and the Shadow database elements – 131 VistA (VA EHR) sites in all. These need to be run continuously but throttled so as not to overwhelm the data systems. All VistAs shall be run from top to bottom of the 131 VAMC list on a regular cadence. The VX130 has been running the GOTS software in both VM’s (Virtual Machines) and Docker containers to perform these checks. These checks must be run between the source VX130 databases on CacheFDW (the Central Aggregation Data Store) and all the mirrored databases in AITC (five (5) systems), one (1) Azure VX130 Mirror and one VX130 Mirror system in Kansas City.

4. Perform backups – the VX130 database requires a source of data that is not subject to mistakes or malicious intent. Mirrors, Storage Area Network (SAN) capabilities (HP, NetApp and others) are used for fast database restoration and must be considered in possible solutions in attaining faster restoration of any database. Traditional backup of 200TB or more are not practical with typical backup software such as Commvault. Some backups of smaller database are required. The Contractor shall perform database Backups and monitor those backups to ensure they are being performed regularly as well as updating the Database Backup and Restoration Strategy and optimize as new capabilities become available.

5. Recover the databases to a specific point in time, as per the requests (estimated at once per month). The Contractor shall perform SAN Snapshots, Snapclones on various SAN arrays.

6. Install, administer, and maintain IRIS Server Instances.

7. Install, administer, and maintain IRIS Mirror Instances.

8. Setup Test, Development, Staging and Production Environments.

9. Setup High-Availability as part of the Disaster Recovery Strategy for the

Databases.

10. Perform Database Capacity Planning of databases as required.

11. Perform Database Encryption and Decryption as required.

12. Perform Change Management activities to analyze and determine the impact of changes, test and implement changes, and document changes.

13. Troubleshoot various problems that arise in day-to-day work and fix the issues.

14. Monitor IRIS Server Error Logs and check e-mail alerts.

15. Provide support 24 hours/day, 7 days/week, 365 days/year, the coverage off business hours and holidays can be on call.

16. Monitor and perform tuning, and optimize for performance.

17. Document database configurations.

18. Perform IRIS SQL commands to check system status

19. Install, configure, patch and upgrade Cache/IRIS Server Software.

20. Set up Special Mirror systems to CacheFDW the central Mirror base.

21. Catch up weekly by starting mirror on Monday morning letting it run till it is current then stopping mirror on BACKUP machine.

22. Copy Journal Files from CACHEFDW/CACHEVX130 daily so that the system could be restored to several days behind.

23. Examine LOG Files and monitor on a regular basis, (at least twice per day) for errors and warnings. Current systems are listed below but the environment is expected to grow by four (4) servers per year:

a. 42 IRIS Instances on Windows 2016, 2019

b. 15 IRIS Instance of IRIS On Linux

c. 6 IRIS Instances in Microsoft Azure

d. 4 IRIS Instances in Oracle/Oracle Health Data Center

24. Check and Monitor Storage

25. Check and Monitor Databases

26. Check and Monitor Journal Files

27. Check and Monitor Operating Systems for both Windows and Linux – Including but not limited to all systems that support instances in #3 above.

28. Check VX130 Application monitoring – Monitor all aspects of the VX130 system.

29. Check and monitor to ensure VistA shadows are up-to-date and keeping up with all transactions.

30. Check and monitor to ensure JournalReaders are running and adding entries to DataFiler Queue.

31. Check and monitor to ensure DataFiler is keeping up with data being added to the DataFiler Queue by JournalReader.

32. Check and monitor the number of records being added to VX130 datastore in Federated Data Warehouse (FDW) Databases are increasing at a normal rate.

33. Mirrored databases are running and transactions are current with each database.

34. Ensemble is moving records to HealtheIntent with no anomalies and/or errors.

35. Process VDI (Vista Data Interface) counts reported from Oracle and reconcile any discrepancies. Create weekly counts reports noting mismatches between VX130 systems and VDI systems.

36. Work with Data Syndication quality team on validation of VX130 data consumed by Databricks. This work shall include the development of Jupyter notebooks or similar technology in Databricks systems to perform data validation.

The Contractor shall document any corrective actions taken in the Weekly Progress Report.

A. Daily Error/Log/Performance Reports Weekly Summary

5.9 DATA MIGRATION IRIS DEVELOPMENT TASKS

The Contractor shall:

1. Build IRIS classes in all environments using the VX130 Class Builder or latest local application.

2. Load SQL tables (Historical Pulls) in all regions/districts for classes in all environments, making code adjustments as needed.

3. Research and solve problems encountered with journaling and data filing using Cache tools.

4. Write user guides and documentation for all development

5. Perform upgrades to IRIS instances and databases.

6. Provide support for high availability IRIS databases.

7. Explore VistA Extraction processes for potential improvements, implementing those improvements and adjusting as necessary to conform to Oracle Health metadata changes.

8. Explore Mirror and IRIS monitoring processes for potential improvements and implement those improvements.

9. Identify potential performance issues with the IRIS databases and processes.

10. Develop Ad hoc IRIS applications (10-20 programs per quarter).

11. Work with InterSystems to solve IRIS database and server problems.

12. Document VistA Extraction and monitoring process and update existing documentation quarterly.

13. Design and API’s and REST interfaces for applications both IRIS and .NET

14. Include the weekly IRIS Issues Report, outlining issues the Contractor was assigned during the time period and their resolution, in the Weekly Progress Report.

15. Create an IRIS Classes Built Document for each IRIS Class assigned and built that includes the project summary, the VX130 Data Domain, and a list of the VistA files and fields contained in the class.

16. Provide or update the CACHE/IRIS Documentation for any software or objects developed or changed.

Deliverables:

A. IRIS System Documentation and Updates B. IRIS Classes Built Document C. IRIS Weekly Progress Report

5.10 DATA MIGRATION HEALTH CONNECT/ENSEMBLE SYSTEM

MANAGEMENT AND DEVELOPMENT TASKS

Ensemble is currently in use to push data from the VX130 Oracle Health mirror into Oracle Health HealtheIntent (HEI) and is a critical element in the migration of data from VA to Oracle Health.

The Contractor shall:

1. Monitor error logs with the ability to sort, filter, and act where possible.

2. Monitor trace logs with the ability to sort, filter, and act where possible.

3. Track messages by domain and reconcile table counts with Oracle Health.

4. Setup and maintain secure data message transmission.

5. Track the number of records sent per message by domain or class.

6. Monitor Queue Depth by service/process/operation.

7. Track retry attempts on storage devices.

8. Track suspended messages.

9. Troubleshoot network issues for both IRIS Enterprise Cache Protocol (ECP) and

External networks

10. Add Domain to Ensemble Production via GUI Interface.

11. Have knowledge and experience with the Ensemble GUI rule builder for routing messages and production data.

12. Edit Domain record type, schema version, status, and payload size via GUI

Interface.

13. Perform Intersystems Ensemble development and administration services.

14. Build Ensemble data flows in current or new Ensemble environments using the

VX130 ClassBuilder or latest local applications.

15. Load IRIS Objects, SQL Tables or other storage structures (Historical Pulls) in all regions/districts for classes in all environments with VistA or Data Syndication data.

16. Create and / or edit Ensemble Services, Processes and Operations.

17. Research and solve problems encountered with journaling and data filing using

IRIS tools.

18. Perform upgrades to IRIS databases and Ensemble as necessary.

19. Explore VistA Extraction and processing for potential improvements.

20. Identify potential performance issues with the IRIS databases and Ensemble processes.

21. Work with the project team to research and improve VistA extraction issues.

22. Develop Ad hoc Ensemble Services, Processes, Productions, and operations as needed.

23. Work with InterSystems to solve IRIS/Ensemble database and server problems.

24. Document Ensemble monitoring processes and update existing documentation quarterly.

25. Include the weekly Ensemble/Cache Issues Report, outlining the issues the

Contractor was assigned during the time period and their resolution, in the Weekly Progress Report.

26. Create and document any Ensemble Productions, Operations, Processes or Services designed and built.

The Contractor shall provide or update the following in Azure or VA DevOps:

a. Health Connect/Ensemble Class Code, Operations, Services, Productions Built Documentation

b. Technical Documentation of Classes and Production Code

c. Configuration, Setup and Operational Guides

d. Mapping of Storage and Processing Documentation

A. Document of Health Connect/Ensemble Class Code, Operations, Services, and Productions Documentation

B. Technical Documentation of Classes and Production Code C. Configuration, Setup and Operational Guides D. Mapping of Storage and Processing Documentation

5.11 STORAGE ADMINISTATION

Storage plays a critical role in the VX130 application and Data Syndication. There are many SANs that comprise the storage environment and it must constantly be monitored and optimized for performance and space.

The Contractor shall:

1 Design configurations, install, administer, and tune 24 Brocade Fiber Channel Switches.

2 Perform Snapshots and Snapclones for NetApp and HP3PAR storage arrays.

Design configurations, install, administer, and tune data storage devices (disk). This consists of 11 HP 3Par Arrays (20850), one (1) NetApp V3220 Array, 12 HP Virtual Library System (VLS) Arrays, one (1) Cluster, and 32 HP Lefthand Storage Nodes, divided amongst four (4) Clusters. The total disk storage capacity is approximately two (2) Petabytes, divided over 40 Arrays/Clusters, and two (2) Two-Node AFF400.NetApp devices.

3 . Utilize and perform backups using the HPE ESL G3 700 tape library utilizing Commvault or other software as needed. Commvault backup are also stored in Azure and shall be sustained. The Contractor shall provide and maintain a report on the storage capacity on all Storage Arrays and the percentage of space utilized as part of Weekly Status Report.

4 Administer Storage Support Consoles and Software including: HP Lefthand Centralized Management Console, HP Command View, HP Storage System Scripting Utility (SSSU), HP 3PAR Management Console, HP 3PAR InServ Command Level Interface (CLI) , HP 3PAR Service Processor Onsite Customer Care (SPOCC) , SANNAV, HP Command View for VLS , HP Research Organization (RO) Centralized Management Console, NetApp OnTap, NetApp OnCommand, Hadoop File System.

5 Design configurations, install, administer, and tune data storage devices (disk). This consists of eight (8) HPE Storage Arrays, including 20850, Primera, and Alletra classes; two (2) NetApp AFF400 Clusters ; two (2) Pure Storage C Class devices;

one (1) Pure Storage Flashblade. The total disk storage capacity for EHRM is approximately nine (9) Petabytes. HPE 3PAR 20800, four (4) HPE 3PAR 20850, eight (8) NetApp AFF300 arrays.

6 Administer Storage Support Consoles and Software including: HPE Infosight, HP HPE SSMC, HP 3PAR Management Console, HP 3PAR InformOS Command Level Interface (CLI) , HP 3PAR Service Processor Onsite Customer Care (SPOCC) , NetApp OnTap, NetApp OnCommand, NetApp Unified Manager, Pure Storage PureOne, Pure Storage Dashboard.

7 Actively monitor disk storage capacity and switches, providing utilization and performance graphs and statistics using monitoring tools and report as part of the Weekly Progress Report. Actively monitor disk storage capacity and switches, providing utilization and performance graphs and statistics using monitoring tools.

The tools consist of HPE SSMC, HPE Infosight, Solarwinds, NetApp Unified Manager. The tools consist of HP Storage Essentials, HP Report Optimizer, Systems, Application and Products Exchange Infrastructure (SAP XI), Crystal Reports, HP 3PAR Performance & Reports Manager, Solarwinds, HP Insight Remote Support, HP B-Series SAN Network Advisor, and HPE P6000 and SAN Navigator.

8 Provide integration services for new disk storage into the DMM (EHRM), BISL, and Azure environments.

9 Provide technical refresh data migration services from existing disk data storage devices to newly acquired disk storage devices.

10 Coordinate support for complex problem resolution of hardware errors, software issues and firmware upgrades with multiple vendors.

11 Perform Backup Operations, including hierarchical storage management, using Commvault and SQL 2022 Backup Manager to Pure Storage Flashblade. Configure and manage MS Azure Cloud Archival Backups.

12 Update the following: System Technical Guide on Configurations and Topology, including Topology Diagrams for all hardware, Build Guide for each system, and Configuration Guide for each system; Provide Mapping of Storage Documentation to each initiative, and Quarterly Storage State and Planning, for all equipment and provide ATO documentation as necessary.

5.12 NETWORK ADMINISTRATION

The Contractor shall provide network administration on the following:

The VX130 and Data Syndication uses local VA WAN’s (To VAMC’s) and LAN’s (Inside AITC to connect servers) and connectivity to Microsoft Azure Cloud and Oracle Health via DoD MedCOI (Medical Community of Interest) network. The VX130 architecture aggregates messages and they move through this network to AITC where they are added to databases then those databases are mirrored to the Oracle Health data center in Kansas City via an external WAN.

The VINCI/CDW/EHRM core network consists of six (6) Nexus 5500 series used for routing north/south traffic heading to and from the WAN circuit and 36 Mellanox 2700, and 2100 series switches for east/west (internal) traffic. The Nexus 5550 series is capable of one (1) and 10-GbE ethernet switching and L3 routing, that is in a fabric extender configuration that provides a high-density switching system that can consolidate more than 900 ethernet connections within a single management plane for easy management as well scalability. The Mellanox 2700 and 2100 series switches provide 40 and 100GbE as a Spine or ToR solution allowing for full rack connectivity to any server at any speed. The Mellanox switches have a high throughput capability of 6.4Tb/s with low latency. These switches provide connectivity to the network which house the VM’s and various Development, Test, Application and Cache/IRIS and SQL databases, and Docker Containers in ERHM environment. The Contractor shall also perform upgrades and maintenance on the Palo Alto or other Firewall equipment and/or software, so that security regulations and conformance to ATO and Authority to Connect (ATC) specifications are followed. Both WireShark, NetScout and Solarwinds network traffic analyzer are currently in use, but other tools maybe selected to monitor performance and statistics. All efforts need to be made to make the system both redundant and resilient, so that downtime can be kept to a minimum.

The Contractor shall:

1. Perform Network Monitoring –Monitor LAN traffic in AITC, Oracle Health, Azure, VA WAN traffic to Medical Centers.

2. Look for abnormal behaviors in traffic patterns.

3. Generate Reports from NetScout, WireShark, Network Traffic Analyzer, Splunk and other monitoring tools and react to aberrant behavior.

4. Constantly monitor the messages to make sure all packets are encrypted.

5. Monitor bandwidth, latency, and application specific messages.

6. Configure, update, and optimize security rules.

7. Upgrade system firmware as needed.

8. Monitor and analyze traffic logs.

9. Review and verify dynamic updates to applications and threats.

10. Configure and maintain high availability between active and passive firewalls.

For the Palo Alto Firewalls, the contractor shall:

1. Configure and update objects and groups.

2. Configure, update, and optimize security rules.

3. Upgrade system firmware as needed.

4. Monitor and analyze traffic logs.

5. Review and verify dynamic updates to applications and threats.

6. Configure and maintain high availability between active and passive firewalls.

7. Troubleshoot specific Palo Alto Firewall issues as needed.

For the F5 Load Balancers the contractor shall:

1. Configure and maintain node, pools, Virtual IPs, profiles and policies.

2. Configure server load balancing based on system owner requirements.

3. Create, update, and maintain certificates for SSL offloading.

4. Upgrade system firmware as needed.

5. Troubleshoot specific F5 issues as needed.

For the Cisco Data Center Switches the Contractor shall:

1. Configure access, trunk, and port-channel interfaces.

2. Configure and maintain Simple Network Message Protocol (SNMP) strings.

3. Monitor interface utilization.

4. Trouble shoot Layer 2 issues.

5. Configure and maintain network security configuration to adhere to industry practice.

6. Perform system firmware upgrades as needed.

7. Configure network redundancy and high availability.

8. Troubleshoot specific Cisco Nexus switch issues as needed.

The Contractor shall document all network administration activities in the Weekly Progress Report.

A. Weekly Network Administration Progress Report

5.13 WINDOWS ADMINISTRATION

1 Manage, Install, upgrade, patch and configure Windows 2016, 2019 and 2021 systems or other Window’s Systems as they come online.

2 Install standard applications (ex. Office365 or similar, ERWIN or similar, SQL server, Python, PowerBI, Jupyter Lab, Visual Studio, Toad and other similar items), configure Roles, Features, policies and settings to meet VA requirements.

3 Work with HP hardware to include Proliant DL Series Servers, Blades and Blade Chassis hardware.

4 Attach systems to a SAN, configure Multipath IO Policy for redundancy and to optimize performance.

5 Configure network interface controller (NIC) teaming for network redundancy, update drivers and apply settings to optimize throughput and performance.

6 Configure Mellanox NICs and network switches and software.

7 Configure Windows Clustering for High Availability, allocate resources to the appropriate Roles to meet the application requirements.

8 Use standard tools and services utilized in enterprise Windows environments to include Active Directory, DNS, DHCP, SMB, RDP, sFTP, WDS, and Group Policies.

9 Add systems to our monitoring tools (SCOM and Solarwinds) 10 Configure SSL/TLS Certs for securing application traffic.

11 Provide support 24 hours/day, 7 days/week, 365 days/year, the coverage off business hours and holidays can be on call.

12 Work with the Backup team to configure systems for backups (Commvault) 13 Work with other teams to evaluate requirements and propose solutions.

14 Configure Visio and office applications for generating and maintaining documentation.

15 Monitor systems daily to limit downtime (Systems at the VA and at Oracle

Health Data Centers).

16 Troubleshoot and address issues as they occur, move cluster instances if necessary to limit downtime.

17 Review system Event logs daily.

18 Review status of the cluster daily, check cluster events.

19 Review notifications from our monitoring environments.

20 Apply system patches monthly, or as needed to address system vulnerabilities.

21 Perform maintenance tasks such as firmware and driver updates as needed.

22 Address vulnerabilities identified by Nessus and Penetration Scans monthly in compliance with ATO and ATC.

23 Configure allocated storage from the SAN team, grow and assign volumes to the appropriate cluster roles.

24 Harden servers per VA / DoD Security Technical Implementation Guidelines

(STIG) guidelines.

25 Perform operating system upgrades and patching weekly or as needed to maintain functionality.

26 Perform monthly security patching and vulnerability remediation.

27 Administer Microsoft HYPER-V and Virtual Machine Manager.

28 Perform Physical and Virtual server builds.

29 Monitor Enclave server, storage, and networking performance.

30 Conduct System optimization and updating.

31 Conduct General Windows System troubleshooting.

32 Conduct Server performance bottleneck troubleshooting.

33 Maintain or create group policies/ local policies required by AITC/VA security entities.

34 Assist with troubleshooting Remote Desktop (RDP) issues from clients.

35 Maintain SSH File Transfer Protocol, (sFTP) system and create user accounts tied to specific directories and maintain security both in transit and at rest.

36 Maintain sFTP Security compliance to retain the ATO/ATC required by VA and DoD, the system uses the DoD VA MedCOI network.

37 Assist Oracle Health, VA, and external users on how to transfer files via sFTP.

38 Update the Windows Administration Technical Guide on Configurations and

Topology with any enhancements or changes as needed by Government PM

39 Document all Windows Administration activities in the Weekly Progress Report.

Deliverables:

A. Weekly Windows Administration Progress Report

B. Updated Windows Administration Technical Guide

5.14 LINUX ADMINISTATION

1. Administer Linux on VX130 Shadow systems with no less than 36 instances of Red Hat Linux.

2. Administer and maintain Red Hat Enterprise Linux (RHEL) and CentOS versions 6.x and 7.x.

3. Support Linux authentication with Centrify Software and Windows Domain Controller authentication (both shall support PIV card authentication).

4. Configure and maintain VET (VistA Extraction Team) and VX130 Cache shadow and mirror servers.

5. Maintain and troubleshoot Docker Containers running in Linux VM and Host systems.

6. Harden servers per VA and DoD STIG.

7. Perform operating system upgrades and patching approximately twice per quarter or as required by VA and/or DoD security officials.

8. Provide support 24 hours/day, 7 days/week, 365 days/year. The coverage off business hours and holidays can be on call.

9. Perform monthly or more frequent security patching and vulnerability remediation.

10. Reconfigure system architecture (for example clustering, adding additional storage, configuring system setting) for new solutions.

11. Work with application administrators to examine and analyze an application for when it is slow, erroneous or non-functional, to ensure a positive customer experience.

12. Monitor server logs and proactively address issues.

13. Respond to and remediate outages – this may occur after business hours.

14. Maintain a consistent configuration baseline.

15. Build Windows and Linux systems from bare metal, both virtual and physical.

16. Implement Linux/Windows interoperability solutions.

17. Identify opportunities for system and environment improvement and report weekly by telephone or email to VX130 technical PM.

18. Minimize impact of security scans and record in system log.

19. Sustain and upgrade security access mechanisms such as PIV and Active directory.

20. Track historical server…