SRP_Appendix_D_-_HHS_EPLC_Projects_Exhaustive_List_of_Operations_and_Maintenance_Tasks_A1.pdf

SRP Appendix D - HHS EPLC Projects Exhaustive List of Operations and Maintenance Tasks

Operation and Maintenance of Existing Safety Reporting Portal (SRP) System The Safety Reporting Portal (SRP) provides a single Web-based portal for the reporting of adverse events associated with FDA-regulated products. The SRP system is comprised of several different, discreet, interoperable systems including the Rational Questionnaire (RQ), the Electronic Submissions Gateway (ESG), the Translator Tool (TT), the Emergency Operations Network (EON), and IERS/PVWorks. The Translator Tool (TT) component of the SRP accepts data from the Electronic Submissions Gateway (ESG), transforms the data as needed, performs a perfunctory validation, and forwards it to the proper target systems, either EON or IERS/PVWorks.

SRP System Overview

The Safety Reporting Portal (SRP) allows the public to enter adverse event and problem product reports through a web interface within the government regulated required time period. Once entered, the reports are passed through to the FDA for processing.

SRP is a high availability/high reliability system where downtime is only permitted in the production environment at CHDC during scheduled maintenances and/or release windows. In the rare event of an unscheduled outage, work to resolve the outage is required on a 365/24/7 basis.

The Translator Tool (TT) is a piece of the overall system. It translates reports as produced by the web interface into a file format that can be processed by the downstream systems. This translated file format may differ by report type, depending on the downstream system ingesting the information.

The diagram below shows the flow of information through the SRP system, including the Translator Tool.

TT is an implementation using three software components:

• Informatica PowerCenter 8.6.1 HF8 32bit – being used as a process designer and synchronizer between the different steps of the process and provides the means to perform DB access.

• Informatica Data Transformation 8.6.2 32bit – parsing tool which provides the means to access data of any format and conversion to any other format

• Oracle 10G – database

Figure 1 – SRP High Level Architecture Diagram for Release 1.0

Transition Tasks

1. The contractor shall define a transition plan due 5 business days after award. The transition plan shall describe the activities and timeline by which the contractor will gain the necessary knowledge of the system and processes from the existing contractor, from FDA staff, and from others, in order to effectively assume operations and maintenance responsibilities defined in this statement of work.

2. The contractor shall then conduct the transition activities defined in the plan, completing transition activities by 20 business days after award. The contractor’s transition activities shall include, but not be limited to:

• Discovery and transfer of all project-related documents and artifacts, to include documents, software code, software development tools, testing artifacts, and others.

• Knowledge transfer to contractor staff from previous contractor.

• Familiarization with all operations and maintenance responsibilities and procedures; code and code repositories, issue tracking procedures, issue repositories and issues status; requirements and requirements status.

• Familiarization with the overall SRP system, its purpose, design, operations, and points of contact.

3. As a part of the transition activities, the contractor shall identify and document deficiencies in SRP design, processes, operations, or communications and make recommendations to address those deficiencies.

Infrastructure O&M Tasks

1. The contractor shall migrate applications/system software within or to other FDA environments that may be either contractor or FDA hosted. The contractor shall assure the functionalities of the applications/systems after the migration.

2. The contractor shall support any testing necessary related to migrating to or operating within FDA environments.

3. The contractor shall assure the functionality of the system and make code adjustments as needed as a result of the following examples, but not limited to, regular patching done by the FDA, the FDA moving to more current technology such as upgrading for Microsoft Internet Explorer, etc.

4. The contractor shall periodically refresh application/system software, database, middle tier software, security software etc. to keep current with FDA’s evolving standards. Refreshing shall be performed within one year of the FDA establishing the next generation as a target standard.

Operational O&M Tasks

1. The contractor shall perform operations and maintenance for the Translator Tool (TT) system as follows:

• The contractor shall perform operations and maintenance services in support of existing software and hardware production systems.

Specifically, the contractor shall be responsible for operations and maintenance services for the Translator Tool (TT) system which is a component of a larger Safety Reporting Portal (SRP) system, as indicated in the Figure 1 below. This work shall include any development/testing/implementation necessary due to new release/changes in the RQ, EON, TT or IERS/PVWorks. The contractor shall perform operations and maintenance on the existing system and maintain TT’s Informatica workflows.

• On Scheduled Maintenance Weekends at CHDC:

a. Contractor shall startup/shutdown so that it is not affected adversely.

This includes any scheduled time between Friday at 5pm and Monday at 7am.

b. Contractor shall verify system is functioning properly after brining it back up on a maintenance weekend

• For Unscheduled Outages at CHDC (FDA does not anticipate more than 10 unplanned outages/year), Contractor shall be oncall 365/24/7 to work to respond to issues within 2 hours and deploy response services within 4 hours.

• Contractor shall be primary POC to CHDC

• When there is an event including but not limited to: scheduled maintenance and testing, Contractor shall be responsible for preparing notifications and FDA FLASH messages that need to go out when business users are impacted and other applications in our shared environment are impacted.

• The contractor shall perform patching of software for the system.

• The contractor shall manage and control configuration and changes to the system.

• The contractor shall fix/resolve new issues and existing issues documented in JIRA. Appendix M provides a summary of the current existing open issues documented in JIRA. The contractor shall perform minor enhancements to keep system functionality consistent with business performance expectations.

• The contractor shall install, configure, upgrade and maintain database(s). This includes updating processes, data flows, and objects (usually shown in diagrams).

• Interface with other functional areas for day-to-day checking/corrections to applications.systems.

• The contractor shall develop and perform data/database backup and recovery routines for data integrity and recoverability.

• The contractor shall create and administer regular backups

• The contractor shall enable the system’s auditing capabilities

• The contractor shall develop or enhance a suite of automated unit tests for the Translator Tool which exercise both the transformation mappings and the translator tool workflow elements. The unit tests should allow for “happy day” scenario tests, as well as all reasonable/foreseeable alternative path processing that could be performed upon failures.

2. The contractor shall lead production operations monitoring, troubleshooting and research for the overall SRP system as follows:

• The contractor shall provide 365/24/7 support for production problem resolution according to the Service Level Agreement.

• When there is an event including but not limited to: scheduled maintenance and testing, Contractor shall be responsible for preparing notifications and FDA FLASH messages that need to go out when business users are impacted and other applications in our shared environment are impacted.

• The contractor shall perform the necessary tasks to reconcile reports submitted to SRP and reports received by the target systems, and to identify and resolve the problem when reports are not reaching the target systems (i.e., missing reports). Support shall include but not be limited to leading the response, leading communications and problem resolution across technical staff of non-TT components of SRP to ensure successful resolution of all messaging/missing report issues.

jacob.ohatnick Cross-Out jacob.ohatnick Inserted Text

• The contractor shall determine if all submitted reports to the SRP system were successfully processed by the TT system and transferred to downstream destinations. If a report is not delivered to its final desintation system, contractor shall determine where issue in SRP exists and shall invoke the SRP Contingency Plan to deliver reports to destination systems within 4 hours and immediately notify the IT Project Manager (IT PM), and others as requested by the IT PM. Although FDA anticipates having a contingency option in place to manually move messages to their destination via electronic methods, that capability does not currently exist. The current contingency option requires the Contractor to pick up a CD with the reports from the Contractor providing O&M support for the SRP frontend (Rational Questionnaire) and upload these messages into the FDA system.

• The contractor shall notify the IT PM, specified distribution list (DL) or distribution method (e.g., posting on a central communications site), and the larger program team if operation of the system is interrupted.

• The contractor shall lead, participate in and contribute to SRP troubleshooting workgroups that involve all components of the overall SRP system. The contractor shall work collaboratively with other contractor and government teams in order to troubleshoot, diagnose, and document problems that prevent the end-to-end processing of reports through the SRP.

• The contractor shall perform and document thorough root cause analysis of existing and new production issues. As this system is integrated with several other systems, the contractor shall lead a coordinated effort with other teams to troubleshoot problems, sharing information and collaborating on testing in order to locate and diagnose problems.

• The contractor shall perform and document a general technical review of the existing SRP system, and known issues. As part of the review, the contractor shall document any new issues, and make recommendations of fixes for existing and new issues.

• The contractor shall perform configuration, security and design reviews/audits to ensure software, system, parameter, and configuration are correct.

• The contractor shall perform production control and quality control functions (job submission, checking and corrections).

• The contractor shall interface with and proactively reach out to CHDC teams and SRP system component teams to understand activities (such as releases, maintenance, upgrades, migrations, patches, and outages) that may impact the operations of SRP. The contractor must be aware of the schedule for releases, maintenance, and migrations of all the components of SRP, as well as the schedule for planned activities in the data centers (CHDC and White Oak Data Center) that may impact SRP. The contractor shall determine and communicate impacts that these activities may have on SRP, as well as lead planning and execution of contingency activities that are necessary as a result of the impacts to SRP.

3. The contractor shall identify, acquire and maintain necessary licenses for the efficient operation of the SRP environment. For some licenses, FDA staff are responsible for license purchases and maintenance. In this case, the contractor shall coordinate the license needs and maintenance with that FDA group.

4. The contractor shall schedule and provide training for new hires, changes in assigned responsibilities, disaster recovery, and administrators (log file analysis)

5. The contractor shall initiate, coordinate, and execute testing of the overall SRP system. While other contractor teams are responsible for unit and system testing of the various system components other than TT, the contractor shall be responsible for unit and system testing of TT, and for leading and conducting testing efforts of the end-to-end SRP system as required by the FDA EPLC. This may include, but not be limited to, user acceptance testing, performance testing, integrated testing, and IV&V.

Documentation

1. The contractor shall track all defects and issues in JIRA or other tool as determined by PM.

2. For any new production issues found, the contractor shall prepare and submit after-action reports.

3. The contractor shall update and maintain required artifacts in accordance with FDA ITIM and FDA EPLC requirements. This shall include updating/reviewing/finalizing/approving previously created EPLC artifacts.

4. The contractor shall maintain an updated project schedule and share the schedule with members of the larger SRP workgroup. The contractor shall communicate status on completing scheduled tasks, particularly where dependencies exist with other teams.

5. The contractor shall provide monthly status reports to include schedule;

potential project risks and mitigations; and EVM reporting. EVM reporting provided in monthly status report will also need to be separately submitted according to FDA format and schedule by the contractor.

6. The contractor shall ensure all processes are documented properly in the Operations Manual.

7. The contractor shall develop and maintain a performance and tuning plan for online process and databases.

8. The contractor shall provide responses to data calls and/or Requests for Inquiries (RFI).

9. The contractor shall review, update and/or create processes, standard operating procedures (SOPs) and reports. Examples include the following:

• Configuration Management, Change Management and Control Procedures

• Audit Logs

• Backup and Restore parameters

• Physical Protection

• Off-site Storage Usage

• Services and Availability

• Output Distribution Process

• System Logs and Reports

• Monitoring Processes

• Lessons Learned

• Post Implementation Reviews

10. The contractor shall update and maintain the investment data.

Security

1. The contractor shall provide responses to RFIs on security related issues.

2. The contractor shall support Authority To Operate (ATO) efforts, including performing required security-related modifications as a result of ATO efforts, including but not limited to, applications, databases, installation procedures, software release packages, user manual/guides.

3. The contractor shall perform self-administered or independent security audits.

4. The contractor shall investigate and report security incidents and events.

5. The contractor shall manage and document incident handling.

6. The contractor shall conduct penetration testing, and create, review and approve Penetration Test Report.

7. The contractor shall conduct vulnerability scan testing, and create, review and approve Vulnerability Scan Report.

8. The contractor shall create, review and approve Self Assessment

Document.

9. The contractor shall conduct System Re-Certification And Re-

Accreditation (as necessary).

10. The contractor shall update and maintain the Plan of Actions and

Milestones.

11. The contractor shall conduct Annual Information Security Risk

Assessment.

12. The contractor shall monitor and mitigate Information Security Risks.

13. The contractor shall manage Cryptographic Keys, User Administration and

Access Privileges.

14. The contractor shall update Application Software

Assessments and Reviews

1. The contractor shall conduct Functional System Assessment (User Satisfaction).

2. The contractor shall conduct an Annual Architecture and Operational Analysis.

3. The contractor shall participate in all aspects of Contingency Planning, including but not limited to, preparing COOP plan and conducting Contingency Plan test exercises.

4. The contractor shall test the Contingency and Disaster Recovery Plans, and review, update and approve the Contingency Plan/Disaster Recovery Plan Test Report.

5. The contractor shall conduct the Annual System Operational Review, and create, review and approve the Annual System Operational Review Package.

Adherence to Applicable IT Standards and Policies

1. The contractor shall adhere to current FDA IT standards, including but not limited to those described within the following:

• FDA Technical Reference Architecture,

• IT Security Requirements,

• Section 508 requirements, and

• FDA’s Transitional SOPs for all system maintenance, releases, etc.

into the White Oak Data Center (WODC) and the Contractor Hosted Data Center (CHDC) environments.

2. The contractor shall forward records-related concerns to the Records Management Officer.

3. The contractor shall ensure that all work performed adheres to the following requirements:

• Must conform to Information and Computer Technologies for the 21st

Century (ICT21) Architecture: The Information and Computer Technologies for the 21st Century (ICT21) investment will enable the FDA, through the development of an agency-wide bioinformatics initiative, to strengthen product development and approval, improve manufacturing and product quality, strengthen post-approval surveillance and safety, support electronic prescribing, and improve clinical decision support.

• Must comply with FDA approved technologies list. The FDA Master Approved Technology List (MAT) contains a list of approved and not approved technologies including applications (software), infrastructure & peripherals (hardware) and also scientific software and devices.

• Must comply with and supply a resource to assist with the FDA Security Authorization process, which closely resembles NIST SP800-

37. Security Authorization is the approach FDA follows to fulfill Federal Information Security Management Act (FISMA), Office of Management and Budget (OMB) and Department of Health and Human Services http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req1#SOW_Req1 http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req1#SOW_Req1 http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req2#SOW_Req2 http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req3#SOW_Req3 http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req3#SOW_Req3 http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req3#SOW_Req3

(HHS) requirements to ensure that information resources have adequate security to protect the Confidentiality, Integrity and Availability of information collected, processed, transmitted, stored, or disseminated by the agency.

• Security controls must comply with NIST SP800-53

• Must comply with all FISMA, NIST and OMB requirements

• Must comply with FDA 3250 series of SMG policies, as well as HHS security policies. Note: SMG 3251.4 Outsourcing and Third Party Arrangements. The FDA Staff Manual Guides (SMGs) are the Agency directives that document organizations and functions; delegations of authority; and administrative and program policies, responsibilities and procedures. A directive is a written communication issued in an organized system to establish organizations, delegations, policies, responsibilities, or procedures; to require action; or to set forth information needed for the effective operation of the system.

• Must comply with HHS information Security Program: Contractor Oversight Guide - Section 2, Defining Contractor Systems and Related Security Requirements.

http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req4#SOW_Req4 http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req5#SOW_Req5 http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req6#SOW_Req6 http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req6#SOW_Req6 http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req6#SOW_Req6 http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req7#SOW_Req7 http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req7#SOW_Req7 http://inside.fda.gov:9003/it/ITSecurity/Communications/ucm230607#SOW_Req7#SOW_Req7