Attachment 4 - SOW_042522 DAS OIMT Oracle DBA Support Task Order.pdf

STATEMENT OF WORK (SOW)

Task Order Title: AGENCY-WIDE ORACLE IDENTITY ACCESS MANAGEMENT SUITE AND

ENTERPRISE SHARED MIDDLEWARE SERVER SUPPORT FOR THE FDA

1. Background

The Office of Information Management and Technology/Division of Application Services (OIMT/DAS) at FDA is responsible for the administration, maintenance, and support of Oracle databases and application servers. These databases and application servers are hosted at both Government and Contractor operated sites and supports the FDA’s Centers.

The support services include standard upgrades, migrations, new environment provisioning Transparent Data Encryption, and Oracle Database Administration Support and Application Server Support required ensuring that all databases and applications comply with the agency’s current enterprise IT architecture requirements.

This t a sk o rder is essential in maintaining the FDA-wide Oracle Identity Access Management (OIAM) Suite components critical to virtually all FDA applications. A second element of this task order is to provide Middleware Server Administration Support and Deployment (Development, Test, Pre-Production and Production environments). A third element of this task order is to provide O&M support services for the Oracle Enterprise Manager Grid Control and Oracle Enterprise Manager Cloud Control 13c currently in use in WODC.

This task order is essential for completing all FDA Oracle database software version upgrades and completing the migration of the databases from the current Oracle Exadata Linux server and x86 hardware platform to the latest iteration of the same platforms. A fourth element of this task order is the Transparent Data Encryption and upgrade of the middle-tier components from WebLogic version 11g and 12c to version 12.2.1.4 or greater.

Once all upgrades and migrations are completed, the contractor will analyze the current resource usage on the WODC Exadata hardware and reclaim unused resources to maximize efficiency. This work is essential in order for the FDA to continue to receive technical support, as the technical support for version 12c database and WebLogic 11g have reached end-of-life or will do so in the coming months. The fifth and final element of this task order is to provide operation and maintenance support for the AppDynamics platform currently supporting various center at the FDA.

2. Objectives

The work to be performed under this task order supports the FDA’s effort to implement changes in technology, including carrying out a series of tasks leading to the successful completion of the upgrade and migration of the backend Oracle databases and middle tier Application Server components to a new version and architectural platform. In addition, the contractor shall recommend and implement changes required for optimization of the database and application server performance in the new configuration and provide ongoing maintenance for and upgrade of the Oracle Identity Access Management infrastructure components, provide Middleware Server Administration Support and Deployment (Development, Test, Pre-Production and Production environments), and provide O&M support services for Oracle Enterprise Manager Cloud Control 13c and greater, Transparent Data Encryption, and essential tools used in the monitoring and maintenance of databases and application servers and also provide Operation and Maintenance support of the APM system in compliance with Health and Human Services

(HHS) Enterprise Performance Life Cycle (EPLC) in the FDA shared environment.

3. Scope

The Contractor shall provide ongoing support for the Oracle Identity Access Management Suite 11g, and 12c for shared enterprise use in development, test, pre-prod and production environments for FDA, including patching and version upgrades, as required.

The Contractor shall provide ongoing Middleware Server Administration Support and Deployment (Development, Test, Pre-Production and Production environments)

The Contractor shall provide ongoing support for Oracle Enterprise Manager Cloud Control 12c and higher for shared, Enterprise use in WODC.

The contractor shall complete the implementation and testing of database server software version upgrades from Oracle 12c to Oracle 19c and higher, and the migration of Oracle databases on end of life hardware to either Oracle Exadata running Oracle Linux, or x86 hardware running Oracle Linux. The contractor shall perform analysis, develop plans, write scripts and documentation for implementation, and perform upgrades and migrations.

Also included is capacity planning, resource utilization analysis and performance tuning for the databases residing on the, x86 and Exadata hardware platforms. All of these upgrade and migration activities shall be done in the Development and Test environments at the White Oak Data Center (WODC), and in the Pre-Production and Production environments at the Ashburn Data Center (ADC).

Upgrade tasks include creating scripts and templates to facilitate the upgrades, and scheduling and coordinating the enterprise-wide effort to upgrade all OIM-managed databases to version 19c or greater.

In addition to the database-related activities, there are tasks associated with the upgrade of the middle-tier components. The contractor will function as the process architect for upgrades of Oracle WebLogic 11g, 12.1.3 and 12.2.1.x to WebLogic 12.2.1.4, including but not limited to creating schedules, identifying roles and responsibilities and guiding the various contractor and application teams through the upgrade process. All of these upgrade and migration activities will be done in the Development and Test environments at the White Oak Data Center (WODC), and in the Pre-Production and Production environments at the Ashburn Data Center (ADC).

Also, the contractor will be responsible to provide the Operation and Maintenance (O&M) support and Project Management of FDA APM system through the EPLC process. This work will include but is not limited to:

Maintenance of AppDynamics controller and monitoring agents in Pre-Production, Production, Intranet and Extranet environments, maintaining configuration of the AppDynamics agents for the FDA applications, maintenance of AppDynamics dashboard for application performance monitoring, and system security annual review for the FDA APM solution and maintain ATO for APM platform.

4. General Requirements

Task 1: Oracle Identity Access Management (OIAM) Application Server Administration, Production Support (11g, 12c, and greater concurrently)

The Contractor shall provide Support for Oracle Identity Access Management (OIAM) Application Server Administration, Production Support which includes supporting 11g, 12c, and greater versions concurrently.

Additionally, provide ad-hoc support for the Oracle Internet Directory (OID) and SSO environments for Center for Drug Evaluation and Research (CDER) FDA Adverse Event Reporting System (FAERS). This includes all environments in WODC and Ashburn Data Center (ADC) – development, test, pre-production, production environments.

The current Identity Management server configuration is as follows:

The CDER FAERS 10g Identity Access Management Suite server configuration includes one Windows Virtual Machine server per environment in WODC and CHDC for a total of six environments in two data centers which includes Development, Test, Training, Migration, Pre-Production and Production.

The Oracle Identity Access Management Suite 11g stack includes Oracle Internet Directory (OID), Oracle Virtual Directory (OVD), Oracle Access Manager (OAM) and Oracle Identity Manager (OIM) with Service- Oriented Architecture (SOA). The current 11g Identity Access Management server configuration includes two Exalogic compute nodes per environment in development and test in WODC. There are an additional two Exalogic compute nodes per environment in pre-production and production at CHDC. The configuration also includes one database per environment, running on two Real Application Cluster (RAC) database nodes on Exadata. The Contractor shall provide O&M support services for a total of eight Exalogic compute nodes, four in each of the two data centers. The Contractor shall also provide application DBA support for the OIAM 11g databases running in each environment.

The Oracle Identity Access Management Suite 12c stack includes Oracle Internet Directory (OID), Oracle Unified Directory, and Oracle Access Manager. The 12c Identity Access Management server configuration consists of two Virtual Machines per environment (development, test, pre-production, and production) (development, test, preproduction, production) and one Oracle RAC database per environment (development, test, preproduction, production). The Contractor shall provide O&M support for a total of eight Virtual Machines. The Contractor shall also provide application DBA support for OIAM 12c databases running in each environment.

The Contractor shall:

1.1 Maintain the Active Directory (AD) to OID synchronization to ensure full user replication between the Authoritative source and the OID in Development, Test, Pre-Production and Production.

1.2 Re-evaluate and adjust hardware resource allocation for FDA Enterprise OIAM systems, including storage, memory, network resources, Operating System configuration, account security for the data and the middle tier levels for the FDA Enterprise shared environment, based on system additions and deletions.

1.3 Maintain and troubleshoot the Windows Native Authentication (WNA) for all intranet application users in the FDA Enterprise OIAM system in Development, Test, Pre-Production and Production.

1.4 Maintain and patch the database tier for the Middleware metadata repository in Development and Test.

1.5 Apply software patches to the FDA Enterprise 11g and 12c or greater OIAM system to keep up with Security and other changing requirements such as OIAM system access using the Windows 10 and Microsoft Edge Chromium technologies in Development, Test, Pre-Production and Production.

1.6 Maintain daily backup scripts and monitor logs for status. Periodically test backup/recovery procedures (quarterly).

1.7 Routinely (monthly) measure FDA Enterprise OIAM Application Servers performance and up-time metrics in Development, Test, Pre-Production and Production.

1.8 Troubleshoot and resolve performance issues.

1.9 Respond to tickets created by Application owners to OIAM-enable their applications, provide documentation, and begin to troubleshoot issues within 2 businesses day. Applications may be located on premise, within the various cloud environments, and/or hosted by a software as a service (SaaS) provider.

1.10 Register new / de-register retired or incorrect middle tier installs and application Virtual hosts with the FDA Enterprise OIAM system in Development, Test, Pre-Production and Production.

1.11 Plan, develop schedule for, and implement a phased migration of all OIAM 11g clients to the 12c or greater system.

1.12 Integrate Oracle WebLogic 11g and 12c with OIAM 12c and greater.

1.13 Maintain FAERS 10g, 11g, or 12c and greater OIAM on Development, Test, Pre-Production, Production, Training and Migration environments.

1.14 Provide x509 configuration as part of OIAM12c upgrade for applications as required and as an alternative solution to WNA.

1.15 Ensure 99.9% uptime, except during agreed-upon maintenance windows. In addition, there shall be no more than 3 unscheduled outages during the Period of Performance, assuming the servers are available and the network access is intact.

Task 2: Middleware Server Administration Support and Deployment (Development, Test, Pre-Production and Production environments)

The Contractor shall provide Full Middleware Administration Services for the Enterprise Shared Environment in the WODC Data Center and Full Middleware Deployment Services for the Pre-Production and Production Environments in the ADC Data Center.

2.1 Act as a Technical Middleware coordinator by interfacing with other teams and vendors, including application, network, storage, systems admin, load balancer, firewall, and security teams as well as external Oracle Support.

2.2 Interface with clients, attend meetings with the customer on various application issues and planning activities and provide recommendations and solutions.

2.3 Provide expertise and recommendations on FDA Reference architecture and ensure that applications architecture aligns with industry best practices.

2.4 Assist Application teams in producing the overall technical design and build of the middleware application deployments.

2.5 Participate and/or lead troubleshooting and tuning sessions with development teams, end users, network team, firewall team, and any other required teams to diagnose and address application deployment issues and deployment-related performance issues.

2.6 Install and configure any additional software components beyond the base-level middleware install provided by the Data Center Middleware Administrator.

2.7 Provide routine application deployment and re-deployment services to all FDA centers

2.8 Provide application upgrade and migration services to all FDA Centers as part of the efforts to upgrade WebLogic software stacks

2.9 Configure, maintain, and tune metadata repositories used by the Middleware components, including purge script deployment for record cleanup.

2.10 Provide integration support between various components in the Web, Application, Data tiers, and the management zone.

2.11 Support FDA load testing efforts.

2.12 Create and guide application teams with deployment Requests for Change (RFCs), Load Balancer

Farm request, and Firewall rule requests. The Contractor shall produce easy-to-use process document for applications to follow for RFC Creation, Load Balancer Farm request, and Firewall Rule request within 30 days of task order award.

2.13 Review Version Description Documents/installation instructions and provide feedback to application teams prior to scheduled release dates.

2.14 Confirm availability of deployment resources with application teams and Change Control Board

(CCB).

2.15 Work with application teams to identify alternate release dates, in the event of resource constraints or conflicts with previously scheduled maintenance activity.

2.16 Work with Division of Infrastructure Operations (DIO) and Security to provide mechanism for staging release packages.

2.17 Evaluate and Assign deployment RFCs based on the requirement and skill level required.

2.18 Work with the Peraton CSS to help Identify new applications for deployment automation via

Application Modernization and DevOps Services(AMDOS)

2.19 Work with application teams to help on-board them to use AMDOS for automated deployments in the Development, Test, Pre-Production and Production environments,

2.20 Provide Legacy Puppet application deployment services to all FDA centers using ADI in the Pre-

Production and Production environments

2.21 The Contractor shall produce a weekly report by Close of Business (COB) Friday of all Open RFCs involving Middleware resources, along with the Middleware resource assigned.

2.22 Evaluate software infrastructure and processes in contractor managed environments for automation and. develop an automation plan for the identified areas.

Task 3: O&M support services for Oracle Enterprise Manager Cloud Control 12c and higher in WODC.

The Contractor shall provide O&M support services for Oracle Enterprise Manager Cloud Control 13c in WODC. This includes eight servers (four Oracle 12c RAC database servers, four Oracle OEM Cloud 13c servers).

3.1 Perform OEM 13c or greater management, including, adding and configuring management agents, adding and managing OEM user accounts, adding and managing managed target groups, and troubleshooting and resolving alerts from agents.

3.2 Configure and implement notification rules for the FDA Enterprise.

3.3 Manage server jobs in the OEM 13c or greater environment.

3.4 Provide technical assistance to application owners while adding agents to servers.

3.5 Provide full and ongoing operations and management support in the 12c or greater environment.

3.6 Ensure 99.9% uptime, except during agreed-upon maintenance windows. In addition, there shall be no more than three unscheduled outages during the period of performance, assuming the servers are available, and the network access is intact.

3.7 Upgrade and patch 13c or greater OMS and agent environments with new patches and releases.

Task 4: Support for Oracle database upgrade and migrations in Development, Test, Pre-Production, and Production environment

Oracle database upgrade/migration activities shall include but are not limited to the following activities.

4.1 Create project plans to migrate any remaining Oracle 11g databases residing on Sun\Solaris LDOMS to Exadata or x86 hardware platforms

4.2 Develop upgrade templates and guidelines for use by the application teams in upgrading their 12c databases to Oracle 19c or greater.

4.3 Schedule and conduct Kick-Off meetings with the Application Owners, Project Managers, Security, and Technical Teams for each migration group to explain migration strategy, identify tasks, and establish timelines.

4.4 Work with Citrix teams to ensure proper connectivity data (tnsnames) is provided in all four environments (Development, Test, Pre-Production, Production).

4.5 Assist application teams in preparing for pending database upgrades and migrations. Perform Request for Change (RFC) process activities related to migrations.

4.6 Represent RFC(s) at Infrastructure Change Control Board (ICCB) meeting in order to gain approval.

4.7 Communicate actual migration dates to technical teams and coordinate transition of applications to

Operations and Maintenance mode, following Standard Operating Procedures provided by FDA.

4.8 Execute upgrade/migration in each environment; Development, Test at WODC and Pre-Production and

Production at ADC

4.9 Coordinate with WODC and ADC data centers contractors to ensure all infrastructures, including target databases, are in place ahead of planned upgrades and migrations.

4.10 Assist application teams in conducting performance tests and functionality tests against this new software version.

4.11 Resolve any problems/issues identified in upgrade/migration of databases at WODC.

4.12 Exadata Performance Tuning -- Assist targeted application teams in post-upgrade performance tuning on the Exadata platform. Note that Exadata performance tuning requires specialized expertise.

Determine resource reclamations possible at the memory and CPU level for all databases deployed to Exadata. Analyze performance data, looking for signs of SGA/PGA over-allocation.

4.13 Provide recommendations to FDA management and application teams for performance improvements and reclamation of unused resources.

4.14 Prioritize databases based on most highly utilized, largest footprints, etc. to help ensure that reclamation actions will provide the greatest benefit

4.15 Assist application teams in reducing memory allocations where appropriate

Task 5: Support for Oracle Application Server upgrades in Development, Test, Pre-Production, and Production environments

Oracle Application Server WebLogic older version to WebLogic version 12c or greater upgrade\migration activities shall include but are not limited to the following activities:

5.1 Create project plans to assist application teams in preparing for pending application server upgrades.

5.2 Schedule and conduct Kick-Off meetings with the Application Owners, Project Managers, Security, and

Technical Teams for each migration group to explain migration strategy, identify tasks, and establish timelines.

5.3 Review planned application server configuration with application teams, application server administrators, and database administrators.

5.4 Guide Business Owners, System Owners, Application Leads, and DBA Leads in preparation for migrations.

Schedule and conduct follow-up migration status meetings to track progress and resolve issues.

5.5 Perform Request For Change (RFC) process activities related to migrations.

5.6 Represent RFC(s) at Infrastructure Change Control Board (ICCB) meeting in order to gain approval.

5.7 Communicate actual migration dates to technical teams and coordinate transition of applications to Operations and Maintenance mode, following Standard Operating Procedures provided by FDA.

5.8 Assist application teams in preparing for and executing upgrade-related tasks including creation of Current State

Diagrams, Target State Diagrams, and firewall rule requests.

5.9 Validate application team artifacts related to Oracle upgrades.

5.10 Coordinate with WODC and ADC data center contractors to ensure all required infrastructure is in place in time for planned application server upgrades and migrations.

5.11 Assist application teams in conducting performance and functionality tests against this new software version.

5.12 Resolve any problems/issues identified in upgrade/migration of databases at WODC and ADC.

Task 6: Operation and Maintenance Support for FDA APM solution using the AppDynamics

6.1 APM Release Support: The contractor shall provide O&M release support throughout the period of performance. This includes providing support to FDA application owners for AppDynamics agents deployment, maintaining multiple copies of APM software in four physical environments (Development, Test, Intranet and Extranet) to handle upgrades and testing software upgrades and patches, creating RFC and Version Description Document (VDD) for Test, Pre-Prod and Production releases, creating roll back scripts, and other supporting technical documentation. Additionally, the contractor shall continue to develop and enhance automation scripts for Linux based AppDynamics machine agent deployments and upgrades. Depending on the availability of hardware and AppDynamics software licenses, the contractor shall plan for and complete the deployment of AppDynamics platform into the FDA’s Test and also AWS GovCloud environments during the period of performance increasing the number of active environments from 4 to 6. Currently FDA roughly has 1500 -licenses for applications performance monitoring

6.2 Operations and Maintenance Release: APM Change Control Board (CCB) will determine and prioritize new change requests and system defects during CCB meeting. The Contractor shall ensure all APM system defects, and bug fixes shall be addressed via O&M release. O&M Releases schedule will be determined by Contracting Officer Representative (COR) and Change Control Board (CCB). The contractor shall maintain APM Defects log and Change Request log in JIRA and shall keep it up to date during entire contract period of performance.

The CCB and COR will determine the priority of defects. If a fix is deemed critical by the COR, it will be fixed with an emergency release. Critical issues must be resolved within a 48-hour timeframe. The contractor shall adhere to Change control procedure written out in the APM CCB Charter. The CCB Charter is available in APM SharePoint document repository.

6.3 Service Helpdesk Support: The contractor shall provide responses and resolutions to APM user’s questions and problems opened in ServiceNow within 24 hours of the inquiry.. The contractor shall conduct root cause analysis within 48 hours of the inquiry. Any issues found that require code changes will require the CCB to prioritize and approval before it can be designated and worked on. The assumption is there is an average of 15 helpdesk tickets per month with minimum of 0.5 hours to 3 hours spent per ticket.

6.4 Application monitoring and agent deployment Support: The contractor shall provide application monitoring and AppDynamics agent deployment support throughout the performance period. Support includes coordinating deployment schedule with application technical lead, working with CCB and making configuration changes via a release or a scripted process. Support also includes supporting the installation of software patches that may be required on the system in order to ensure that the system complies with FDA Security and/or data center standards. The contractor shall do thorough analysis of the root cause of bad data and apply fixes as directed by the COR the contractor shall make necessary database updates, testing and will ensure data clean up.

6.5 Application Monitoring Dashboard Support: The contractor shall ensure that all existing Application monitoring dashboards including the Executive Dashboards and canned reports continue to function properly and will be responsible for making minor enhancement as request by application teams.

6.6 System Security Annual Review following NIST Security compliance guidelines: The contractor shall meet and comply with all FDA IT Security policies and all NIST Security standards, guidelines and documentation for annual system certification and accreditation. This also includes deploying and maintaining the audit logging feature for APM, closing any Plan and Action Milestones (POAM) controls found during APM annual C&A process

7 Project Management

The Contractor shall provide the following:

7.1 Contract Level Program Management

Provide the technical and functional activities at the contract level needed for program management of this SOW, including productivity and management methods such as Quality Assurance, Configuration, Work Breakdown Structure, Schedule and Change Control. Provide the centralized administrative, clerical, documentation and other related functions.

7.2 Task Order Management

The Contractor shall prepare a Task Order Management Plan describing the technical approach, organizational resources and management controls to be employed to meet the cost, performance and schedule requirements throughout the task order execution. The Task Order Management Plan shall contain a Work Breakdown Structure and Project Schedule for each task as specified in this Statement of Work. The Contractor shall submit the Task Order Management Plan to the Contracting Officer’s Representative (COR) for approval within 15 days of contract award.

7.3 In Progress Review Support

Provide a monthly status report monitoring progress against the Task Order Management Plan and the project schedule applied to the task order. List everything that has been accomplished during the prior month, broken down according to the project that was worked on. This report shall be delivered by the fifteenth day of the following month.

7.4 Deliverable and Document Dissemination

Each document deliverable shall be delivered via e-mail. When a new document is delivered, or an update to existing documentation is made, the Contractor shall alert the Contracting Officer’s Representative of these changes.

7.5 Meeting Minutes and Agendas

The Contractor shall maintain and provide minutes of all meetings held with the FDA. These minutes shall include the following:

• Summary description of all issues discussed;

• Action Items;

• Personnel to whom the Actions Items are assigned;

• List of meeting attendees, including name, organization, phone number, and e-mail address.

All agendas shall be delivered at least one working day prior to the meeting unless the meeting is called at the last minute.

All meeting minutes shall be delivered within four working days following the meeting.

7.6 Weekly Status Meetings

Weekly status meetings shall be held with the FDA Project Team to discuss general status of the project and any issues as they occur, as well as to gain any FDA management updates and feedback.

Attend ad hoc meetings as needed. Assist in producing reports to management detailing data center migration status and answering various data calls. Review of project risks shall occur four times a year as a part of the status meeting. (January, April, July, October)

All agendas shall be delivered at least one working day prior to the meeting.

7.7 Informal Project Review Meetings

Informal project reviews will occur throughout the project. The primary purpose of these informal reviews shall be to discuss specific project activities and to address any potential problem areas that might arise, or have already arisen. These reviews may be called by the FDA COR or by the Contractor (as approved by the COR) on an as-needed basis and may include participants from FDA senior management.

Two formal Project Risk Reports will be held during the Period of Performance.

All agendas shall be delivered at least one working day prior to the meeting.

8. Deliverables

Task

Description Estimated Quantity / Frequency

Due Date

Delivered to

1.6 Log Status Review Report Monthly By the 15th of each

month

COR

1.6 Backup/Recovery Test Results Quarterly December, March, June, September

COR

4.1.1

4.1.2

Provide project schedule detailing tasks and timeframes for each database upgrade\migration

As needed

Ongoing

COR

4.1.3

4.1.5

Develop upgrade templates for database upgrade\migration

As needed Ongoing

4.1.6 Provide written proposal for performance

tuning and resource reclamation

Initial version then updated as needed

Within 30 days of task order award

COR

4.2.2 Provide project schedule detailing tasks

and timeframes for each application upgrade

1.7, 3.6 Application Server/OEM 12c Uptime and Performance Report

Monthly By the 15th of each month

COR

1.11 Schedule for OIAM 11g and 12c client

upgrades As needed Within 45 days of task order award

2.7 Routine application deployment and re-

deployment services

15 per week

2.11 RFC, Load Balancer and Firewall Rule

Process Document

One

Within 45 days of

2.18

RFC Assignment

Within 3 business days of creation

6.1

Weekly report of all Open RFCs Weekly

Friday COB

6.2 Task Order Management Plan

One (1)

Within 15 days of

6.4 6.5, 6.6

Monthly Status Report Monthly

By the 10th of each month

6.4 6.5, 6.6

Meeting Agenda

One (1) day prior to meeting

6.5 Meeting Minutes

Four (4) days after meeting

6.7 Project Risk Report

Two (2)

July, September

9. Place of Performance

The Contractor will be working onsite at the Government facility at 11601 Landsdown Street, Rockville, Maryland.

10. Hours of Performance

The Contractor shall ensure that the Development, Test, and Pre-Production environments are up and running during the workweek from 07:00am through 7:00pm unless pre-approved COR or COR designee scheduled downtime has been approved.

The Contractor shall ensure that the production environment is up and running 24/7/365 in order to meet specific customer requirements unless COR or COR designee scheduled downtime has been approved.

11. Government Furnished Equipment

Government Furnished Equipment will consist of laptop computers and telephones.

12. Contract Type

The contract type for this Task Order is Firm-Fixed-Price.

13. Period of Performance

The anticipated period of performance shall be Base: May 1, 2022 through October 31, 2022.

Optional: Option Year 1: November 1, 2022 through October 31, 2023 Optional: Option Year 2: November 1, 2023 through October 31, 2024 Optional: Option Year 3: November 1, 2024 through October 31, 2025 Optional: Option Year 4: November 1, 2025 through October 31, 2026

14. Technical Guidance Letter

1. As necessary, technical guidance or clarification concerning the details of specific services to be performed under the terms of this contract/order shall be given through issuance of Technical Guidance Letters (TGL) by the Contracting Officer's Representative (COR) using email or other electronic means. To be valid a TGL, it must be within the general scope of work stated in the contract/order, and it would not require any increase to the negotiated price and/or adjust the delivery terms under the contract.

2. Each TGL issued hereunder is subject to the terms and conditions of this contract. It shall be in writing and include, as a minimum, the following information:

a. Effective date of TGL;

b. Contract/Order and sequential TGL number;

c. Reference to the relevant section in the statement of work; and d. Requirements to be performed.

3. The Contractor shall not comply with any TGL, if the Contractor believes it is not a valid TGL. In the event of a conflict between a TGL and the scope of the contract/order, the terms of the contract/order shall prevail. If the Contractor believes or has reason to believe that a TGL is not valid, the Contractor shall notify the COR and Contracting Officer thereof by email within two (2) days of receiving the TGL in question. The Contracting Officer will give appropriate direction to the Contractor and COR to resolve the TGL issue.

4. Oral technical directions may be given by the COR only in emergency circumstances.

The Contractor shall notify the Contracting Officer by email that it has received such direction within 24 hours of having received the directions. If the COR does not follow-up the oral directions within two (2) working days by issuing a written TGL, the Contractor is to notify the Contracting Officer and cease compliance unless the Contractor concurs the direction is a valid TGL.

5. Technical Guidance provided in meetings with minutes submitted by the Contractor for Government acceptance does not need to be documented via TGL. The Technical Guidance shall be documented in the meeting minutes.

6. The Contractor’s failure to comply with this clause is grounds for finding that incurred costs are not allowable.

15. Method of Quality Monitoring

Technical Performance: The Contractor will be evaluated monthly as to the quality of the output of their work and results will be documented in the COR’s file. The Contractor's personnel shall be technically competent in the tasks identified in the SOW or other ordering document under the contract. Included in the technical performance is the Contractor’s contribution in meetings and reviews, the quality of the Contractor's technical reports, the Contractor's productivity and the overall quality of the technical support provided.

SOW

Item No.

Description

Performance Standard

Acceptable Quality Level

Surveillance Method &

Assessment 2.18 6.1, 6.2, 6.4, 6.5, 6.6

Documents

All documents are submitted in a timely manner

Documents are 95% accurate, complete, concise, and clear

100% Inspection

4.1.1, 4.1.2, 4.1.3, 4.1.15, 4.2.2, 4.2.3, 6.2, 6.3

Documents

All documents are submitted in a timely manner

Documents are

95% accurate, complete, concise, and clear

100% Inspection

2.7, 4.6 Complete scheduled application deployments and re-deployments

Application deployments completed successfully within timeframe specified in

RFC

Issues logged and resolutions noted in application server issues repository

95% application deployments completed on time

Random Sampling

Customer Feedback

2.16 RFCs Assignment RFCs are assigned to a

resource within three (3) business days.

100% of Middleware RFCs have resource assigned

Random Sampling

Customer Feedback

Inspection and Evaluation Definitions:

• 100 Percent Inspection - With this method, performance is inspected/ evaluated at each occurrence.

This method is often costly but can be necessary due to health, safety and other considerations.

• Random Sampling - Random sampling works best when the number of instances of the services being performed is very large and a statistically valid sample can be obtained.

• Customer Feedback - Although usually not a primary method, this is a valuable supplement to more systematic methods. For example, in a case where random sampling indicates unsatisfactory service, customer complaints can be used as substantiating evidence. In certain situations where customers can be relied upon to complain consistently when the quality of performance is poor, e.g., dining facilities, building services, customer surveys and customer complaints may be a primary surveillance method, and customer satisfaction an appropriate performance standard. In all cases, complaints shall be documented, preferably on a standard form.

16. Government Points of Contact

Project Manager and Technical Point of Contact:

Frank Farrell U.S. Food and Drug Administration Office of Information Management and Technology 3WFN, Room 11C66 11601 Landsdown Street North Bethesda, MD 20852 301-796-2970 Frank.Farrell@fda.hhs.gov

Contracting Officer’s Representative (COR) Delores Johnson U.S. Food and Drug Administration Office of Information Management and Technology 3 White Flint North: Rm 11A21 11601 Landsdown Street North Bethesda, MD 20852 301-796-7771 delores.johnson@fda.hhs.gov

Contracting Officer (CO)

TBD

The COR may be changed at any time by the Government without prior notice to the Contractor by a unilateral modification to the Contract. The responsibility and limitation of the COR are as follows: (1) The COR is mailto:Frank.Farrell@fda.hhs.gov mailto:delores.johnson@fda.hhs.gov mailto:delores.johnson@fda.hhs.gov responsible for the technical aspects of the project and serves as the technical liaison with the Contractor. The COR is also responsible for the final inspection and acceptance of all reports, and such other responsibilities as may be specified in the contract. (2) The COR is not authorized to make any commitments or otherwise obligate the Government or authorize any changes which affect the Contract price, terms or conditions. Any Contractor request for changes shall be referred to the Contraction Officer directly or through the COR. No such changes shall be made without the expressed prior authorization of the Contracting Officer (CO). The CO may designate assistant or alternate COR to act for the COR by naming such assistant/alternate(s) in writing and transmitting a copy of such designation to the Contractor.

17. Payment and Invoice Instructions The Contractor shall invoice 1/12th the value of the Financial Systems Support requirement task order (FFP) monthly.

FDA Electronic Invoicing and Payment Requirements - Invoice Processing Platform (IPP) (Jan 2022)

(a) All Invoice submissions for goods and or services must be made electronically through the U.S.

Department of Treasury’s Invoice Processing Platform System (IPP).

http://www.ipp.gov/vendors/index.htm

(b) Invoice Submission for Payment means any request for contract financing payment or invoice payment by the Contractor. To constitute a proper invoice, the payment request must comply with the requirements identified in in FAR 32.905(b), “Content of Invoices” and the applicable Payment clause included in this contract, or the clause 52.212-4 Contract Terms and Conditions – Commercial Items included in commercial items contracts. The IPP website address is: https://www.ipp.gov.

(c) -----

(1) The Agency will enroll the Contractors new to IPP. The Contractor must follow the IPP registration email instructions for enrollment to register the Collector Account for submitting invoice requests for payment. The Contractor Government Business Point of Contact (as listed in SAM) will receive Registration email from the Federal Reserve Bank of St. Louis (FRBSTL) within 3 – 5 business days of the contract award for new contracts or date of modification for existing contracts.

(2) Registration emails are sent via email from ipp.noreply@mail.eroc.twai.gov. Contractor assistance with enrollment can be obtained by contacting the IPP Production Helpdesk via email to IPPCustomerSupport@fiscal.treasury.gov or phone (866) 973-3131.

(3) The Contractor POC will receive two emails from IPP Customer Support, the first email contains the initial administrative IPP User ID. The second email, sent within 24 hours of receipt of the first email, contains a temporary password. You must log in with the temporary password within 30 days.

(4) If your company is already registered to use IPP, you will not be required to re-register.

(5) If the Contractor is unable to comply with the requirement to use IPP for submitting invoices for payment as authorized by HHSAR 332.7002, a written request must be submitted to the Contracting Officer to explain the circumstances that require the authorization of alternate payment procedures.

(d) Invoices that include time and materials or labor hours Line Items must include supporting documentation to (1) substantiate the number of labor hours invoiced for each labor category, and (2) substantiate material costs incurred (when applicable).

(e) Invoices that include cost-reimbursement Line Items must be submitted in a format showing expenditures for that month, as well as contract cumulative amounts.

(1) At a minimum the following cost information shall be included, in addition to supporting documentation to substantiate costs incurred.

- Direct Labor - include all persons, listing the person's name, title, number of hours worked, hourly rate, the total cost per person and a total amount for this category;

- Indirect Costs (i.e., Fringe Benefits, Overhead, General and Administrative, Other Indirects)- show rate, base and total amount;

- Consultants (if applicable) - include the name, number of days or hours worked, daily or hourly rate, and a total amount per consultant;

- Travel - include for each airplane or train trip taken the name of the traveler, date of travel, destination, the transportation costs including ground transportation shown separately and the per diem costs. Other travel costs shall also be listed;

- Subcontractors (if applicable) - include, for each subcontractor, the same data as required for the prime Contractor;

- Other Direct Costs - include a listing of all other direct charges to the contract, i.e., office supplies, telephone, duplication, postage; and

- Fee – amount as allowable in accordance with the Schedule and FAR 52.216-8 if applicable.

(f) Contractor is required to attach an invoice log addendum to each invoice which shall include, at a minimum, the following information for contract administration and reconciliation purposes:

(1) list of all invoices submitted to date under the subject award, including the following:

- invoice number, amount, & date submitted

- corresponding payment amount & date received

- total amount of all payments received to date under the subject contract or order

- and, for definitized contracts or orders only, total estimated amounts yet to be invoiced for the current, active period of performance.

(g) Payment of invoices will be made based upon acceptance by the Government of the entire task or the tangible product deliverable(s) invoiced. Payments shall be based on the Government certifying that satisfactory services were provided, and the Contractor has certified that labor charges are accurate.

(h) If the services are rejected for failure to conform to the technical requirements of the task order, or any other contractually legitimate reason, the Contractor shall not be paid, or shall be paid an amount negotiated by the CO.

(i) Payment to the Contractor will not be made for temporary work stoppage due to circumstances beyond the control of U.S. Food and Drug Administration such as acts of God, inclement weather, power outages, and results thereof, or temporary closings of facilities at which Contractor personnel are performing. This may, however, be justification for excusable delays.

(j) The Contractor agrees that the submission of an invoice to the Government for payment is a certification that the services for which the Government is being billed, have been delivered in accordance with the hours shown on the invoices, and the services are of the quality required for timely and successful completion of the effort.

(k) Questions regarding invoice payments that cannot be resolved by the IPP Helpdesk should be directed to the FDA Employee Resource and Information Center (ERIC) Helpdesk at 301-827-ERIC (3742) or toll-free 866-807-ERIC (3742); or, by email at ERIC@fda.hhs.gov. Refer to the Call-in menu options and follow the phone prompts to dial the option that corresponds to the service that's needed. All ERIC Service Now Tickets will either be responded to or resolved within 48 hours (2 business days) of being received. When emailing, please be sure to include the contract number, invoice number and date of invoice, as well as your name, phone number, and a detailed description of the issue.

18. Section 508 Standards

HHSAR Clause 352.239-74 Electronic and Information Technology Accessibility is applicable to this task order.

See clause section for full text.

19. Key Personnel

The Key Personnel specified in each Order are considered to be essential to work performance. At least 30 days prior to diverting any of the specified individuals to other programs or contracts (or as soon as possible, if an individual must be replaced, for example, as a result of leaving the employ of the contractor), the contractor shall notify the Contracting Officer and shall submit comprehensive justification for the diversion or replacement request (including proposed substitutions for Key Personnel) to permit evaluation by the Government of the impact on performance under each Order. The contractor shall not divert or otherwise replace any Key Personnel without the written consent of the Contracting Officer. The Government may modify the Orders to add or delete key personnel at the request of the contractor or Government.

Role Principal Consultant III (Database Management Specialist) Principal Consultant III (Oracle Identity Access Management (IAM) Suite Engineer) Principal Consultant III (Middleware Engineer) Principal Consultant III (AppDynamics Engineer)

20. Security and Privacy Requirements

A. Baseline Security Requirements

1) Applicability. The requirements herein apply whether the entire contract or order (hereafter “contract”), or portion thereof, includes either or both of the following:

2) Access (Physical or Logical) to Government Information: A Contractor (and/or any subcontractor) employee will have or will be given the ability to have, routine physical (entry) or logical (electronic) access to government information.

a. Operate a Federal System Containing Information: A Contractor (and/or any subcontractor) will operate a federal system and information technology containing data that supports the HHS mission. In addition to the Federal Acquisition Regulation (FAR) Subpart 2.1 definition of “information technology” (IT), the term as used in this section includes computers, ancillary equipment (including imaging peripherals, input, output, and storage devices necessary for security and surveillance), peripheral equipment designed to be controlled by the central processing unit of a computer, software, firmware and similar procedures, services (including support services), and related resources.

3) Safeguarding Information and Information Systems. In accordance with the Federal Information Processing Standards Publication (FIPS)199, Standards for Security Categorization of Federal Information and Information Systems, the Contractor (and/or any subcontractor) shall:

a. Protect government information and information systems in order to ensure:

• Confidentiality, which means preserving authorized restrictions on access and disclosure, based on the security terms found in this contract, including means for protecting personal privacy and proprietary information;

• Integrity, which means guarding against improper information modification or destruction, and ensuring information non-repudiation and authenticity; and

• Availability, which means ensuring timely and reliable access to and use of information.

b. Provide security for any Contractor systems, and information contained therein, connected to an FDA network or operated by the Contractor on behalf of FDA regardless of location. In addition, if new or unanticipated threats or hazards are discovered by either the agency or contractor, or if existing safeguards have ceased to function, the discoverer shall immediately, within one (1) hour or less, bring the situation to the attention of the other party. This includes notifying the FDA

Systems Management Center (SMC) within one (1) hour of discovery/detection in the event of an information security incident.

c. Adopt and implement the policies, procedures, controls, and standards required by the HHS/FDA Information Security Program to ensure the confidentiality, integrity, and availability of government information and government information systems for which the Contractor is responsible under this contract or to which the Contractor may otherwise have access under this contract. Obtain the FDA Information Security Program security requirements, outlined in the FDA Information Security and Privacy Policy (IS2P), by contacting the CO/COR or emailing your ISSO.

d. Comply with the Privacy Act requirements and tailor FAR clauses as needed.

4) Information Security Categorization. In accordance with FIPS 199 and National Institute of Standards and Technology (NIST) Special Publication (SP) 800-60, Volume II: Appendices to Guide for Mapping Types of Information and Information Systems to Security Categories, Appendix C, and based on information provided by the ISSO or other security representative, the risk level for each Security Objective and the Overall Risk Level, which is the highest watermark of the three factors (Confidentiality, Integrity, and Availability) of the information or information system are the following:

Confidentiality: [ ] Low [ X ] Moderate [ ] High

Integrity: [ ] Low [ X ] Moderate [ ] High

Availability: [ ] Low [X ] Moderate [ ] High

Overall Risk Level: [ ] Low [ X] Moderate [ ] High

Based on information provided by the Privacy Office, system/data owner, or other privacy representative, it has been determined that this solicitation/contract involves:

[ ] No PII [ X ] Yes PII

Complete this section using the information obtained from the Security and Privacy Checklist in Appendix A, parts A and B.

Personally Identifiable Information (PII). Per the OMB Circular A-130, “PII is information that can be used to distinguish or trace an individual's identity, either alone or when combined with other information that is linked or linkable to a specific individual.” Examples of PII include, but are not limited to the following: Social Security number, date and place of birth, mother’s maiden name, biometric records, etc.

PII Confidentiality Impact Level has been determined to be: [ ] Low [ X ] Moderate [ ] High

5) Controlled Unclassified Information (CUI).