Att 2 - RFI_FY26-ICE-FFMS-OM_SOW_25FEB25.pdf

FFMS Infrastructure O&M SOW

Statement of Work (SOW) for

Federal Financial Management System (FFMS) Infrastructure

Operations & Maintenance (O&M)

February 25, 2025

Table of Contents

1.0 General

1.1 Background

1.2 Scope

1.3 Objective

2.0 Specific Requirements/Tasks

2.1 Project Management - CLIN 0001

2.2 Independent Verification and Validation (IV&V) Testing – CLIN 0002

2.3 Infrastructure Operations & Maintenance (O&M) – CLIN 0003

2.4 OFM General Support Systems (GSS) Support – CLIN 0004

2.5 User & Interface Password Management Support – CLIN 0005A (Optional)

2.6 Business Intelligence Reports and Dashboards Support– CLIN 0005B (Optional)

2.7 Integrated Data Services Support - CLIN 0005C (Optional)

2.8 Reports, Dashboards and API Development – CLIN 0005D (Optional)

2.9 Surge Support – CLIN 0005E (Optional)

2.10 Other Direct Costs (ODCs) – CLIN 0009

3.0 Contractor Personnel

4.0 Other Applicable Conditions

5.0 Government Furnished Equipment (GFE)

6.0 Contractor Furnished Property

7.0 Government Acceptance Period

8.0 [Reserved]

9.0 Deliverables

10.0 Section 508 Requirements

10.1 Section 508 Requirements for Technology Services

10.2 Section 508 Deliverables

11.0 Attachments

1.0 General

1.1 Background

Federal Financial Management System (FFMS), a commercial-off-the-shelf (COTS) single integrated application, was acquired in 1997 by the U.S. Immigration and Naturalization Service (INS). The system transitioned to ICE when the department and agency were formed on March 1, 2003. As part of the transition, ICE became the financial service provider for seven other DHS Components Cybersecurity Infrastructure Security Agency (CISA), Office of Biometric Identity Management (OBIM), United States Citizenship and Immigration Services (USCIS), Science and Technology (S&T), Office of Health Affairs (OHA), and DHS Management (MGMT)) and FFMS continued as the financial system of record. ICE and its customers use FFMS to support the tracking and recording of transactions through each agency’s financial process. This data is immediately available only to authorized users within each agency.

The FFMS Infrastructure is the enterprise-wide solution that supports the delivery of FFMS to users and is composed of virtual resources that process, store, analyze, and direct the flow of FFMS data. The FFMS Infrastructure consists of production environment in Amazon Web Services (AWS) East and disaster recovery (DR) and test (UAT) environments in AWS West.

The FFMS and Office of Financial Management General Support System (OFM GSS) infrastructure consists of virtualized Microsoft Windows servers. The FFMS application utilizes using Oracle Fusion Middleware (FMW) as well as serverless clusters while OFM GSS utilizes Java, Spring Boot, Microsoft Access, and Desktop Visual Basic. The FFMS database engine is Oracle DBMS and the OFM GSS database engine is Microsoft SQL both are hosted on RedHat Enterprise Linux for administrative and technical applications/utilities supporting FFMS. The contractor manages the infrastructure related to FFMS and OFM GSS, including standing up new instances as needed. Below is a summary of the current resources in the FFMS environments. The current technology stack is subject to software updates/changes/enhancements.

Production Windows VMs:

• 4 SSO Servers

• 4 Application Pooled Servers

• 1 Jump Server

• 1 Interface Server

• 1 Management Tools and App Server

• 1 OFM GSS Host Server

Production RHEL VMs:

• 1 Transactional Database Instance with 9 pluggable databases

• 1 Reporting Database Instance with 13 pluggable databases

• 1 OFM GSS SQL DB

Production Serverless Instances

• 2 BI Reporting Tool

• 2 Process Automation

Non-Production Windows VMs

• 4 SSO Servers

• 2Application Pooled Servers

• 1 Interface Server

• 1 Management Tools and App Server

• 1 OFM GSS Host Server Non-Production RHEL VMs

• 1 Transactional Database Instance with 10 pluggable databases

• 1 Reporting Database Instance with 9 pluggable databases

• 1 OFM GSS SQL DB

Non-Production Serverless Instances

• 2 BI Reporting Tool

• 2 Process Automation Tool

Disaster Recovery Windows VMs

• 4 SSO Servers

• 2 Application Pooled Servers

• 1 Interface Server

• 1 Management Tools and App Server

Disaster Recovery RHEL VM

• 1 Transactional Database Instance matching production

AWS Web and Application components

• Network and Security

• Load Balancing

• CloudWatch Monitoring

• Code Pipelines

• Lamda Functions

• Other VM functionality to support the infrastructure

The current technology stack of the middle-tier to be supported consists of the following software and utilities or their equivalent:

Third Party Software Subcomponent Vendor Oracle HTTP Server (OHS) Fusion Middleware Oracle Oracle App Server / WebLogic Server

(WLS)

Fusion Middleware Oracle

Oracle Forms Fusion Middleware Oracle Oracle APEX Fusion Middleware Oracle Oracle Reports Fusion Middleware Oracle Oracle Access Manager (OAM) Fusion Middleware Oracle Oracle Internet Directory (OID) Fusion Middleware Oracle Coherence Fusion Middleware Oracle SQL Loader Fusion Middleware Oracle Oracle Forms WebUtil Server-Side

Libraries Oracle

Sterling/IBM Connect Direct P2P Integration IBM SFTP Client Reflection for Security IT Server

P2P Integration MicroFocus

GPG4Win Encryption P2P Integration The Gpg4win initiative 7-Zip Developer Tools Igor Pavlov NotePad++ Developer Tools Don Ho

Jasper Reports Server BI Reporting Tool Tibco Software Inc.

Airflow Automation Apache Software Foundation JDK Application Support Open Source ASP.NET Core Application Support Open Source

1.2 Scope

The Contractor shall perform the infrastructure support services within these high-level areas:

• Architecture Management

• Availability Management

• Capacity Management

• Configuration Management

• Incident Management

• IT Operations

• Problem Diagnostics

• Release and Deployment Management

• Security Management

• Technical Management

• Performance Management

• Database management

1.3 Objective

The objective of the FFMS Infrastructure is to support firmware and software for O&M services for the existing and future environments. This shall ensure FFMS remains adaptable, accessible, interoperable, connected, secured, and standards compliant. The scope of this SOW does not include FFMS application software maintenance support. The support for FFMS Infrastructure O&M includes support for all deployments (i.e. interface code is updated by the FFMS application software vendor, and the deployments to the FFMS application are completed by the contractor. This shall also ensure that OFM GSS and other identified software tools that support the Chief Financial Officer (CFO) remain adaptable, accessible, interoperable, connected, secured, and standards compliant.

2.0 SPECIFIC REQUIREMENTS/TASKS

2.1 Project Management - CLIN 0001

The contractor shall collaborate with ICE OCIO and ICE OFM Information Technology Program Managers (ITPMs), for but not limited to FFMS and other projects as defined within the statement of work, to implement all agreed upon governance and other applicable processes and to define the appropriate service level and other service requirements.

The Contractor shall complete the following tasks but not limited to:

o Ensure all work complies with contract terms and conditions.

o For releases, draft and track project plan elements, including associated documentation, requirements management, report progress of tasks and milestones in accordance with ICE SLM (System Lifecycle Management).

o Assign, administer, and manage contractor personnel and resources based on project plan.

Communicate progress to ICE OCIO and ICE OFM ITPMs, and the ICE Contracting

Officer’s Representative (COR).

o Provide advance notification of any deviation from schedule or resources. If the deviation from the project schedule or milestone date is greater than three business days, then the contractor shall immediately notify the government.

o Provide assistance and complete documents for FFMS Advisory Board (FAB) or Change Control Board (CCB) meetings.

o Ensure all contract personnel and subcontractors adhere to the Department security policies directives, and guidance.

o Review all contract deliverables for accuracy.

o Provide and deliver occasional/ad hoc presentations and briefings, as needed and obtain government approval for materials used in any Contractor led meetings.

o Support the ICE OCIO agile process methodology to include: 1) attend daily stand-up meetings, 2) Sprint planning and retrospective sessions, 3) participate in the backlog grooming sessions, 4) maintain project status updates in JIRA to include task updates and Kanban board tracking/prioritization, 5) participate in program Integrated Project Team (IPT) meetings.

o Provide a Transition Plan to document key activities and plan to transition out, as needed.

o On a monthly basis, provide project status, including but not limited to; 1) helpdesk ticket metrics, 2) Jira ticket metrics, and 3) overall projects risks.

o Perform a constant assessment and validation of applicable implementation of the Agile methodology and processes as it is mapped for the DHS ICE OCIO client in relation to the FFMS application o Ensure project team support and compliance with security requirements and audits.

o Track and manage running of application, database, operating system and container level scans.

o Ensure that program POA&Ms specific to management of the systems infrastructure are documented and maintained through remediation by the contractor.

o Prioritize security weaknesses for mitigation o Reallocating resources to maximize performance and efficiency o Provide responses to technical inquiries.

o Establish and manage project plans for Infrastructure and application initiatives and epics.

o Manage projects utilizing Agile best practices maintaining at least an agency Agile level II certification.

o Establish a Scrum Master to facilitate Sprint Planning, Daily Stand-ups, Sprint Reviews, and

Retrospectives.

o Manage and deploy infrastructure and supporting software related releases to provide SLM documentation to the ITPM to submit to ELMS and submit change Requests (CRs) into ServiceNow and present to ICE/DHS CCB as well as follow other CCB required processes.

o Support CR and ECR releases as needed for infrastructure and supporting software.

o Support and respond to application data calls.

2.2 Delivery and Independent Verification and Validation (IV&V) Testing – CLIN 0002 Software system change request (SCR) delivery and IV&V shall be performed by a contractor who is not directly engaged in the development of the project, with the purpose of assessing the correctness and quality of the project’s product. The contractor shall review, analyze, evaluate, inspect, and test the project’s product and processes. This analysis shall include the operational environment, hardware, software, interfacing applications, documentation, operators, and users to ensure that the software is well-engineered and is being developed in accordance with customer requirements.

• The Contractor shall complete the following tasks but not limited to:

o Testing of application and application interfaces.

o Testing for application releases (scheduled, emergency, security, etc.)

o Testing of applications and applications supporting software components.

o Testing of the FFMS reporting database functionality o Testing of application interfaces o Testing of business analytics reports o Testing of infrastructure components o Testing of business continuity o Testing of Infrastructure components o Independent Validation and Verification of application changes and interfaces.

o Provide guidance and facilitate the creation of the System Lifecycle Management (SLM) documentation related to the Testing and IV&V activities o Create and review test plans o Create testing verification reports o Perform 508 compliance testing of applications o Support performance in integration and load testing o Participate in user Acceptance testing sessions.

o Participate in daily standups and application IPT meetings.

o Support audit and security inquiries and request as pertaining to IV&V and testing processes.

2.3 Infrastructure Operations & Maintenance (O&M) – CLIN 0003 The contractor shall be responsible for the day-to-day maintenance for but not limited to FFMS and other projects as defined within the statement of work Infrastructure to ensure delivery of contracted IT services to ICE. The contractor shall maintain software support and related resources on the infrastructure, such as servers, VMs, interfaces and their connectivity, scripting, load balancing services, and database suite of products and tools. Infrastructure support shall include development and maintenance of the infrastructure components such as but not restricted to user management, interface reporting, certificate management, and infrastructure component availability status. The contractor shall NOT be responsible for maintaining the FFMS commercial off-the-shelf (COTS) application development. Support shall be provided as defined in but not limited to the following categories:

• Hardware, Virtual Machine, Scripts, Storage, and Serverless Environments Maintenance and Support o Maintain infrastructure console management and tagging of infrastructure resources.

o Maintain the hardware, virtual machines, and serverless environments that support the application.

o Ensure security and effective configuration of hardware, virtual machines, and serverless environments and their security groups.

o Establish and manage implementations for effective resource conservation of the hardware, virtual machines, and serverless environments.

o Establish lambda functions or equivalent solutions for support of the hardware, virtual machines, and serverless environments.

o Implement serverless functionality were feasible.

o Set up monitoring of infrastructure performance and establish baselines performance levels.

o Provide technical recommendations and guidance for hardware, virtual machines, and serverless environments.

o Perform preventative maintenance of the hardware, virtual machines, and serverless environments.

o Ensure that the hardware, virtual machines, and serverless environments is deployed using established enterprise Continuous Integration/Continuous Development pipelines using Ansible playbook and Terraform or alternate enterprise software.

o Standup and Decommission hardware, virtual machines, and serverless environments.

o Provide hardware, virtual machines, and serverless environments break fix solutions.

o Conduct monthly system optimization through scalable configuration (performance, cost, etc.)

o Ensure that all hardware, virtual machines, and serverless environments are documented within confluence.

o Ensure that hardware, virtual machines, and serverless environment configurations are documented or linked to in the System Design Document and that SOPs are established for administering the environment.

o Utilize single point for web farm management to reduce operational maintenance o Provide infrastructure guidance for improved stability of the system environment.

o Research and document key steps related to hardware, virtual machines, and serverless environment upgrades that may be required to ensure continued Contractor support of ICE’s financial system of record infrastructure.

o Review and monitor daily databases backup logs and ensured backup dataset to be copied into AWS S3 bucket for Production and UAT.

o Maintain AWS S3 bucket storage and automatically delete over 30 days backup datasets to reduce storage cost.

o Generate and distribute Monthly Database Uptime by Percentage.

o Maintain and modify databases backup scripts and codes in which using Oracle Recovery and

Manager tool (RMAN) when these database’s environment changed.

• Operating System, Supporting Software and Middle Tier Maintenance and Support o Ensure that all application supporting software is patched to the latest version of release by the vendor as supported by the application within established patching baselines.

o Ensure testing of software patches in all environments to include testing of compatibility and operability between all application components.

o Support middle tier patching on a quarterly schedule and perform manual OS patching as needed if the patching process fails or for OS components not patched through SCCM.

o Configure the middle tier in accordance with vendor provided documentation.

o Configure the operating system in accordance with vendor provided documentation.

o Ensure that all managed software is documented within confluence.

o Ensure that all managed software configurations are documented or linked in the system design document and that SOPs are established for all software administration.

o Support software break fix investigations and solutions.

o Provide testing of operating system configuration changes, middle tier updates, and other application supporting software.

o Monitor servers to ensure proper management and cleanup of the user sessions in Forms and

Reports/APEX components o Monitor report runtimes to address/avoid long running reports and cancelling reports as needed.

o Tune, optimize and monitor the reporting solution to ensure streamlined report execution.

o Perform changes to the Server software platform to accommodate possible hardware/operating system infrastructure changes or upgrades o Research and document key steps related to OS and/or DBMS upgrades that may be required to ensure continued Contractor support of ICE’s financial system of record infrastructure o Provide supporting software configuration guidance for improved stability of the system environment.

o Provide guidance and technical expertise for implementation of automation or artificial technologies.

o Configuring, optimizing, testing and tuning of single sign-on components supporting applications.

o Support compliance implementation of Operating Systems hosting the application middle tier and databases, including submission of requests to appropriate support teams.

o Manage patching and upgrades of the Operating System supporting middle tier components.

o Support patching and upgrades of the Operating System supporting the FFMS databases.

o Support for COS monitoring including checking that it is running daily, restarting COS monitoring and supporting updates, upgrades, and fixes.

• Network, Interface and Information Sharing Maintenance and Support o Track and manage and application and interface certificates; submit requests for new reference and authentication codes to the ITPM or coordinate with external partners to pull down external certificates.

o Manage, generate, and install certificates on the system ALB or coordinate with network engineering for any certificates that need installed at the reverse proxy.

o Analyze network changes, submit requests, and support network configuration.

o Manage and execute interface job scheduling and manual runs with partners.

o Update inface connection methodologies as needed.

o Coordinate testing of interfaces with external trade partners.

o Monitor the ALB to ensure efficient session management and application response time and coordination of any required optimization and/or changes o Monitor, maintain and manage all existing and future interfaces within the system architecture.

o Provide network configuration guidance for improved stability of the system interfaces.

o Serve as subject matter expert (SME) in providing in-depth knowledge of the network, system load balancing and tuning.

o Support Cadi/Bear and non-FFMS data retrieval as needed including but not limited to the following:

• PICS, PERSECS or data files that are requested in support of any audit or security related activities.

• Run the CADI/BEAR Java utilities weekly or as requested, and provide report output to stakeholders o Managing the interface schedule and manually running the travel interface as needed.

o Support implementation of new interfaces as well as upgrades and changes to existing interfaces as needed.

o Support for data subscriber connections to the reporting database environments, including setting up subscriber accounts and views for customers.

o Support for existing current interfaces for FFMS and other applications with business financial system data needs including but not limited to: Bonds Management Information

System (BMIS), Enterprise Travel Services (ETS) Concur ETS-2, Concur FundsCheck, FileOnQ, mLinqs, General Services Administration (GSA) System for Award Management (SAM), Department of Agriculture (USDA) NFC, Treasury IPP, Treasury Payment, Treasury DonNotPay, Treasury G-Invoicing

• Administration, Operations, and Management Support o In the event the system performs infrastructure, platform or application upgrade or an enterprise financial system modernization in part or entirely, the Contractor shall provide analysis, recommendations and technical implementation support that includes design analysis, interface transition support, data migration support and technical tasks project planning support. In addition, the Contractor shall update and transition relevant software system documentation to include architecture infrastructure diagrams/documents.

o Ensuring documentation for all vendor performed processes is up to date and available for artifact reviews.

o Analyzing system logs for potential problems and ensuring that logs are available in the enterprise log management tool SPLUNK or equivalent.

o Supporting data archiving and migration efforts, including data refreshes between environments.

o Ensuring failover capabilities to meet maximum tolerable down-times are available.

o Support password changes for interface, schema/service, and enterprise accounts that are managed by the application.

o Perform desktop compatibility and security testing for new Java and/or Adobe patches/upgrades to ensure compatibility within the FFMS environments on DHS ICE imaged workstations.

o Manage privilege user account administration.

o Establish and maintain automation for secure data transfer through ETLs.

o Monitor for application and infrastructure functionality and performance alerts o Maintain tools associated user management such as Single Sign On components and the

Password Management Utility (PMU).

o Attend collaboration meetings in person or via telecom such as scrums, integrated product team meetings, ad-hoc meetings, etc.

o Provide appropriate system documentation with monthly reviews as necessary to include system Standard Operating Procedures (SOP), Topology diagrams and process workflows as well as other similar documentation.

• Audit, Security, and Incident Management Support o Provide Tier-III support and troubleshooting for ICE service tickets for resolution related to the infrastructure and supporting applications. Tickets shall be managed through the enterprise ticketing system ServiceNow or equivalent.

o Support Software Maintenance Tier 3 Support trouble tickets in accordance with the service level agreements within the enterprise ticket management tool ServiceNow or equivalent.

o Ensure that incident ticket assignment and response is less than one business day.

o Investigate incidents and perform problem resolution, providing the government with root cause analysis (BLUF-Bottom Line Up Front) summaries related to the supported system components.

o Maintain and participate in the Federal Information Security Management Act (FISMA) activities.

o Ensure confidentiality, integrity and availability (CIA) of the system where possible to include, but not limited to, Authority to Operate (ATO) Support, Privacy support, etc.

o Ensure system adheres to DHS financial auditing standards o Performing security action items and mitigation strategies as directed by ITPM o Analyzing and remediating security findings and Plans of Actions and Milestones (POA&M) items o Producing artifacts to support remediation activities, security assessments and audit requests.

o Tracking, developing, acquiring and providing security requests for artifacts.

o Coordinating security vulnerability remediation among technical teams or with support from the ITPM to coordinate with the customer.

o Provide SME for all functions performed by the contractor for audit assessments.

o Review and submit remediation guidance on Web, Host, Database, and Container level scans.

o Perform annual failover testing or equivalent security compliant failover testing.

• Reporting Database Environment Management and Support o Provide technical support for the standalone database replication environment.

o Provide support for the payroll processing interface and the associated database, such as but not limited to supporting changes to data connectivity, data structure, format changes, file type changes, data reconciliation, report generation and data ingest requirements.

o Maintain all databases within the database container ton include but not limited to: LIDS, PERL. BTS, CIS, IAI, ICE, OHA, OPS, SAT, DBRD, LIDS, JSP, and PERL).

o Establish materialized views are used to replicate the data from production to the stand-alone reporting database environment.

o Ensure the reporting instance will comply with security, availability and performance requirements.

o Support customer requests for management of the sync times between the transaction and reporting database.

o The FFMS reporting database environment shall allow users to generate standard reports, ad-hoc reports, scheduled batch aggregation jobs, yearly summary reports, perform extraction, transform, and load (ETL) extractions. The instances shall generate aggregation jobs such as Open Document File (ODF) and Treasury Information Executive Repository (TIER).

Reporting instances at a minimum shall refresh every hour, but not adversely impact the performance of the production system.

o Support update and patching of the database hosting server.

o Support the creation and management of database instances.

o Manage database operations

• Start/stop databases

• Storage management (logical/physical/logs)

• Manage schema objects

• Manager users (profile/privileges/user creation/drop)

• Support application deployments (releases)

• Perform data refresh as needed

• Oracle database wallet management

• Manage Oracle connectivity

• Manage database issues o Manage monitoring and performance

• Manage Oracle Enterprise Manager (OEM)

• Monitor database performance

• Configure database alerts

• Monitor database alerts

• Perform database tuning

• Perform SQL/Query tuning o Manage Database Backup and Recovery

• Perform database backups as needed

• Perform database recovery as needed

• Create flashback restore points as needed

• Manage database replication (Data Guard, Gold Gate Replication)

• Create/manage DR databases as needed o Database Security

• Implement DISA STIGS for Oracle databases

• Configure SOC scans

• Configure AUDIT/SPLUNK

• Respond to audit findings and provide artifacts as requested

• Advise on database security

• Create SOP on routine procedures

• End of Year Close and Next Fiscal Year Budget Support o Provide yearly technical support for OFM’s Current Year End Close and Next Fiscal Year Budget allotment, appropriations, and apportionments from last two week on September to first two or three weeks on October based on OFM requirements.

o 24/7 end of year monitoring of query runtimes and resource utilization.

o Monitor Production network, servers and databases CPU, memory, I/O, data files, tablespace, storage, resource usage, log files, etc.

o Provide the production databases performance and activities reports in every two hours from

8:00 am to 12:00 am, 7 days a week, in four- or five-weeks’ timeframe.

o Work with stakeholders to terminate query sessions over utilizing resources and/or those running longer than 2 hours.

o Generate and distribute reports to stakeholders from OEM displaying running sessions and sessions run times.

• Interface, Connectivity, and Infrastructure Availability Dashboards O&M o The Contractor shall complete the following tasks for Interface, Connectivity, and

Infrastructure Availability Dashboards utilizing tools such as but not limited to OFM Interface Dashboard:

Ensure that all interface statuses are available on the dashboard as well as through reports.

Maintain and monitor all interface components.

Maintain monitoring of SSO components are monitored to support effective user access to supported applications.

Maintain monitoring of certificate expiration date tracking is available via the dashboard and or reports to prevent certificate related outages.

Maintain monitoring of DataMart replication on the dashboard and reports are available as needed. Support and advise on replication changes as needed.

Maintain OCIO management reports and dashboards to support the system such as but not limited to usage monitoring, performance metrics and access management.

Maintain user access management for Dashboards and Reports.

Support development and upgrade functionality such as but not limited to platform migrations, security remediation requirements, and changes to connectivity and network requirements.

Support for migration to alternate enterprise solutions as applicable

2.4 Office of Financial Management (OFM) General Support Systems (GSS) Support – CLIN 0004 The contractor shall be responsible for the day-to-day maintenance of the OFM GSS Infrastructure. The Contractor shall maintain software support and related resources on the infrastructure, such as servers, VMs, interfaces and their connectivity, scripting, load balancing services, and database suite of products and tools. Support shall be provided as defined in the following categories:

• Operating System, Database and Application Support o Ensure that all application supporting software is patched to the latest version of release by the vendor as supported by the application within established patching baselines.

o Ensure testing of software patches in all environments to include testing of compatibility and operability between all components.

o Ensure that all managed software is documented within confluence.

o Ensure that all managed software configurations are documented or linked in the system design document and that SOPs are established for all software administration.

o Provide testing of operating system configuration changes, middle tier updates, and other application supporting software.

o Configure the operating system in accordance with vendor provided documentation.

o Perform changes to the Server software platform to accommodate possible hardware/operating system infrastructure changes or upgrades o Provide guidance and technical expertise for implementation of automation or artificial technologies.

o Ensure SSO compatibility with enterprise tools.

o Support compliance implementation of Operating Systems hosting the application middle tier and databases, including submission of requests to appropriate support teams.

o Support patching and upgrades of the Operating System and databases.

o Support upgrade, releases and enhancements to the application.

• Administration, Operations, and Management Support o Analyzing system logs for potential problems and ensuring that logs are available in the enterprise log management tool SPLUNK or equivalent.

o Support password changes for interface, schema/service, and enterprise accounts that are managed by the application.

o Manage privilege user account administration.

o Attend collaboration meetings in person or via telecom such as scrums, integrated product team meetings, ad-hoc meetings, etc.

o Provide appropriate system documentation with monthly reviews as necessary to include system Standard Operating Procedures (SOP), Topology diagrams and process workflows as well as other similar documentation.

o Review and assess the validity of the user stories and new/added business requirements o Integrate business requirements supporting agency financial reporting as applicable within

OFM GSS.

• Audit, Security, and Incident Management Support o Investigate incidents and perform problem resolution, providing the government with root cause analysis (BLUF-Bottom Line Up Front) summaries related to the supported system components.

o Maintain and participate in the Federal Information Security Management Act (FISMA) activities.

o Ensure confidentiality, integrity and availability (CIA) of the system where possible to include, but not limited to, Authority to Operate (ATO) Support, Privacy support, etc.

o Performing security action items and mitigation strategies as directed by ITPM o Analyzing and remediating security findings and Plans of Actions and Milestones (POA&M) items o Producing artifacts to support remediation activities, security assessments and audit requests.

o Coordinating security vulnerability remediation among technical teams or with support from the ITPM to coordinate with the customer.

o Ensure 508 compliance and testing.

o Review and submit remediation guidance on Web, Host, Database, and Container level scans.

o Perform annual failover testing.

Optional Tasks:

2.5 User & Interface Password Management Support– CLIN 0005A (Optional) The contractor shall be responsible for the day-to-day maintenance of the Password Management Utility.

The Contractor shall maintain software support and related resources on the infrastructure, such as servers, VMs, interfaces and their connectivity, scripting, load balancing services, and database suite of products and tools.

• The contractor shall complete the following tasks for Privileged User Password Management utilizing tools such as but not limited to Password Management Utility (PMU):

o Apply all patches and vendor updates.

o Install/configure/test all new software versions.

o Manage connectivity to application components.

o Manage the supporting infrastructure and other related support components.

o Provide ongoing operations and maintenance support o Manage user access and provide user support for access.

o Support new development and upgrades such as but not limited to platform migrations, security remediation requirements, virtualization/containerization, modernization, and changes to connectivity and network requirements.

o Support for migration to alternate enterprise solutions as applicable.

o Implement automation of password delivery of Password Management Utility with automatic updates of Non-Person Entities (NPEs).

o Improve upon existing services by developing web services for secure delivery of passwords.

o Integrate with account and identity management tools utilized for single sign on.

o Modernize the infrastructure on which the PMU tool resides.

o Work shall be done in accordance with activities as described in Infrastructure Operations &

Maintenance o Automate the provisioning and secure distribution of generated compliant passwords to account owners and perform automatic updates of interface and schema accounts that are not managed by individuals.

o Maintain and manage Password Management Utility (PMU) application and documentation.

o Maintain and manage internal and external interface, schema accounts and passwords.

o Work with stakeholders to ensure these accounts passwords had been reset before accounts expired or to be locked to avoid interface accounts access disconnections, transaction data loss, and/or other general business process disrupted impact.

o Decommission and transition functionality once no longer required to support requirements within this contract.

2.6 Business Intelligence Reports and Dashboards Support– CLIN 0005B (Optional) The contractor shall be responsible for the day-to-day functionality maintenance of the business intelligence (BI) reports and dashboards. The contractor shall maintain software support and related resources on the infrastructure, such as servers, VMs, interfaces and their connectivity, scripting, load balancing services, and database suite of products and tools.

• The Contractor shall complete the following tasks for Business Intelligence Reports and Dashboards utilizing tools such as but not limited to Jaspersoft Reports Server/IO and Jaspersoft Studio/Web Studio:

o Apply all patches and vendor updates.

o Install/configure/test all new software versions.

o Install/configure/test additional licensed components and new software as required.

o Manage connectivity to application components.

o Manage the supporting infrastructure and other related support components.

o Provide ongoing operations and maintenance support o Manage user access support related to Jaspersoft.

o Support new development and upgrades such as but not limited to platform migrations, security remediation requirements, virtualization/containerization, modernization, and changes to connectivity and network requirements.

o Support for migration to alternate enterprise solutions as applicable.

o Provide training to the government power users to ensure operational adoption and training of the regular users.

o The contractor is to schedule power user training as needed by the government and provide regularly scheduled power user trainings at least quarterly.

o Provide advanced training to include report development support so the government can create their own regular and ad-hoc reports.

o Provide ongoing maintenance and support for business intelligence functions as needed to support agency financial reporting needs.

o Integrate business requirements supporting agency financial reporting as applicable.

2.7 Integrated Data Services Support - CLIN 0005C (Optional)

The contractor shall be responsible for the day-to-day maintenance of the integrated data services for data sharing among the agencies. The Contractor shall maintain software support and related resources on the infrastructure, such as servers, VMs, interfaces and their connectivity, scripting, load balancing services, and database suite of products and tools.

• The Contractor shall complete the following tasks for Integrated Data Services utilizing tools such as but not limited to Apache Airflow and reporting solutions such as but not limited to Jasper or Tableau:

o Apply all patches and vendor updates.

o Install/configure/test all new software versions.

o Manage connectivity to application components.

o Maintain system and data feed availability within the established SLA times o Manage the supporting infrastructure and other related support components.

o Provide ongoing operations and maintenance support o Manage user access and provide user support for access.

o Support new development and upgrades such as but not limited to platform migrations, security remediation requirements, virtualization/containerization, modernization, and changes to connectivity and network requirements.

o Support for migration to alternate enterprise solutions as applicable.

o Ensure reconcilable, frequently refreshed/sync, and BI optimized dataset and views to expand the ad-hoc reporting capability o Provide training to the government users to ensure operational adoption.

o Provide advanced training and report development support so the government can create their own regular and ad-hoc reports.

o Continue the O&M support to accommodate changes in the source database without interruption to the existing data sets.

o Provide ongoing monitoring of operations and ad-hoc reporting of operational datasets and data flows.

o Support the operations of APIs and make sure that customers have the ability to build and manage their APIs.

o Decommission and transition functionality once no longer required to support requirements within this contract.

2.8 Reports, Dashboards and Application Programming Interface (API) Development– CLIN 0005D (Optional) The contractor shall develop data structures and build workflows to support agency reports and dashboards. This shall include the establishment of APIs to support agency data sharing needs and the establishment of required Directed Acyclic Graph (DAGs) utilizing tools such but not limited to Apache Airflow.

• The Contractor shall complete the following tasks but not limited to:

o Communication with the business owner and stakeholder to establish requirements.

o Coordinate with security for approvals of new interfaces.

o Document required security interconnection documentation such as but not limited to ISAs and ICAs.

o Perform analysis of the required data tables, schemas and directories based on business requirements.

o Create and configure DAGS for establishment of data workflows.

o Create visualizations, dashboards and reports to support business needs.

o Expansion of existing capabilities as needed.

2.9 Share Drive Support – CLIN 0005E (Optional)

The contractor shall monitor and maintain customer created VBA applications, access databases, and SQL queries pertaining to supporting agency critical financial operations as stored in the share drive.

• The contractor shall complete the following tasks but not limited to:

o Support of the legacy mini apps and utilities and other collections and libraries of MS

Access, MS Excel, VBA and other utilities, which are reliant on direct connections to FFMS DBs via GFE setup JBDC/ODBC connections.

o Support of Static Files that are used by various FSC users and shared for editing.

o Support for scripts and semi-automation scripts and the various batch scripts and MS

Office spreadsheets/docs that ingest and/or correlate the data from Static files.

o Harden the utilities or re-factor as needed.

o Deletion of unrequired files.

o Manage user access to directories.

o Support backup and restoration of files from backup.

2.10 Surge Support – CLIN 0005F (Optional)

• The Surge Support task area will be used for new tasks currently unidentified in the SOW regarding, but not limited to, the following task areas: Development and Maintenance of new interfaces, Modernization Activities, Reports Development, etc.

2.11 Other Direct Costs (ODCs) – CLIN 0006

• If ODCs arise, the Contractor is not authorized to incur any ODCs in connection with this Task Order, without the prior written approval of the ITPM and COR. Any ODCs incurred without prior written approval from the ITPM and COR will not be reimbursed by the government.

ODCs shall include, but are not limited to, software and developer licenses and analytic and testing tools. Reimbursement for local travel under this Task Order is not authorized.

3.0 CONTRACTOR PERSONNEL

The Contractor shall provide staff with considerable knowledge and demonstrated experience in supporting FFMS, its database technologies, and various network configurations applicable to the application, including the current configuration in use at ICE. The day-to-day supervision and direct control over the work performed by Contractor personnel will be the sole responsibility of the Contractor

Prior to adding a new employee, the Contractor shall submit the following required information to the OCIO COR for approval: the employee’s security application and proposed labor category. When the Office of Professional Responsibility (OPR) provides a favorable suitability determination of the employee’s security application, the COR will notify the IT Project Manager and Contractor. The employee may not begin any work on the contract until he/she receives a favorable suitability determination from OPR and approval from the COR.

3.1 Qualified Personnel

The Contractor shall provide qualified personnel to perform all requirements specified in this SOW.

3.2 Continuity of Support

The Contractor shall ensure that the contractually required level of support for this requirement is maintained at all times. The Contractor shall ensure that all contract support personnel are present for all hours of the workday. If for any reason the Contractor staffing levels are not maintained due to vacation, leave, appointments, etc., and replacement personnel will not be provided, the Contractor shall provide e-mail notification to the Contracting Officer’s Representative (COR) prior to employee absence.

Otherwise, the Contractor shall provide a fully qualified replacement.

3.3 Key Personnel

Before replacing any individual designated as Key by the Government, the Contractor shall notify the Contracting Officer no less than 15 business days in advance, submit written justification for replacement, and provide the name and qualifications of any proposed substitute(s). All proposed substitutes shall possess qualifications equal to or superior to those of the Key person being replaced, unless otherwise approved by the Contracting Officer. The Contractor shall not replace Key Contractor personnel without approval from the Contracting Officer. The following Contractor personnel are designated as Key for this requirement. Note: The Government may designate additional Contractor personnel as Key at the time of award.

For this Task Order, there are three (3) Key Personnel required, one (1) the Project Manager, Level II, one

(1) System Engineer, Level II and one (1) Systems Architect, Level II.

The Project Manager shall have the following qualifications:

• Four (4) years of experience in IT related projects, financial management, or business process reengineering directly related to the work to be performed.

• Working knowledge of applicable federal, state, and local laws, regulations, and guidance.

• Demonstrated leadership experience with:

o Critical soft skills o Active listening and verbal communication skills o Strong written and verbal communication o Flexible and adaptable attitude o Can conform to shifting priorities, demands, and timelines o Ability to elicit cooperation from a wide variety of sources, including other divisions

• Strong computer skills in MS Suite, GitHub, Jira, Confluence and Windows/Linux servers

• Ability to discuss technical issues with non-technical, executive-level government officials

Preferred Skills:

• Familiarity with financial system technology platforms

• Knowledge of ICE OCIO IT policies and procedures and business processes

• Knowledge of DHS/ICE technology and architecture

• Experience within DHS and/or its components

The System Engineer shall have the following qualifications:

• Five (5+) years of experience using key technologies:

o Load balancer devices o Lambda functions o Containerization o Oracle Databases o Oracle Middleware o Java/J2E o Microsoft Office 365

• Five (5) years’ experience in development and operations of cloud and other virtualized environments.

• Technically competent with MS Suite.

• Experience on 1 or 2 similar efforts within the last three (3) years or still in progress expertise in load balancer devices, Windows application servers, JAVA, and Oracle middleware to handle business process orchestration and automation/deployment tools such as Ansible, Terraform, Airflow, and Liquibase.

• Technical ability to discuss technical issues with non-technical, executive-level government officials.

• Experience on 1 or 2 similar efforts within the last three (3) years or still in progress expertise in supporting financial system platforms. This support would include their experience supporting financial year-end processes, which may require close monitoring of systems during periods of intense workload, such as financial year-end processing.

• Familiarity with financial system technology platforms.

• Experience within DHS and or its components and within the law enforcement context.

• Experience in deploying and maintaining systems or services in a Windows and Linux environments.

• Experience in developing and deploying business intelligence reporting solutions.

• Expert in AWS cloud service.

• Expert in Oracle suite administration.

The Systems Architect II shall have the following qualifications:

• Five (5+) years of experience using key technologies:

o Load balancer devices o Lambda functions o Containerization o Oracle Databases o Oracle Middleware o Java/J2E o Microsoft Office 365

• Five (5) years’ experience in conducting and identifying development operations of cloud and other virtualized environments.

• Technically competent with MS Suite.

• Experience on 2 or 3 similar efforts within the last three (3) years in evaluating technical trends and providing recommendations for technology and architecture to meet business objectives.

• Experience on 2 or 3 similar efforts within the last three (3) years performing research on emerging technologies to support proof-of-concepts (POC) capabilities and assisting in implementing the solutions.

• Experience on 1 or 2 similar efforts or expertise in load balancer devices, Windows application servers, JAVA, and Oracle middleware to handle business process orchestration and automation/deployment tools such as Ansible, Terraform, Airflow, and Liquibase.

• Technical ability to discuss technical issues with non-technical, executive-level government officials.

• Experience on 1 or 2 similar efforts within the last three (3) years or still in progress expertise in supporting financial system platforms. This support would include their experience supporting financial year-end processes, which may require close monitoring of systems during periods of intense workload, such as financial year-end processing.

• Familiarity with financial system technology platforms

• Experience within DHS and or its components and within the law enforcement context.

• Experience in deploying and maintaining systems or services in a Windows and Linux environments.

• Experience in developing and deploying business intelligence reporting solutions

• Expert in AWS cloud service

• Expert in Oracle suite administration

3.4 Project Manager

The Contractor shall provide a Project Manager who shall be responsible for all Contractor work performed under this SOW. The Project Manager shall be a single point of contact for the Contracting Officer and the COR. The name of the Project Manager, and the name(s) of any alternate(s) who shall act for the Contractor in the absence of the Project Manager, shall be provided to the Government as part of the Contractor's proposal. The Project Manager is further designated as Key by the Government. During any absence of the Project Manager, only one alternate shall have full authority to act for the Contractor on all matters relating to work performed under this contract.