DRAFT_DREAM PWS_v2 Jan 2020.docx

U.S. Department of State Section C- Performance Work Statement Bureau of Consular Affairs Data Replication Engineering and Management (DREAM) U.S. Department of State Bureau of Consular Affairs Office of Consular Systems and Technology

Data Replication Engineering and Management (DREAM) Performance Work Statement (PWS)

Purpose The purpose of this Data Replication Engineering and Management (DREAM) contract is to acquire engineering and management support capabilities that provide database operations management, database engineering and architecture, enterprise service bus, data sharing operations management, Tier III database operations support, and database applications development and integration. These services are the critical backbone to an array of activities vital to ensuring strong United States (U.S) border security and facilitating legitimate travel to the U.S. DREAM is the nexus for integrating complex information technology systems and infrastructure that serve as the foundation of achieving the Consular Affairs mission domestically and abroad.

Background The U.S. Department of State (“DOS”, “State”, or “the Department”) leads America’s foreign policy through diplomacy, advocacy, and assistance by advancing the interests of the American people, their safety and economic prosperity. DOS’ Bureau of Consular Affairs (CA) is the public face of the Department for millions of U.S. citizens and foreign nationals around the world. CA is responsible for the welfare and protection of U.S. citizens abroad, for the issuance of passports and other documentation to citizens and nationals, for the protection of U.S. border security, and for the facilitation of legitimate travel to the U.S. Responsibility for these functions is vested with the Department’s Assistant Secretary for CA and for their implementation abroad in consular officers assigned worldwide. CA is also the Department’s largest Bureau in terms of domestic personnel and is almost entirely funded through revenue generated by consular fees. In fiscal year 2017, CA adjudicated 12.4 million non-immigrant visas and issued 21 million U.S. passports contributing to $4.58 billion in revenue, making CA the equivalent of a Fortune 600 company.

To achieve this mission, CA actively pursues business, technology, and management enhancements and improvements. Within CA, the Office of Consular Systems and Technology (CST) is tasked with providing CA's mission critical information technology (IT) resources to support more than 300 Consular locations including approximately 276 locations abroad, 29 domestic passport agencies, 2 domestic visa processing centers, and an array of information, book/card print, passport centers, DOS annexes, and contractor sites.

Within CA/CST, the Service Integration & Innovation (SII) Division is responsible for crosscutting initiatives aimed at standardizing and improving CA/CST. SII identifies new technologies to support CA’s mission, ensures alignment of systems to business goals through enterprise architecture modernization, and identifies ways to improve CA/CST processes to ensure optimal delivery of services. SII’s Integrated Services (IS) Branch is the lead for CA information sharing with external agencies. IS defines solution architectures and information models, supports the transition to ConsularOne (one of CA/CST’s modernization efforts), and supports continued, sustainable capability delivery.

SII/IS is responsible for the DREAM contract.

Scope The scope for this effort includes:

· Supporting approximately 15,000 DOS and 30,000 other federal agency users located at more than 500 sites worldwide

· Building enterprise database solutions for big data, including more than 1,000 servers and 2,000 databases worldwide, that provide optimal IT data integrity and accessibility that ensure performance quality, reliability, and 24x7x365 IT systems availability that support border security and the facilitation of legitimate travel

· Supporting the consumption and provision of information services among internal bureaus and with external partner agencies - both synchronous and asynchronous request/response information exchange transactions

· Publishing/subscribing information disseminations, and in some cases, may require maintenance of transaction order and guaranteed one-and-only-one delivery

· Developing, orchestrating, and deploying open standards-based, interoperable, and transportable web services based on commercial-off-the-shelf service oriented architecture, micro services architecture, enterprise service bus (COTS SOA ESB) development and deployment software suite (Oracle SOA/ESB Suite Tool)

· Supporting a complex, 440 TB central remote database management system spanning 16 Oracle Real Application Cluster (RAC) databases, each deployed at multiple data centers

· Supporting massive data files in a system that supports millions of online transaction processing (OLTP) per day, including partitioning and archiving schemes The Contractor shall recommend approaches to the Government for meeting all contract requirements. These approaches shall maximize efficiency, streamline operations, improve results, strengthen security, and minimize operational costs.

The Contractor shall work collaboratively, professionally and proactively with other CA/CST Contractors and CA/CST entities to improve system performance and decrease incidents/tickets. The Contractor will provide database support that includes:

1. program management to include transition services

2. database operations and management

3. database engineering and architecture

4. enterprise service bus

5. data sharing operations and management

6. database application development and integration

7. security operations

3.1 Program Management

Robust program management is critical to the ability of the Contractor and CA to meet its objectives. The Contractor shall be able to effectively coordinate operations across bureaus, contracts, Consular locations, and more. Execution of this contract must be based on three guiding principles: 1) no risk to current and future continuity of services; 2) essentiality of proactive communication with appropriate stakeholders; and 3) flexibility. Active Contractor leadership to instill a contractor workforce culture that embraces these guiding principles will help ensure a practical, low risk approach to transition and will work well in partnership with CA. Contractor Program Management will be dynamic throughout the entire period of performance. The Contractor will be expected to provide frequent briefings to different stakeholders to apprise them of program objectives and plans, and projects. The Contractor must stress the coordination of the various contract activities listed herein. These fall solely on the Contractor to perform and to coordinate activities to manage unexpected situations that will arise during contract performance.

The Contractor shall provide strong Program Management services for this contract in executing this performance work statement (PWS). The Contractor shall be responsible for the effective management and administration of all efforts performed under this contract. These services include the requirements for: management plans, progress reports, deliverables, invoices, staffing, security, property management and control of Government Furnished Equipment (GFE), and other specific topics.

The Contractor shall be responsible for ensuring that all work activities are performed in a timely, efficient, and cost effective manner while maintaining the highest quality of performance. The Contractor shall institute and maintain an effective, efficient, and responsive management organization that shall provide administrative support for all tasks under this contract. The Contractor shall, as directed, communicate and coordinate with identified stakeholder groups throughout the project lifecycle. Additionally, the Contractor shall oversee all contracted personnel and subcontracted resources used in the performance of this contract.

The Contractor shall be responsible for program control and management for this effort. In this role, the Contractor will execute activities essential to a successful contract performance. These activities include but are not limited to the following:

A. Resource Management

a. Manage the Contractor’s personnel and staffing (including development, maintenance of staff clearance and evaluation)

b. Maintain and meet all training and certification requirements set by the Government which include, but are not limited to DOS, DS, CA, and CST

c. Allocate and schedule the Contractor’s resources

d. Provide a technically proficient and professionally capable staff that is established and maintained throughout the life of the contract

e. Minimize personnel turnover and motivate individuals to achieve excellent performance

f. Provide seamless transition of Contractor personnel without interruption

g. Identify all resources used for performance of work under the contract and clearly define their roles

h. Identify all subcontractor(s) used for performance of work under the contract and effectively manage their work

i. On-Boarding

· Visitor Access Request (VARs) forms shall be completed and submitted to Diplomatic Security (DS) for verification of security clearance (required before issuance of badge and invoicing)

· Ensure training requirements for contract team members are completed (e.g., annual security training, and as directed/approved by the COR/GTM)

· Badges are required for OpenNet access

· CA account access/delete requests B. Cost Management

· Ensure all tasks, projects, deliverables stay within the contract budget

· Report budget overrun or over burn in a timely manner

· Manage travel and other direct costs closely to prevent over burn C. Quality Management & Quality Assurance

a. Identify and implement process improvements on a continuous basis in an effort to improve both the timeliness and quality of the Contractor’s work products

b. Manage quality effectively and in accordance with the Contractor’s Quality Control Plan (QCP)

c. Bring to the Government’s attention possible new approaches, ideas, innovations, process improvements, methods, or technologies to enhance/improve operations.

d. Develop and manage processes to ensure quality across all PWS tasks and subtasks D. Risk Management

a. Ensure risks that may impact the Contractor’s performance are identified, assessed, managed, and reported

b. Resolve issues while minimizing the impact on schedule, scope, and cost of affected contract task

c. Maintain risk logs E. Schedule Management

a. Establish and baseline schedules for all projects as agreed upon with the Government

b. Establish an Earned Value Management process to manage and track project schedules and costs, and ensure that schedule and cost variances remain within acceptable levels according to the agreed upon SLA

c. Deliver reliable, effective, and efficient services on time

d. Resolve problems with minimal disruption to the activities being performed under the contract F. Scope Management

a. Implement processes to manage the Contractor’s scope across the PWS including projects.

b. Support and adhere to all IT policies, guidance, instructions, and direction provided by DOS G. Communication Management

a. Coordinate and communicate with stakeholders (Government management and technical resources, contractors, etc.)

b. Report on contract performance and task/project/program status using government-provided standard templates – and comply with updates to templates throughout the life of the contract

c. Work cooperatively with other Contractors, stakeholders, Government personnel and management

d. Document lessons learned that may enhance or streamline future program activities H. Business Relationships and Presentations shall include:

a. Monthly presentation of the Contract Status Report (CSR) as directed by the Government

b. Demos of systems, releases, or patches to business units, stakeholders as directed by the COR/GTM Specific support required also includes the following: kick-off meeting participation, program management plan development and submission, weekly activity reporting, monthly contract status reporting, standard operating procedures support, and subcontracting performance report. These are described below.

3.1.1 Kick-off Meeting

The Contractor shall attend a Post-Award Orientation/“Kick-Off” meeting within ten (10) business days of contract award. The Contractor shall be prepared to review the draft Transition-In Plan (TiP) and outline its transition-in staffing, schedule, activities, milestones, deliverables, and communications plan. The kick-off meeting shall include, but is not limited to the following:

A. Expectations B. Roles and Responsibilities C. Review of CA’s Immediate Needs D. Review of Schedule Activities and Milestones E. Risks F. Resources G. Communication H. Transition Tasks and Requirements I. Status on Obtaining Personnel Security Requirements J. OpenNet requirements K. Onboarding requirements L. Status on Implementing the Transition Plan

3.1.2 Program Management Plan (PMP)

The Contractor shall develop and maintain a comprehensive PMP describing the technical approach, organizational resources, and management controls to be employed to meet the cost, performance, and schedule requirements throughout contract execution. The PMP shall provide a detailed narrative defining ongoing services to be provided in quantitative and qualitative terms while at the same time providing descriptive instructions on how to achieve/complete specific operational functions. The PMP is a living document and can be changed in response to innovative performance improvements. The PMP shall detail the roles/responsibilities of the “provider” (Contractor) and the “client” (U.S. Government), clearly defining and describing where (when applicable) one provides a service to another. The PMP should utilize a combination of text and diagrams to clearly illustrate processes. The PMP will be submitted for Government review and comment, followed by a final draft which is subject to approval by the COR.

A joint Government and Contractor annual review of the PMP will be conducted 60 to 90 days prior to exercising any contract option period to determine if the plan remains valid or requires update. The Contractor shall be responsible for coordinating the meeting with the COR.

The Contractor shall base the PMP on Service Strategy and Portfolio Management (SSPM) Program Control Processes/Requirements such as the Software Development Life Cycle (SDLC) and Scaled Agile and Project Management guidelines. These requirements have been developed to aid CA/CST management to oversee all current and future programs and projects that support CA/CST’s Enterprise Architecture (See attachments C-E).

At a minimum, the PMP shall address the following:

A. Program Goals B. Program Scope C. Management Strategy D. Assumptions and Dependencies E. Stakeholders F. Security G. Deliverables H. Schedule I. Work Breakdown Structure (WBS) J. Risk and Issue Management K. Contractor’s Quality Control Plan (QCP) L. Configuration Management Plan M. Communications Management Plan N. Program Activities Management Plan O. Program Organization Plan (includes Staffing Plan) P. Security Operations Plan Q. Performance Metrics/SLAs R. Compliance with all aspects of Certification and Accreditation (C&A) and Authority to Operate requirements S. Management of Government Property T. Capacity Management Plan

3.1.3 Weekly Activity Reports (WARs)

The Contractor shall submit WARs delivered using a template to be agreed upon with the COR. The WAR must include both performance and transition activities, capturing activities completed the week prior. These reports shall include, but not be limited to, the following information:

a. Contract number

b. Contractor name and address

c. Date of report

d. Period covered by report

e. Transition-In Activities

f. Include all efforts applied to each PWS task and subtask in detail, for example:

a. Description of progress made during period reported, including problem areas encountered and recommendations, if any, for solutions

b. Plans and recommendations for activities during the following reporting period

c. Schedule, Risk, Issues, Scope, and Cost Status (score criteria provided in template)

d. Project Summary and Scope

e. Key Accomplishments

f. High Priority/Severity Issues

g. Project Completion and Calculated Schedule Variance Percentage

h. Standard Integrated Service Lifecycle (ISL)/SDLC Milestones (planned/actual dates and status)

i. Major Project Milestones

j. Key Dependencies

k. Decision points for the Government

l. Advanced notification of timeline slippage

m. PWS specific data as indicated in PWS Section 3.2 and thereafter

3.1.4 Monthly Contract Status Report (CSR)

The Contractor shall prepare and submit the CSR using the SSPM provided CSR template. Monthly, the Contractor shall review the CSR with the COR and GTM(s), and quarterly, the Contractor shall present the CSR in person at a scheduled meeting to which CA/CST leadership may attend (See attachment B).

3.1.5 Database Statistics Report

Monthly, the Contractor shall submit a Database (DB) Statistics Report delivered using a template to be agreed upon with the COR. The DB Statistics Report must include database monthly data for the past twelve months, cumulative totals for data in the database(s) for the reported month, web requests by location for the reported month, clearance statistics for the reported month, external organizations users and web requests for the reported month, report statistics for the reported month, and other information as agreed upon with the COR. These reports shall include, but not be limited to, the following information from all databases supported (e.g., CCD, PPT, FR, etc.):

a. Contract number

b. Contractor name and address

c. Date of report

d. Period covered by report

e. Metrics and Statistics

3.1.6 Trend Analysis

Monthly, the Contractor shall submit a Trend Analysis Report delivered using a template to be agreed upon with the COR. These reports shall include, but not be limited to, the following information from all databases supported (e.g., CCD, PPT, FR, etc.):

a. Documentation and reporting is required to support the Integrated Services branch and Incident Management, including: statistical trend analysis of ticket reports to show distribution and frequency of recurring incidents and proactively suggest ways to correct the root cause of the problem to reduce the number of service calls.

b. Problem Management – Perform problem management and provide root cause analysis for one or more incidents. Contractor shall create a Problem Management Investigate (PBI) in response to one or more incident, per Government or situational approval. The Contractor shall perform trend analysis for reoccurring system events, service interruptions, outages, etc. in order to manage potential problems proactively as well as reactively.

c. Provide trend analysis and reporting across multiple data sources.

d. Define, develop, and implement tools that allow for effective capacity monitoring, trending of IT infrastructure, applications, and IT components. Capture trending information and forecast future CA/CST IT capacity requirements based on CA/CST IT defined thresholds. Use the information from Capacity planning to define and establish thresholds. Ensure data feeds into monitoring systems.

3.1.7 Project Management

The Contractor shall provide project schedules to be integrated with CA/CST’s Integrated Master Schedule (IMS). IMS aggregates individual schedule inputs from multiple projects to aid sequencing project tasks in an optimal order to meet CA/CST’s overall priorities. The IMS is a vehicle for analysis and management of dependencies between projects, resource utilization, identification of resource constraints, and prioritization of activities across the enterprise.

Note: Management of the CA/CST IMS is not within the scope of this contract. However, submission of project status into the IMS is a requirement and within the scope of this PWS.

Each project schedule will be maintained and updated to reflect progress after each COR/GTM status update, review, or milestone change.

CA/CST’s SDLC process and approach for IT project management and execution incorporates SAFe Agile practices. CA has adopted a SAFe Agile approach for Legacy operations, maintenance, and engineering projects, and expects operations of CA/CST services to flow from this approach. The Contractor shall adhere to the CA/CST practices and guidelines that facilitate this approach.

3.1.8 Standard Operating Procedures (SOPs)

The Contractor shall develop and revise SOPs based on best practices, lessons learned, and/or new innovations to include:

A. Develop new SOPs (See attachment F) B. Review and update existing SOPs at a minimum annually or as directed by the Government.

C. Develop clear documentation from requirements gathering, to design, to final implementation state D. Maintain and manage the SOP Library in Government-provided and owned repositories (e.g., on OpenNet or on ClassNet) E. Maintained in version control F. Demonstrate continued, routine use of SOPs in performing work G. Ensure operational procedures are documented and approved, and user manuals are complete H. Document unit test plans and results

3.1.9 Transition

Transition encompass both transition-in and transition-out activities. The Contractor shall support full transition of phase in/out services in accordance with the approved Transition Plan and FAR 52.237-3, "Continuity of Services." The 120-day transition period applies to both the incoming and outgoing Contractors.

3.1.9.1 Transition-In

The Contractor shall perform transition-in activities within 120 calendar days (i.e., transition-in period), culminating in its ability to successfully execute all tasks outlined in this PWS. Immediately upon contract award, the Contractor shall work with the incumbent Contractor to transition support, including:

a. Managing and coordinating the transition with the incumbent Contractor.

b. Obtaining documentation associated with the existing system and coordinating with the incumbent Contractor’s system development team to take ownership of system databases and database operations during transition and in accordance with the TiP.

c. Transitioning database management, data security, data reporting, and data system integration management.

d. Information transfer of knowledge of the system, processes, and procedures.

e. Assessing risk, including impacts on affected personnel, systems, and processes.

f. Establishing and adhering to transition-in schedule and milestones.

g. Obtaining property necessary for incoming individuals, such as DOS building passes and office keys.

h. Obtaining all Government assets to be provided, including equipment, software, documentation, databases, and repositories.

i. Obtaining DSS facility clearance as described in Section H and obtaining DOS OpenNet connection.

j. Planning the transition of all systems and functions in accordance with the PWS.

k. Coordinating with the Government and incumbent Contractor for of each system and agree on a final date that the new Contractor will assume all responsibility of each system.

l. Taking source code and executables from the CST/CC repository for each system/application. Verification is required that the source code compilation was successful and that all documentation and IP is available and matching production. Where applicable, byte-for-byte comparisons may be required.

m. Drafting a plan to improve software quality tool scores for baseline and releases for future implementation.

The Contractor shall provide a TiP that outlines the allotted 120-calendar day transition of services, including its plan to accomplish the activities delineated in Section C.3.1.9.1.The Contractor should plan to assume and begin all work from the outgoing Contractor during the transition period and shall propose its timeline for doing so. The Contractor shall deliver the final TiP within ten (10) business days after the kick-off meeting.

In addition to the aforementioned TiP requirements, the TiP shall describe/include:

a. How the Contractor will perform and manage the transition to successfully migrate support and maintain continuity of services without loss or degradation from the incumbent Contractor and/or the Government to its proposed solution.

b. A list of all Government dependencies and assumptions for Government services to be used during the transition, technology and information transfer processes and procedures among the various organizations, transition risks and risk mitigation recommendations, and a transition schedule that includes, at a minimum, specific tasks to be performed and the resources assigned to them, task durations and dependencies, and milestones.

After the transition-in period, the Contractor shall provide a Transition Report that demonstrates all transition-in activities are completed and shall detail:

a. Transition activities in progress and completed;

b. Training and knowledge transfer;

c. Listing of systems, documentation, assets, and artifacts transitioned;

d. List of any services, activities, and assets not transitioned and risks;

e. GFE assets received from the incumbent Contractor;

f. Demonstration of Contractor’s facilities, access, and ability to support operations and maintenance of the systems;

g. Lessons learned; and

h. Recommendations/Plans for successfully completing any remaining tasks

3.1.9.2 Transition-Out

The Contractor shall perform transition-out activities, which shall commence 120 calendar days before the end of this contract’s period of performance (i.e., transition-out period). This transition may be to a Government entity, another Contractor, or to the incumbent Contractor under a new contract/task order.

The Contractor shall:

a. Assist the Government in planning and implementing a complete transition from this Contract to the new Contractor. This shall include formal coordination with Government staff and successor staff and management and hosting the incumbent Contractor at its location.

b. Cooperate and negotiate with its successor to develop a comprehensive Transition-Out Plan (ToP) for phase-out services. This ToP shall include:

1. Coordination with Government representatives

2. Review, evaluation and transition of current support services

3. Transition of historic data, documentation (including processes and other necessary artifacts) to new Contractor system

4. Take source code and executables from the CST/CC repository for each system/application. Verification is required that the source code compilation was successful and that all documentation and IP is available and matching production. Where applicable, byte-for-byte comparisons may be required.

5. Transfer of hardware warranties and software licenses (if applicable)

6. Transfer of all necessary business and/or technical documentation, including policies and procedures

7. Transfer of compiled and un-compiled source code, to include all versions, maintenance updates and patches, and checked into configuration control

8. Orientation phase and program to introduce Government personnel, programs, and users to the new Contractor's team, tools, methodologies, and business processes

9. Disposition of Contractor purchased Government owned assets, including facilities, equipment, furniture, phone lines, computer equipment, etc.

10. Transfer of Government Furnished Equipment (GFE) and Government Furnished Information (GFI), and GFE inventory management assistance

11. Applicable DOS debriefing and personnel out-processing procedures

12. Roles and responsibilities

13. Current Work Breakdown Structures (WBS)

14. Current in-progress projects including: Project Management Plans, Risks, Schedules, Status of required artifacts

15. Turn-in of all Government keys, ID/access cards, and security codes

c. Provide sufficient experienced personnel during the transition-out period to ensure that the services required by this contract are maintained at the required level of proficiency while fully supporting transition, knowledge services, and providing artifacts, such as electronic and physical files, to successor Contractors

3.2 Database Operations Management

CA/CST is responsible for supporting CA’s IT efforts through the design, development, deployment, and maintenance of databases to enable consular professionals to effectively and efficiently adjudicate visa and passport applications. The Contractor shall support this endeavor by maintaining modernized and legacy database technologies, replicating all data collected from embassies, consulates (posts), and passport facilities, ensuring all data is available centrally (in our central database—CCD) to the appropriate users, and resolve replication errors.

Desired Outcomes:

· Data is readily available for the purpose of adjudicating visas and passports.

· Data is replicated in near to real time to post and central databases.

· Databases are running at DOS/CA/CST optimal standards.

· Data is accurate, accounted for, and not corrupted.

A. Engineering, maintenance, and support of the Consular Consolidated Database (CCD) - one of the largest data warehouses in the world at a size of 440 Terabytes (TB), post databases, and other databases such as passport databases (MS-SQL). It is also one of the world’s largest distributed database replication environments. Replication occurs concurrently across 300+ databases 24x7x365 globally. CCD uses 16 databases, spanning 50 instances across three datacenters. CCD holds current and archived data from all CA databases. It contains more than 220 million visa cases, 340 million passport applications, and 413 million photographs, utilizing billions of rows of data, and has a growth rate of approximately five (5) TB a month. Management of CCD across several environments includes, but is not limited to:

· Production Environment (OpenNet)

· Demilitarized Zone (DMZ) B. Support for other CA/CST software development environments (Sandbox, Integrated Development Environment, and Integrated Testing Environment)

a. Database software tools, database support, and connectivity with other applications.

b. Review integrated security patches, software and hardware upgrades (i.e. converged/engineered solutions) in each development environment; perform engineering/unit testing before deployment.

C. Database production support for multiple systems and environments as follows:

a. Manage interface with other systems, such as Identix, Facial Recognition, CLASS

b. Consular Affairs databases

c. Inter-database network communication D. Database replication services:

a. Set up replication

b. Create and modify CCD replication scripts as required by the different development teams using CA/CST approved replication products (currently Oracle Advanced Replication (OAR) and Oracle Golden Gate (OGG))

c. Provide database administrator (DBA) support on replication transition project

d. Create, review, and modify replication procedures

e. Resolve errors and conflicts elevated by Tier II support team E. Management of Engineered Systems and Databases (currently, Exadata)

a. Provide preventative maintenance on CA engineered database systems (Primary and Standby)

b. Build and deploy production primary and standby databases for CA/CST’s database infrastructure

c. Maintain interconnectivity between engineered systems F. Enterprise level database planning, analysis, design, development, and implementation services for CA. Performs capacity planning to process enterprise-wide data and maintain historical resource utilization and performance metrics;

G. Continuity of Operations (COOP) management capability/recoverability for CA databases, optimize system efficiencies, generate performance reports H. Implement and incorporate monitoring and reporting for all applicable databases I. Tier III-level response to alerts generated via database monitoring software implemented within the Enterprise Event Management and Monitoring capability J. Troubleshooting, diagnostic, and resolution support for databases K. Coordination and support of production troubleshooting efforts with other Tier II and Tier III CA support teams L. Optimize database performance, tuning databases operations

3.3 Database Engineering and Architecture

Desired Outcomes:

· Streamlined and modernized database infrastructure to include more efficient data processing to facilitate and support the Department’s strategic initiatives (i.e., integration and implementation planning for leveraging any new technology—currently the plan is to move to the cloud).

· Continually engineer and implement modernized best of breed database systems and services architecture.

· As part of CA’s strategy, reengineer and transition legacy databases to a modernized architecture.

· Cloud technologies are leveraged and standards for cloud deployed CA databases and systems are implemented.

· CA database redundancy, reliability, and maximum availability architectures are improved.

· Database system components are operational (no outstanding critical or open issues), secure, accurate, current, and complete.

A. Research, evaluate, and recommend future database architecture and new database platform technologies (i.e., Exadata), developing the future of CA database architecture (i.e., In-memory database architecture, high-availability systems, cloud integration, and Consular One);

B. Implement and test project architecture designs;

C. Implement database standard policies and best practices to database designs;

D. Support database migration efforts through planning and engineering (i.e., Oracle, Linux, MS SQL, etc.);

E. Evaluate, test, and support other CA/CST contractors with upgrades to current database structures and software as it relates to database systems;

F. Research, evaluate, and recommend new database technologies (i.e. NoSQL, Hadoop, Big Data, Rapid Home Provisioning (RHP), etc.);

G. Reverse engineer database issues, such as database corruptions and restoration to uncorrupted versions;

H. Research new initiatives and engineering efforts, from planning to integration and implementation (i.e. cloud technologies).

I. Engineer solutions that will allow legacy and modernized databases to coexist in a diverse ecosystem (both on premise and in the cloud).

J. Create sustainable data models/schemas/data lakes/NoSQL databases for modernized systems.

K. Continually improve data replication standards, using best practices.

L. Establish and enforce database standards on legacy and modernized databases (such as Oracle, MS SQL, NoSQL, data warehouses).

M. Design and implement data interfaces within DOS and between external agencies N. Provide database design support to internal and external users. Gather, define, and update requirements for creation of multiple databases to support CA’s need for data/reports.

a. Research standards and best practices for database management, and make recommendations on implementation

b. Develop and maintain Enterprise Data Model (i.e. Logical, Physical, Data Dictionary)

c. Sanitize personally identifiable information (PII) data and provide required production-like data to non-production environments as needed

d. Document all database activities, to include but not limited to, data models, SOPs, installation guides, etc.

e. Create ad hoc reports, as directed

3.4 Enterprise Service Bus (ESB)

Desired Outcome:

Increase organizational agility and scale by reducing development time for new initiatives using ESB. Developed and deploy services in a loosely coupled framework allowing them to communicate in a consistent and manageable way.

A. Architect, engineer, develop, and implement secure Service Oriented Architecture (SOA) and micro services, using web services standards such as Simple Object Access Protocol (SOAP), Representational State Transfer (REST), Web Services Description Language (WSDL), and other service-based technologies; Enhance reference implementations, best practices, and use cases using industry standards such as NIST standards, W3C, and OASIS web service specifications;

B. Develop standards for SOA, JavaScript Object Notation (JSON), REST and Microservices implementations;

C. Work with other CA/CST Contractor teams to create a modernized SOA infrastructure that supports high availability and disaster recovery;

D. Create automated deployments; design, develop, test, and deploy core data services, independent of vendors, products, and technologies to support continuous integration and continuous delivery;

E. Develop, enhance, and maintain CCD Based Web Services (CCDWS), Consular Systems Modernization (CSM) generic reusable services and other CA/CST initiatives for data exchange or transfer;

F. Design, implement, and maintain data interfaces for bidirectional data transfers between CCD applications and external agencies using application program interfaces (API);

G. Provide operational support of ESB infrastructure, including application server(s), Oracle Fusion Middleware environment, Oracle Service Bus, XML security gateway, performance optimization, Layer 7 or API gateways, PKI implementation, deployment support, patching, and troubleshooting;

H. Enhance, upgrade, and maintain ESB middleware infrastructure environment (i.e., Oracle Fusion Middleware, WebLogic Server, Oracle SOA Suite tools);

I. Implement and incorporate SOA monitoring and reporting for all applicable services

3.5 Data Sharing Operations Management

Desired Outcome:

Ensure CA’s data sharing environment and data is accurately and appropriately shared, in a timely manner, with CA interagency partners to allow a collective mission to maintain border security.

A. Architect, design, implement, and maintain data sharing environment. (Currently, the data sharing environment includes various web services, database links, and Oracle Advanced Queuing.)

B. Implement and incorporate monitoring and reporting for all applicable data sharing environments C. Support troubleshooting of production data sharing issues to support the vetting process and uphold national security.

D. Enhance or establish new data share feeds and liaise with network firewall team to establish secure communications with interagency partners.

E. Re-architect CA’s legacy data share environment.

3.6 Tier III Database Operations Support

Desired Outcome:

Timely, responsive, and accurate Tier III services are provided to CA/CST customers.

Note: Tier I and II services are not in scope of this contract; they are provided by the CAEIO Contractor. The largest Posts are located in China and India, therefore operating hours can begin as early as 6:00 PM east coast time. Any issues identified from these Posts can be systemic worldwide.

The Contractor’s support team shall be available on-call during non-business hours and weekends 24x7 with initial contact through the CA Service Desk via a Government-issued device.

The table below is a snapshot of Tier III tickets that CST typically deals with on a weekly basis.

Remedy Ticket Summary Example

Weekly Remedy Ticket Report: 11/08/18 – 11/14/18

Tasks: The Contractor shall provide Tier III support for CA/CST databases in accordance with performance standards.

The Contractor shall:

A. Respond to critical incidents, trouble tickets, and problem incidents (Attachment G).

B. Provide engineering solutions to fix database and database application issues that cannot be handled by Tier II database support.

C. Identifies patterns in tickets and work on short and long-term solutions.

D. Provide support to various database solutions such as Oracle, MS SQL, NoSQL, Hadoop, Cloudera, and MapR.

E. Ensures standard operating procedures, knowledgebase articles, and other relevant technical documentation is up to date, accurate, and available for Tier I and II teams for reference.

F. Troubleshoot system problems that cannot be resolved by the onsite systems management personnel or end-user support personnel, including the Tier I and Tier II Help Desks, which are provided by resources other than the DREAM Contractor.

G. Since administrator level access is not available remotely, critical and high tickets will require onsite troubleshooting at the contractor and/or government location. This applies to deployments as well.

H. Accept tickets via the CA/CST ticket system, email, or phone and open a ticket if one does not exist.

I. Use the CA/CST ticket system to manage tickets and to calculate help desk metrics. CA/CST currently uses Remedy to identify, record, and manage IT related incidents and tickets; however, the ticket management system may evolve or change with CA/CST mission priorities.

3.7 Database Application Development & Integration

Desired Outcome:

CA/CST databases are developed, integrated, and interoperate with client/server applications and web-based applications to support approximately 40,000 DOS and interagency users across multiple locations worldwide.

Note: In 2018, there was an average of approximately 32 million CCD transactions/web requests processed per month, with a total of 413,792,143 for the entire year.

Tasks:

1. Ensure database development is integrated with existing web sites, web services, and web-based applications.

1. Modify databases and database related applications for which the database applications team is responsible and in support of other application teams.

1. Maintain centrally hosted database applications.

1. Maintain database applications hosted at remote work sites.

1. Enhance and maintain legacy web-based applications that are hosted inside the database.

1. Develop and maintain documentation for which the database applications team is responsible (e.g. SOPs, install guides, interconnection security agreements, software design specifications).

1. Support modernization initiatives related to database application development.

3.8 Security Operations

Desired Outcome:

Fortify the security posture of CA’s databases by supporting CA/CST’s security architecture, while maintaining its scalability and cohesiveness, as well as its ability to adapt to new technologies and new threats. Ensure all identified Plan of Action and Milestones (POA&M) and cyber incidents are resolved quickly.

Tasks:

The Contractor is required to manage security and information assurance (IA) compliance as it applies to development, engineering, and architectural design standards of all non-production and production operating environments under this contract.

The Contractor shall ensure data security, data quality, and access controls.

A. Review application/database scripts for security violations B. Review databases for compliant security posture and violations C. Review POA&Ms, and create remediation scripts for IVV testing and deployment D. Review roles and privileges submitted through access approval, keeping with least privilege theory E. Participate and fulfill the needs of the Authority to Operate (ATO) process, including annual security assessments, boundaries, database security, etc.

F. Resolves database level security issues, such as POA&Ms and Cyber Incidents.

G. Ensures that all database system components are operational, secure, accurate, current, and in compliance with DOS technical security foundations.

H. Develop and maintain all security documentation for which the security operations team is responsible (e.g. Interconnection Security Agreements (ISA), System Security Plans (SSP), Information System Contingency Plans (ISCP), Privacy Impact Assessments (PIA), etc.).

3.8.1 Security Operations Program and Controls

The Contractor shall document the security controls using the DOS processes and templates. The Contractor shall provide support as required for conducting security tests to validate that required security controls are properly implemented, operate as intended, and produce the desired outcome. The Contractor shall improve system security where possible and appropriate.

The Contractor shall fully cooperate with DOS audits, reviews, evaluations, tests, and assessments of Contractor systems, processes, and facilities. The Contractor shall comply with security controls as specified in National Institute of Standards and Technology (NIST) SP800-53 Rev 4 or later.

The Contactor shall build databases according to DOS configuration guides, with a focus of minimizing the amount of POA&Ms that can be found in a scan when obtaining an ATO.

3.8.2 Security Operations Guides

The Contractor shall ensure compatibility and adherence to Security Operations Guides. If no Diplomatic Security (DS) guide exists, the Contractor will develop a CA configuration guide based on other Federal configuration guides (e.g. Defense Information Systems Agency [DISA] Security Technical Implementation Guides [STIGs]) or industry best practices) for Government approval. The Contractor shall baseline configuration guides, following the CA Configuration Management (CM) process, and maintain release baselines. The Contractor shall review all approved baselines biannually to confirm that no changes/updates are required.

The Contractor shall notify the CA/CST Information System Security Officer (ISSO) of planned software changes to allow for assessment and authorization (A&A) before deploying into production and shall maintain current information boundaries in the system security plan (SSP).

3.9 Continuity and Disaster Recovery

The Contractor shall coordinate and collaborate with other CA/CST teams to develop a comprehensive, enterprise-wide Contingency/Disaster Recovery Plan(s), ensuring it encompasses all aspects of the system(s) and its sub-systems. The Contractor shall be prepared to assist or perform with an annual test, or when significant changes to the architecture occur.

A. Support facility contingency planning, disaster recovery, and testing, as needed.

B. Provide support and continuity back-end architecture documents associated with the system.

C. Assist with a data recovery and Continuity of Operations (COOP) plan(s) with standard failure detection and recovery procedures.

4.0 CONSTRAINTS AND ASSUMPTIONS

A. The Contractor shall conduct IT related services in accordance with this PWS and in compliance with all relevant U.S. Government and DOS regulations, policies, acts, guidelines, processes, and documents, to include the following:

a. CA/CST SAFe Agile & SDLC

b. CA/CST Enterprise Architecture (EA) standards and target architecture

c. Oracle Database current environment

d. SQL Server Database current environment

e. Project Management Institute's (PMI) Guide to the Project Management Body of Knowledge (PMBOK)

f. Information Technology Infrastructure Library (ITIL) v3

g. Enterprise Change Management (ECM) Process

h. IT-CCB and CA-CCB processes

i. CA SOA environment and service development that aligns and complies with CA/CST standards

j. CA data sharing Memorandums of Understanding and Interconnection Security Agreements and interagency security standards

1. Non-Production Environment Requirements

B. Hold and maintain for the duration of the contract the following certifications:

a. CMMI for Development (CMMI-DEV) v1.3 or v2.0 Maturity Level 4 or higher

b. CMMI for Services (CMMI-SVC) v1.3 or v2.0 Maturity Level 3 or higher

c. ISO 27001:2013 Information Security Management

d. ISO 9001:2015 Quality Management

e. Top Secret Facility Clearance

C. Comply/follow and maintain with CA/CST’s:

a. Continuous Integration / Continuous Delivery (CI/CD)

b. Configuration Management/Control

c. Quality Control

D. Provide related services in accordance with this PWS so these services will withstand scrutiny by the DOS Office of the Inspector General (OIG) and the Government Accountability Office (GAO).

E. Manage the workload using scalable processes and techniques that allow for surges in activities and for varying product priorities.

F. Provide a standards-based, best practices approach to providing the services and support based on Capability Maturity Model Integration (CMMI).

G. Provide, manage, and track annual training for employees, including DOS required training, and ongoing individual training to ensure staff are kept abreast on the latest…