SPLUNK__Project_Concept_and_Scope__RDF-_No_IGCE-1.pdf

Project Concept and Scope of Work Splunk> FY2016

Prepared by: IRM/IA/ISSO/OPS Document Release: 1.0

31 July 2015

United States Department of State

BUREAU OF INFORMATION RESOURCE MANAGEMENT (IRM)

UNCLASSIFIED Project Concept for Splunk

07/31/2015 Page 2 of 18

Document Control Information

Version Date Additions/Modifications Prepared/

Revised by

Draft 30-07-2015 Rough draft Robert D. Fuentes

1.0 31-07-2015 Released Robert D. Fuentes

07/31/2015 Page 3 of 18

1. Project Overview Any government agency, whether it is a civilian, defense or intelligence agency, depends on information technology to help support data integrity, reduce mission risk and ensure the confidentiality and availability of information.

1.1 Project Leadership

Government Project Manager Information

Name, Title Robert D. Fuentes, Division Chief

Bureau/Office IRM/IA/ISSO/OPS

Phone Number 703-217-5719

Table 1 Government Project Manager

Project Sponsor Information

Name, Title David Rowles, Director

Bureau/Office IRM/IA/ISSO

Phone Number 703-812-2201

Table 2 Project Sponsor

Contractor Project Manager Information

Name, Title TBD

Bureau/Office

Phone Number

Table 3 Contractor Project Manager

07/31/2015 Page 4 of 18

1.2 Business Need

Splunk is a highly scalable engine for machine generated IT data that collects, indexes, and harnesses data from across physical, virtual, or cloud infrastructures in real time. Unstructured machine data is given structure through time-based indexing so that analytics can be applied to that data to gain insight and understanding. Splunk captures and monitors real-time data streams from applications, network devices, hosts, security devices and software. In addition to real-time streams, Splunk analyzes historical data looking for trends, patterns, and anomalies. Splunk supports continuous monitoring and also delivers a security-related context you can apply to any event from any layer of an IT infrastructure, and using Splunk will avoid and prevent business impacting incidents through visibility and proactive monitoring. IRM/IA needs funding and resources to engage Splunk professional services for subject matter expert consultants for Splunk engineering and architect advisors, project management, software installation and configuration, systems administration, and application deployment, training, and maintenance.

2. Scope Statement The scope of IRM/IA’s Splunk FY2016 effort will consist of a tools development project of five points plus a Splunk Development and Operating Training Strategy (Splunk>DOTS).

Figure 1: Splunk>FY2016 5-Point Development Project

2.1 Subject Matter Experts (SME): Engineering Advisors and Technical Consultants Procure an experienced Splunk Architect/Advisor and an expert technical consultant, who will work closely with IRM/IA Management to put together a detailed project plan and technical architecture for each of the Splunk>FY2016 project points described herein; develop, install, and administer Splunk>FY2016 applications; and assist and enable IRM/IA Management to develop and execute plans for application deployment and knowledge transfer training.

2.1.1 SME Advisory Engineer.

Will participate in meetings to understand architectural and technical challenges, resource constraints, IRM/IA goals and priorities, etc.. and will help IRM/IA to create a realistic project plan, provide project management support and reporting, recommend appropriate level of training and knowledge transfer, etc. The SME Advisory Engineer will be involved in creating the project plan, creating the standard operating procedures and instructions for the ISSO log monitoring solution to be deployed to all foreign and domestic Department posts. The SME will deliver a Splunk architecture for the 5-Point project and Splunk>DOTS solutions; work closely with the onsite Splunk technical consultant to ensure solutions meet the IRM/IA requirements and the agreed upon project plans.

07/31/2015 Page 5 of 18

Finally, the SME will advise on best practices for the various use cases and suggest other use cases to IRM/IA management using the agency’s existing technologies to bring a more robust security and compliance solution to the Department.

The Splunk Advisory Engineer will work on site at Department of State Annex (SA-15), in Rosslyn, Virginia, 5-days per week for one year, during the current period of performance.

General Work Product(s) and Frequency for Subject Matter Expert (SME) Architecture and Engineering Advisory and Planning

Splunk will assign an experienced Splunk architect/advisor who will work closely with IRM IA Management to put together a detailed Project Plan and technical architecture for each of the three projects listed above. The Splunk Advisory Engineer will participate in State Department meetings to understand architectural and technical challenges, resource challenges, goals, etc. that will help them create a realistic project plan and will enable them to recommend the appropriate level of training and knowledge transfer.

The SAE will be involved in creating the project plan, creating the standard operating procedures and instructions for the ISSO log monitoring solution to be deployed to all foreign and domestic State Department Posts. The SAE will deliver a Splunk architecture for the three solutions and work closely with the onsite Splunk technical consultant to ensure the solutions meet the State Department requirements and the agreed upon project plan. Finally, the SAE will advise on best practices for the various use cases and suggest other use cases to the IRM IA management using the agency’s existing technologies to bring a more robust security and compliance solution to the State Department.

The Splunk Advisory Engineer will work on site at SA-15, State Department IRM IA in Rosslyn, VA (5 days per week for one year).

• Status Report (Weekly)

• Statement of objectives, requirements, & constraints (w/in 48hr)

• ROMs (w/in 48-hr)

• ConOps (w/in 48-hr)

• Work Breakdown Structure (w/in 48-hr)

• Org. Breakdown Structure (w/in 48-hr)

• Service & work product design(w/in 48-hr)

• Resourced project plan/sched (w/in 48-hr)

• Basis of Estimate (w/in 24-hr)

• Budget (w/in 48-hr)

• Staffing plan (w/in 48-hr)

• Position description drafts (w/in 48-hr)

• Prototype solutions (w/in 8-weeks)

• Training and information support (as agreed upon @ request)

• Phone and email support (w/in 2-hr)

• Work progress reports (Ad hoc) (w/in 48-hr)

• Financial status reports (w/in 48-hr)

• Test reports (w/in 48-hr after testing completed)

• Customer feedback(w/in 24-hr)

Table 4: Work Products and Frequency - SME Advisory Engineer

2.1.1.1 SME Technical Consultant.

07/31/2015 Page 6 of 18

Will be responsible for developing Splunk solutions to meet the 5-Point Project goals. This consultant will provide knowledge transfer to the Department project team. The SME consultant will have experience in Splunk platform, search language, GUI interface, and a knowledge of other Security and Compliance tools and how they integrate with Splunk.

The Splunk SME Technical Consultant will be on-site at Department of State Annex (SA-15), in Rosslyn, Virginia, for an estimated 16 weeks to complete the 5-Point Project. The specific time and specific deliverables will be determined after the project plan is completed.

for Subject Matter Expert (SME) Technical Consultant

Splunk will assign a hands-on Splunk expert who will be responsible for developing Splunk solutions to meet the three goals. This consultant will also provide knowledge transfer to the State Department project team. The Splunk consultant will have experience in Splunk platform, search language, GUI interface and knowledge of other Security and Compliance tools and how they integrate with Splunk. Splunk estimates the consultant will be on-site 16 weeks to complete the three projects. The specific time and specific deliverables will be determined after the project plan is completed.

• Status Report (Weekly)

• Server standup & tear-down, installation, & config (w/in 72-hr)

• Server administration, maintenance, & management (w/in 72-hr)

• ROMs (w/in 48-hr)

• ConOps (w/in 48-hr)

• Prototype solutions (w/in 8-weeks)

• Training and information support (as agreed upon @ request)

• Phone and email support (w/in 2-hr)

• Test reports (w/in 48-hr after testing completed)

• Customer feedback (w/in 24-hr)

• Security categorization forms (w/in 5-Days)

• Security plans (w/in 5-Days)

• Contingency plans (w/in 15-Days)

• Help desk support as needed/requested (w/in 2-hr)

• Application configuration (as needed/requested) (w/in 48-Hrs)

• User guides (Initial/incremental updates) (w/in 30-Days)

Table 5: Work Product(s) and Frequency - SME Technical Consultant

2.1.1.2 SME Project Manager Consultant (SME PMC)

Will be focused on planning the Department’s Splunk>FY2016 architecture hardware and sizing, storage requirements, customer locations, roles and responsibilities; identifying training requirements (for those responsible to manage the Splunk environment as well as for Enterprise users) and helping IRM/IA Management to determine and schedule Splunk Education. The SME PMC will also enable next customer security Use Cases and the data sources required are reviewed and prioritized. (This provides details to guide the deployment of Splunk forwarders, Technical Add On’s, and additional Splunk applications.)

The SME PMC will also help develop an integrated Enterprise Security project plan to manage and schedule resource tasks; a communication plan for the project team and key stakeholders as well as weekly team status calls. The SME PMC will Issue and project risks, monitoring and tracking those as part of the weekly status updates.

07/31/2015 Page 7 of 18

Further, when the required hardware, storage, and operating systems are installed, Splunk Professional services resources will install Splunk Core and Enterprise Security, Splunk Deployment server, Search Heads and Indexers.

In parallel coordinating with the customer infrastructure team the Splunk forwarders are deployed. Onboarding of data sources will begin and be validated within Splunk Enterprise Security application. The initial 5-Points Splunk>FY2016 enterprise security dashboards, custom searches, and alerts and notifications are tested and validated by the SME PMC.

The SME PMC will also configure, test, and validate any additional custom views and dashboards; schedule a final review with Enterprise Security users and administrators in preparation for IRM/IA sign off and acceptance

The project manager will be be on-site at Department of State Annex (SA-15), in Rosslyn, Virginia, for one day per week during the same 16 weeks period for the SME Technical Consultant for Subject Matter Expert (SME) Project Manager Consultant (PMC)

Splunk project management is focused on delivering value incrementally within global organizations. With the customer team we plan the Splunk architecture hardware and sizing, storage requirements, customer locations, roles and responsibilities.

Training requirements for those responsible to manage the Splunk environment as well as Splunk Enterprise Security users are determined and scheduled with Splunk Education. Next customer security Use Cases and the data sources required are we reviewed and prioritized. This provides details to guide the deployment of Splunk forwarders, Technical Add On’s, and additional Splunk apps.

Once there is an understanding of the scope and objectives we develop an integrated Enterprise Security project plan to manage and schedule resource tasks. A communication plan will be developed for the project team and key stakeholder as well as weekly team status calls.

Issues and project risks are also monitored and tracked as part of the weekly status updates.

When the required hardware, storage, and operating systems are installed Splunk Professional services resources will install Splunk Core and Enterprise Security, Splunk Deployment server, Search Heads and Indexers. In parallel coordinating with the customer infrastructure team the Splunk forwarders are deployed. Then the onboarding of data sources will begin and

• Status Report (Weekly)

• Server standup & tear-down, installation, & config (w/in 72-hr)

• Server administration, maintenance, & management (w/in 72-hr)

• Statement of objectives, requirements, & constraints (w/in 48hr)

• ROMs (w/in 48-hr)

• ConOps (w/in 48-hr)

• Budget (w/in 48-hr)

• Prototype solutions (w/in 8-weeks)

• Training and information support (as agreed upon @ request)

• Phone and email support (w/in 2-hr)

• Test reports (w/in 48-hr after testing completed)

• Customer feedback (w/in 24-hr)

• Security categorization forms (w/in 5-Days)

• Security plans (w/in 5-Days)

• Contingency plans (w/in 15-Days)

• Help desk support as needed/requested (w/in 2-hr)

• Application configuration (as needed/requested) (w/in 48-Hrs)

• User guides (Initial/incremental updates) (w/in 30-Days)

07/31/2015 Page 8 of 18 validated within Splunk Enterprise Security application.

The initial Enterprise Security dashboards, custom searches, and alerts and notifications are tested and validated. Any additional custom views and dashboards will be configured, tested, and validated. A final review is scheduled with Enterprise Security users and administrators in preparation for customer sign off and acceptance.

After the required customer Use Cases and dashboards have been completed and tests the Enterprise Security User roll out can be planned and executed.

The project manager will be assigned (1) one day per week during the 16 week.

2.1.2 Server Log Monitoring (SLM) Dashboard

IRM/IA is developing a deployment plan, which will include Standard Operating Procedures (SOP) and training for ISSOs/IMOs on how to use the SLM Dashboard. (As a result of the unexpected retirement of its Splunk SME/developer, IRM/IA has had to delay its deployment planning.) As the next step, the SLM Dashboard portion of the 5-Point Project will review and analyze the SLM Dashboard architecture, ensuring that it can be integrated with the rest of IRM/IA’s 5-Point Splunk>FY2016 Project. A plan for deploying the SLM Dashboard to ISSOs located throughout the world will be created, and standard operating procedures (SOP) will be developed and promulgated.

2.1.2.1 SLM Dashboard Description

SLM Dashboard was designed so that each post would require its own installation of Splunk on a single server that would act as both a Search Head (front-end) and an Indexer (back-end). The Indexer would ingest feeds from the various, on-site Microsoft Windows Servers at its post. The Forwarder would be installed on the same server and would send a copy of the event logs entries generated by the post servers. Original events would remain on the single server at post. Once received, event logs would be indexed and the data normalized. ISSOs would access the data through the Search Head using the web interface provided by Splunk. The SLM Dashboard would be configured within every Search Head, which would show critical event types that must be monitored.

(A master server (running the same version of Splunk) will reside in Washington, DC. The role of the master server will be to act as a licensing server and a configuration master. Thus, each of the deployed Splunk servers would receive applications and settings from the master server.)

2.1.3 FISMA N800-53 Controls Monitoring

IRM/IA’s Splunk>FY2016 Project point for FISMA N800-53 Controls Monitoring tool is intended to create and implement a layer of automated continuous monitoring to reduce the time and labor spent poring over gigabytes/terabytes of data from workstations, servers, applications and other elements across the IT infrastructure. Using the FISMA N800-53 Controls Monitoring tool, data could be cross-referenced directly to specific and appropriate NIST 800-53 controls. This tool will provide a highly scalable data management engine that would scale linearly across commodity servers and support volumes of systems NIST controls data.

07/31/2015 Page 9 of 18

The FISMA N800-53 Controls Monitoring tool will provide a framework for continuous monitoring of specific NIST 800-53 controls for Windows and Unix/Linux systems that will ease the burden of assessment and authorization (A&A) compliance. The following controls could be continuously monitored with searches and dashboards within the tool:

• AC-2 Account Management

• AC-3 Access Enforcement

• AC-4 Information Flow

• AC-6 Least Privilege

• AC-7 Unsuccessful Login Attempts

• AC-12 Session Termination

• AC-14 Permitted Action w/o Authentication

• AC-17 Remote Access

• AC-18 Wireless Access Restrictions

• AU-2 Auditable Events

• AU-3 Content of Audit Records

• AU-5 Response to Audit Processing Failures

• AU-8 Time Stamps

• AU-9 Protection of Audit Information

• CM-4 Configuration Changes

• PE-11 Emergency Power

• PE-14 Temperature Controls

• SC-10 Network Disconnect

• SI-3 Malicious Code Protection

• SI-11 Error Handling

• IA-5 Authenticator Management

• PS-4 Personnel Termination

The FISMA N800-53 Controls Monitoring tool will also provide an easy-to-read graphical view of these controls in the “Continuous Monitoring” dashboard that supports continuous monitoring for ACM – Account Management; ACP – Privileged Access; and ACL – Login Access. The tool will provide a simple and clean view with high-level gauges, historical trending, and heat map to enable ISSOs and othes to quickly determine if any controls need further investigation. The will give a detailed view with interactive charts and tables that enable ISSOs to immediately drill down into the original event data and further understand what is causing the increased risk.

The FISMA N800-53 Controls Monitoring tool will be based on a common information model (CIM) to allow for easy mapping of new data types and sources into the tool.

2.1.4 Continuous Diagnostics and Monitoring Tools (CDMT) Dashboard The intent of CDMT Dashboard is to enable IRM/IA to continuously look crossfunctionally to quickly diagnose any issues that affect system and service availability across application management, security and IT operational areas CDMT’s Splunk platform for machine data and app ecosystem will provide the means by which IRM/IA can rapidly enjoy the benefits of CDM and identify issues before problems begin.

2.1.4.1 Foundation for hardware, software, and configuration management - CDM Functional Areas 1, 2, and 3.

The CDMT Dashboard will enable IRM/IA to collect asset information from Active Directory (AD), enterprise configuration management databases (CMDB), management systems, network scanning tools, custom scripts and directly from hosts – as needed and authorized by CISO. The data collected can be correlated and enriched with information from other sources, including ForeScout, lookup tables within Splunk, external relational databases (using Spunk DB Connect), and so forth.

Figure 2: The continuous monitoring process steps in NIST 800-

07/31/2015 Page 10 of 18

The CDMT Dashboard will also enable the output from vulnerability scanning to be ingested Splunk software to provide a comprehensive view of risk across the Department’s IP-addressable assets. Using Search Processing Language (SPL

), CDMT will enable ISSO Ops Remediation Coordinators, ISSOs, and other cybersecurity professionals within the Department to quickly search through assets, asset network activity, intrusion and firewall activity, and any open vulnerabilities on the assets. (This functionality will serve as the foundation for vulnerability management identified as CDM Functional Area 4.)

2.1.4.2 Managing trust in people granted access - CDM Functional Area 6 CDMT Dashboard will monitor and alert on difficult-to-detect malicious insider activity by correlating identification, authentication, authorization and accounting (IAAA) data, as well as other network-based activity. It will support IAAA across AD infrastructures, workstations, email systems and other sources to allow identification of a single user identity across multiple online nomenclatures such as email addresses, LAN accounts and social media profiles. CDMT’s Splunk identity correlation will capture and log attempted access across a multitude of platforms and network devices, tracking unwanted users in the network with multiple repeated login failures, unauthorized access attempts and inappropriate privilege escalation. By monitoring use of credentials across multiple domains and authorization granted to those credentials, CDMT’s Splunk software will ensure that identities on the network are who they claim to be and are only accessing resources to which they are authorized/entitled. Patented after the Splunk App for Enterprise Security Identity Center dashboard (see Figure 3 below), the CDMT Dashboard will expedite reporting on these and other dimensions of IAAA.

Figure 3: Splunk App for Enterprise Security Identity Center dashboard

2.1.4.3 Managing security related behavior - CDM Functional Area 7.

(Splunk software indexes every field, word, character – including punctuation -- and whitespace of the data it ingests. By analyzing the patterns of data, trends in network and host access behaviors, an ISSO will be able to quickly identify activity and patterns that lie outside of the norm and drill down to the original source events for corroboration and additional granularity (for example, see Figure 4 below).

- SPL enables users to createreal-time alerts, conduct advanced statistical reporting and create data visualizations on machine-generated data

07/31/2015 Page 11 of 18

Figure 4: Sample visualization of Splunk predictive analysis

2.1.4.4 Managing Audit Information - CDM Functional Area 14

The CDMT Dashboard will enable Splunk software to collect and index the machine data generated by almost every device and platform on the network, making auditing of events quick and efficient (see Figure 5). It will provide a consistent interface and experience across all tiers of ISSOs and offer a single location to examine all audit logs, including both real-time and historical events.

Figure 5: Sample of FISMA compliance and auditing support

2.1.4.5 Managing Operation Security - CDM Functional Area 15

Drawing on Splunk software’s core strength (the ability to consume large volumes of machine data and make it accessible and usable by all communities of interest, CDMT Dashboard will enable reports that are valuable to IRM/IA, ISSO, and other security analysts and executives in the Department to collect, correlate, visualize, and analyze data from a myriad of sources within the Department’s IT environment, such as from ForeScout, FoundStone, Bromium, etc.

07/31/2015 Page 12 of 18

3. SPLUNK>FY2016 Project Information

3.1 System Concept Overview

(See, Appendix A for a larger diagram)

Figure 6: Splunk>FY2016 Overview

07/31/2015 Page 13 of 18

3.2 Estimate Project Timeline

(See, Appendix B for a larger diagram)

Figure 7: Project Timeline

07/31/2015 Page 14 of 18

3.3 Government Costs Estimate: Project Costs

-- INTENTIONALLY REMOVED --

Table 6: Estimate Project Costs

07/31/2015 Page 15 of 18

4. Security

4.1 Contractor Identification

4.1.1 Contract performance may require contractor personnel to attend meetings with government personnel and the public, work within government offices, and/or utilize government email.

4.1.2 Contractor personnel must take the following actions to identify themselves as non-federal employees:

4.1.2.1 Use an email signature block that shows name, the office being supported and company affiliation (e.g. “John Smith, Office of Human Resources, ACME Corporation Support Contractor”);

4.1.2.2 Clearly identify themselves and their contractor affiliation in meetings;

4.1.2.3 Identify their contractor affiliation in Departmental e-mail and phone listings whenever contractor personnelare included in those listings; and

4.1.2.4 Contractor personnel may not utilize Department of State logos or indicia on business cards.

4.1.3 Department Of State Personal Identification Card Issuance Procedures

4.1.3.1 The Contractor shall comply with the Department of State (DOS) Personal Identification Card Issuance Procedures for all employees performing under this contract who require frequent and continuing access to DOS facilities, or information systems. The Contractor shall insert this clause in all subcontracts when the subcontractor’s employees will require frequent and continuing access to DOS facilities, or information systems.

4.1.3.2 The DOS Personal Identification Card Issuance Procedures may be accessed at http:// www.state.gov/m/ds/rls/rpt/c21664.htm .

4.2 Security Requirements

4.2.1 A facility security clearance at the [SECRET] level is required for contract performance in accordance with the DD Form 254, Department of Defense Contract Security Classification Specification.

4.2.1.1 Since it will be necessary for some Contractor personnel to have access to classified material and/or to enter into areas requiring a security clearance, each Contractor employee requiring such access must have an individual security clearance commensurate with the required level of access prior to contract performance. Individual clearances shall be maintained for the duration of employment under this contract, or until access requirements change.

4.2.1.2 The Contractor shall obtain a Department of State building pass for all employees performing under this contract who require frequent and continuing access to Department of State facilities in accordance with Bureau of Diplomatic Security Instructions.

4.2.1.3 Performance of this contract shall be in accordance with the attached DD Form 254, Department of Defense Contract Security Classification Specification and FAR 52.204-2 “SECURITY REQUIREMENTS.”

4.2.1.4 Classified material received or generated in the performance of this contract shall be safeguarded and disposed of in accordance with the National Industrial Security Program Operating Manual (DOD 5220.22-M).

07/31/2015 Page 16 of 18

4.2.2 This clasuse applies to the extent that this contract involves access to information classified “Confidential,” “Secret,” or “Top Secret.” The contractor shall comply with –

4.2.2.1 The Security Agreement (DD Form 441), including the National Industrial Security Program Operating Manual (DoD 5220.22M); and

4.2.2.2 Any revisions to that manual, notice of which has been furnished to Contractor.

07/31/2015 Page 17 of 18

Appendix A: System Concept Overview

07/31/2015 Page 18 of 18

Appendix B: Project Timeline FY-2016