Attachment B - Evaluation Criteria.pdf

Rating

Acceptable

Unacceptable

Evaluation Criteria

Evaluation Criteria # / Description

Provide a single console/interface by which to perform analysis, monitoring nad management of DPCI infrastructure, to include central policy management, device management, and report creation capabilities. (PWS Architecture 4)

Infrastructure/Application/Vendor Specs

Furnish a Data Access API which exposes services for import, playback, and regeneration of captured network traffic, as well as for automated query and search capability from outside the application. (PWS Architecture 8)

Decode and search files, such as those sent as e-mail attachments, encoded in any kid of Base64 or other schemes. (PWS Architecture 14)

Provide rapid ( 5 minutes or less), accurate ingestion and classsification of incoming data packets at wire speeds based on TCP/IP protocol/service. Permit extraction of packet captures from those packets in 15 minutes or less. (PWS Architecture 1)

Capture 100% of all network traffic traversing the network border, and reconstruct network applications and artifacts from the captured packets. (PWS Architecture 2)

Allow for the configuration of data retention objectives. (PWS Architecture 5)

Solution shall be architected to support 99% avaliability and uptime Disaster Recovery requirements for DoS users on a 24 x 7 x 365 basis. (PWS Architecture 6)

General Score Description

Provide for close integration--either by native API, system server, database connection or other means--with DoS' current enterprise tools, Splunk & Syslog log aggregation solution (PWS Architecture 7)

Permit a minimum of 50 concurrent ad hoc queries on captured packets to be launched by analysts in parallel, without any significant degradation or latency. (PWS

Architecture 15)

Allow and support a mininum of 75 automated, continuously running custom scripted queries with alerting. (PWS Architecture 16)

Make accessible to analysts the full range of all DPCI data collected within the network, regardless of the analysts’ geographic location. (PWS Architecture 9)

Have the ability to queue up data when a network line is down; the Contractor’s solution shall state explicitly the time range or storage capacity for this scenario. (PWS

Architecture 10)

After outages in network links have been resolved by the responsible third party, the solution rapid attain a rapid, complete reconciliation/normalization of DPCI metadata across POPs, ensuring analysts always have access to an unbroken continuum of 30 days' worth of DPCI packet capture data. (PWS Architecture 11)

Interoperate with solutions which intercept and decrypt Secure Socket/Transport Layer Security (SSL/TLS) packets. The Contractor’s solution shall demonstrate how increasing use of SSL in the Internet, coupled with higher bit levels in server SSL certificates (e.g. sites migrating from 512-bit to 1024-bit and 2048-bit certificates), will impact the overall performance of the DPCI solution. (PWS Architecture 12)

Function across 2 geographically separate points of presence (POPs) (PWS Architecture 3)

The solution shall support the ability to ingest 3rd party threat feeds and time-sensitive custom signature updates that can improve detection and alerting capabilites of interesting traffic. (PWS Architecture 13)

Description

Proposal clearly complies with all requirements of the PWS/specifications/salient characteristics and solicitation

Proposal does not clearly comply with all requirements of the PWS/specifications/salient characteristics and solicitation and/or lacks sufficient detail to be properly evaluated

Architecture Requirements

The Architecture set of requirements of the DPCI solution encompasses those functions which are built-in or organic to the device(s)

Infrastructure/Application/Vendor Specs

Allow scripted queries to be created, deployed, and saved by analysts. (PWS Analyst Functions 9)

Infrastructure/Application/Vendor Specs

Give analysts the ability to search captured network traffic by multiple metadata types including IPv4 address, IPv6 address, Web site Uniform Resource Locator (URL), or

Internet domain as well as URLs contained within traffic streams, e-mail messages, or documents. The content search syntax shall support Boolean logic in addition to regular expressions, in order to simplify certain queries. (PWS Analyst Functions 7)

Enable analysts to search on content hash values (MD5 or SHA-1) of email messages, as well as any files attached to those messages. (PWS Analyst Functions 8)

Provide reputation scoring of Internet domains, URLs, email attachments and other sources of potential malware. (PWS Analyst Functions 10)

Include tools for analysis of the captured packets, including anomaly detection capabilities. (PWS Analyst Functions 1)

Furnish a complete toolkit with which to create comprehensive rules and custom alerts, of varying degrees of severity, for analysts. (PWS Analyst Functions 2)

Discern ways in which IPv4 and IPv6 protocols and their associated services are being misused for potentially malicious purposes, apart from those defined in Internet

Engineering Task Force (IETF) Requests for Comments (RFCs). (PWS Analyst Functions 4)

Identify and alert on traffic using inherently insecure TCP/IP protocols, such as FTP and Telnet, as well as on traffic employing packets crafted for potentially malicious purposes. (PWS Analyst Functions 3)

Provide the ability to export discrete files, based on captured network data as their source. (PWS Analyst Functions 6)

Allow the export of any set of captured packets to generic Packet Capture (pcap) format. (PWS Analyst Functions 5)

Solution shall have an interface speed, sustained ingest rate, and application throughput inspection rate of a minimum of 40 gigabits per second (Gbps) at each of the 2 points of presence (POPs). (PWS Engineering Performance 1)

NetFlow anomaly analysis (PWS Analyst Functions 16)

Include 80% of the metadata requirements listed in the “DPCI Functional-Operations Requirements” spreadsheet. (PWS Analyst Functions 17)

Provide drill-down capability into captured network data which is ambiguous (i.e. is not immediately recognizable from metadata). (PWS Analyst Functions 13)

Generate reports/export datasets in multiple formats (e.g. Comma Separated Value (CSV), JavaScript Object Notation (JSON) and Extensible Markup Language (XML)), according to various criteria including source or destination IP address, or e-mail address. (PWS Analyst Functions 15)

Permit assignment of DPCI users into hierarchical roles or groups, which allows various levels of access or management capability (RBAC). (PWS Analyst Functions 11)

Feature documentation, in either HTML or fixed-file format (e.g. PDF), detailing its advanced features, such as how to create/modify custom rules and signatures. (PWS

Analyst Functions 12)

Furnish reporting tools within the application to allow analysts to create their own reports on selected data sets. (PWS Analyst Functions 14)

Customizable “don’t store packets ” function based on data type and source (for example: don’t store streaming video packets, don’t store files from YouTube) (PWS

Analyst Functions 18)

Customizable "store packets' function based on data type (for example: store packets involving medium, high and critical security alerts) (PWS Analyst Functions 19)

Engineering & Peformance Requirement

Set of requirements includes those functions and operations, of which DoS cybersecurity analysts will utilized

Set of requiremens defines the minimum acceptable levels of operation for overall DPCI solution, as defined by industry-standard criteria

Analyst Function Requirements

4 Describes provision for DPCI solution, throughout the acquisition, installation and operations phases of the solution lifecycle

Services and Support

Afford 99.9 % or better uptime of DPCI systems, achieved through a well-planned, resilient architecture and through high-quality hardware components which allow for redundant solution hardware configurations, network connectivity, hot swapping of defective components, etc. Examples of such components include power supplies, RAID controllers, and hard drives. The Contractor’s solution shall demonstrate how this is to be achieved. (PWS Engineering Performance 4)

Solution shall provide analysts access to a minimum of 30 (thirty) calendar days’ worth of captured packets. To accommodate the storage capacity required for this time period, the Contractor's solution shall include one or more options demonstrating scalabiltiy and versatility in DPCI storage with cost-effective hardware configurations.

Preference will be shown to storage solutions emphasizing low cost throughout the DPCI solution 5 year lifecycle, and minmum number of hardware components. (PWS

Storage solution shall provide 30 days of on-hand packet capture for analysis at existing bandwidth rate of 20 gigabits per second (Gbps) at each of the 2 points of presence

(POPs). Contractor shall demonstrate storage solution scalability supporting 30 days of on-hand packet capture for analysis at 40 gigabits per second (Gbps)traffic rate.

(PWS Engineering Performance 3)

Furnish root-level access to operating system, application, and all components to DoS for full system lifecycle management and administration. (PWS Engineering

Performance 5)

The management solution shall provide a management interface at each of the 2 points of presence (POPs), supporting solution management redundancy, to allow for the configuration, monitoring, and reporting for all components of the DPCI solution from either POP location.

The contractor shall outline clearly defined Returned Mechandise Authorization (RMA) processes, with 4-hour drop shipment of replacement components (PWS Service and

Support 6)

Contractor must provide US- based support accessible 24 hours a day x 7 days a week, by phone, web, email and a dedicated account subject matter expert (SME) for DoS, as well as a clearly defined escalation and resolution process included within the solution. (PWS Service and Support 1)

Contractor shall deliver detailed description of warranty, license, and maintenance support to include SLA and mean time to resolution projections for standard and severe support incidents. (PWS Service and Support 2)

The contractor shall furnish specialized training options to include, classroom and web-based training, for analysts and securrity engineers to include a) a. Administration of system for security engineering staff, to include full life-cycle support of all components, b)Junior / Mid-level analyst training, c) Senior analyst training. (PWS Service and

Support 3)

The contractor shall forumulate a detailed installation/integration/implementation plan accompanied by a schedule and resource allocation plan. (PWS Service and Support

4)

The new equipment/components shall be of the newest avaliable model, without being on any end-of-maintenence, end-of-sale, or end-of-life list for a minimum of 5 years from date of purchase. (PWS Engineering Performance 9)

The solution shall be able to support soft client access to management capabilities at both POPs as a backup solution. In addition to this, the solution should have a federate dashboard where the analyst can work from both POPs. (PWS Engineering Performance 10)

Solution shall function between 2 points of presence (POPs) and monitor 100% of traffic internal and external. (PWS Engineering Performance 6)

The contractor shall provide access to a secure customer web portal with the miniumum characteristics: a) inventory of all assets, b) SOPs, adminstration guides, installation guides, white papers, c) A repository of queriable knowledgebase articles (KBAs)for engineers and analysts, d) online ticketing for break/fix requests and historical tracking. (PWS Service and Support 5)

Services and Support

Preference will be shown to storage solutions emphasizing scalability of solution to support collection and storage of up to and or exceeding unbroken 6-months (180 days) of packets. (PWS Engineering Performance 7)

Contractor shall issue notice of recurring software system updates via email: these updates to be accessible by a secure download method to provide a) New feature updates, b) Bug fix and security updates, c) New knowledge base documents, d) Updated risk signatures, security content, threat intelligence, and indicators of compromise

(PWS Service and Support 7)