B-2 PWS IRDA - updated.pdf

PERFORMANCE WORK STATEMENT

Inter-Domain Routing Data Analytics Framework - Support Services

I. BACKGROUND

The Advanced Network Technologies Division (ANTD) in the Information Technology Laboratory (ITL) of the U.S. Department of Commerce, National Institute of Standards and Technology (NIST) conducts research and development to advance test and measurement technologies and standards design and development for advanced network technologies. ANTD is currently involved in the Internet Engineering Task Force (IETF) effort to design and standardize technologies to increase the robustness and security of the Internet’s inter-domain routing system, and in particular the Border Gateway Protocol (BGP) (see:

https://www.nist.gov/programs-projects/robust-inter-domain-routing). ITL and ANTD also have programs focused on researching the applicability of artificial intelligence technologies to address security and robustness issues today's networks (see: https://www.nist.gov/programs-projects/trustworthy-intelligent-networks).

One such BGP security technology is Route Origin Validation (ROV) based upon a Resource Public Key Infrastructure (RPKI). RPKI technologies are beginning to be deployed in operational networks around the world. As more commercial deployments emerge, there is a need for test and measurement tools to provide more detailed diagnostic information that will help precisely identify when and where RPKI data and global routing data conflict.

In 2013, NIST developed and deployed the first version of an RPKI Deployment Monitor. This first version of the “NIST RPKI Monitor” focused entirely on providing a statistical characterization of the state of deployment of the emerging global RPKI infrastructure and its correlation to the global BGP routing infrastructure. This version of the NIST RPKI Monitor has been widely used by the Internet community to track the state of deployment of RPKI technologies.

In FY2019 and FY2020, software development of a completely new RPKI monitor and data analysis framework was conducted. The framework is comprised of new software modules for data gathering, data analysis and visualization / user interface. The framework provides all of the data analyses in the first generation NIST RPKI monitor, as well as extensions to provide advanced analysis capabilities. These new capabilities include: the ability to interactively explore measurement data; the ability to characterize the potential impacts of RPKI data on Internet routing; the ability to provide diagnostic information for changes in RPKI data that could trigger changes in BGP routing; and the ability to detect anomalies in changes in RPKI and routing data that might be indicative of actions that could adversely affect global BGP routing.

This new, version 2.0, framework has been deployed and is now operational (see: https://rpki-monitor-v2.antd.nist.gov/https://www.nist.gov/services-resources/software/nist-rpki-deployment-monitor).

This Performance Work Statement (PWS) defines NIST requirements for a Contractor to design, develop, test and deploy software extensions to the RPKI monitor v2.0 framework to (a) enhance its scalability and performance, (b) enhance its analysis capabilities by incorporating new data sources such as geolocation data and Regional Internet Registry (RIR) data on resource ownership, (c) add new capabilities for users to define a subset of Internet resources to monitor and receive update notices for, and (d) to prototype and evaluate machine learning techniques to detect anomalous changes in global routing and/or RPKI data sets.

https://www.nist.gov/services-resources/software/nist-rpki-deployment-monitor

II. SCOPE OF WORK

a. OBJECTIVES/PURPOSE

The purpose of this acquisition is to procure contractor support services for the following task areas:

1. Develop, test and deploy performance and feature enhancements to the existing RPKI

Monitor v2.0 framework.

2. Research and evaluate potential machine learning techniques to add the ability to identify anomalies in RPKI and global routing data.

3. Develop final documentation of the enhanced RPKI Monitor framework.

Deliverables shall be in the form of technical presentations, design and specification documents, open source software prototypes, documentation for all software tools and reports on test results. Results from these tasks will be incorporated by NIST into technical contributions to Standard Development Organizations (SDOs) and/or publications in technical conferences and journals.

b. SPECIFIC REQUIREMENTS:

The Contractor shall provide all support for project oversight, administration, and technical execution of this requirement. The Contractor shall complete the following tasks and provide the following deliverables:

Task 1: Enhancements to the RPKI Monitor v2.0 framework.

a) Develop, test and deploy software enhancements to the RPKI monitor v2.0 framework to improve the performance of interactive user selected analysis parameters.

b) Incorporate new data sources, including geolocation data for address blocks and autonomous systems, and dynamic access to RIR data sets to enable accurate representation of the organization that holds specific address blocks and AS numbers.

Task 1 Deliverables:

a) Software enhancements to the monitoring framework that improve the performance and scalability of the system.

• The enhanced system scalability of the system to enable an increase in the historical data maintained to 6 months while maintaining the same performance of the current system for any interactive request made through the web interface.

b) Software enhancements to the monitoring framework that incorporate on-line RIR data sources and geolocation data to improve the detail of existing analysis graphs and tables.

• The enhanced system is expected to support the capability to optionally show details such as the organization that owns an address or autonomous system.

c) Deploy the enhanced analysis framework as RPKI monitor version 2.1. Ensure that the deployed software is fully operational and has acceptable performance in the final deployment environment.

d) An internal report providing a quantitative description of the performance and analysis improvements of the Monitor 2.1 framework.

• The report shall be in the form of a presentation documenting the measured performance improvement of the system for all supported forms of analysis.

e) An internal git repository with all software associated with Monitor 2.1 framework.

• Final software deliverable shall be fully documented source code published to a NIST internal GitLab repository.

Task 2: ML-based algorithms for BGP and RPKI anomaly detection.

a) Conduct a survey of previous research in the use of machine learning and other advanced analysis techniques to identify malicious and anomalous events in the global BGP routing and RPKI systems. Identify promising machine learning algorithms for further empirical analysis.

b) Develop labeled data sets necessary for training and testing identified machine learning algorithms, using global BGP trace data and synthetically generated data of anomalous events.

c) Prototype and evaluate software to apply the identified machine learning algorithms identified in the survey to BGP and RPKI data sets.

Task 2 Deliverables:

a) An internal report providing a summary of relevant prior research and proposing technique(s) to prototype and empirically evaluate their effectiveness in detecting BGP and RPKI anomalies.

• The report shall be in the form of a presentation documenting the techniques surveyed, a summary of their technical approach and data requirements, and reported performance. The report is expected to identify one or more promising techniques to prototype.

b) A repository of generated testing / training data sets for the evaluation of machine learning algorithms for BGP and RPKI anomaly detection.

c) An internal report on prototype evaluation of the selected machine learning techniques.

• The report shall be in the form of a presentation documenting the techniques prototyped and their measured effectiveness for anomaly detection using the generated data sets.

d) A repository with all software associated with machine learning and data generation techniques prototyped.

• Final software deliverable shall be fully documented source code published to a NIST internal

GitLab repository.

Task 3: Final Software and Documentation.

a) Produce final internal documentation on the design, implementation, maintenance, and use of the RPKI Monitor 2.1 framework.

Task 3 Deliverables:

a) Final software and report on the system design and implementation. Reports and documentation shall cover all aspects of the enhanced monitor 2.1 framework and provide clear and concise instructions for the operation and maintenance of the system.

• The contractor shall provide an acceptance test demonstration of all deliverables.

• The report shall be in the form of a NIST Technical Note special publication and document all aspects of the design, implementation, and operation of the enhanced RPKI monitor system.

Task 4: Reporting

a) The Contractor shall provide monthly progress reports that document the progress and accomplishments for the previous month. The reports shall address activities accomplished, deliverables completed, outstanding deliverables, any delays, reasons for delays and plans to mitigate them, and recommendations that were brought to the attention of the Technical Point of Contact (TPOC). See tables below for Deliverables and Performance Requirements for additional details.

III. PERIOD OF PERFORMANCE

The period of performance shall be one (1) year from the date of the award.

IV. PLACE OF PERFORMANCE

All work shall be accomplished at the NIST Campus located in Gaithersburg MD, or through off-site telework when approved by the technical point of contact. Given the current campus access restrictions, should the restrictions be in place at the time the purchase order period of performance begins, the contractor is authorized to begin work remotely or via telework. Normal duty hours are Monday through Friday, 9:00 a.m. to 6:30 p.m. except for Federal holidays.

V. RISK LEVEL

The HSPD-12 designated risk level for this requirement is IT-Low Risk.

VI. TRAVEL

No travel is required or authorized for this requirement.

VII. GOVERNMENT FURNISHED PROPERTY

Upon award of this requirement, the Government will provide the Contractor with equipment needed in the performance of this requirement. The Government will also furnish the Contractor with an office, a desk, chair, computer, and telephone. Government-furnished equipment will be provided in an as-is and working condition.

All property, data, and information provided by the Government in the performance of this requirement remains the property of the Government and shall be surrendered to the government upon completion or termination of this requirement. Likewise, all deliverables generated under this requirement shall be the property of the Government.

VIII. DELIVERABLES

Task Quantity Due Date Task 1: Enhancements to the RPKI Monitor v2.0 framework.

1 5 months after award.

Task2: ML-based algorithms for BGP and RPKI anomaly detection.

1 10 months after award.

Task 3: Final Software and Documentation 1 11 months after award.

Task 4: Reporting. 12 No later than the tenth day of the next month.

IX. PERFORMANCE REQUIREMENT SUMMARY

Task.

Required Deliverable

Performance Standard

Monitoring Method

Task 1:

Enhancements to the RPKI Monitor v2.0 framework.

Software for enhanced implementation of the RPKI Monitor framework.

New RPKI Monitor v2.1 software tested and operationally deployed.

Software and tools shall be well structured, documented, and packaged in a git repository such that others can easily install and deploy the entire system.

Timely completion in accordance with the Deliverable Schedule

Software, tools, and functional test reports will be reviewed and approved by the NIST Technical Point of Contact.

Task2: ML-based algorithms for BGP and RPKI anomaly detection.

Internal reports and software prototypes of machine learning techniques applied to BGP and RPKI data.

Presentations and reports shall provide a detailed analysis of the ML algorithm selected and quantitative analysis of its effectiveness in detecting anomalies.

Deliverable Schedule

Presentations and report will be reviewed and approved by NIST Technical Point of Contact

Task 3: Final Documentation.

Develop final documentation and reports on the enhanced RPKI monitor v2.1 design and implementation.

Technical report that details all aspects of the software system and provides clear and concise quantitative results for the tests conducted.

The technical report will be of publication quality and will be submitted to NIST internal WERB review prior to submission for publication

Technical Reports will be reviewed and approved by NIST

Deliverable Schedule

Task 4: Reporting. Accurately document the activities accomplished, deliverables completed, outstanding deliverables, any delays and methods to mitigate them, reasons for delays, and recommendations.

Reports shall provide a clear, concise, and accurate depiction of progress to date and any issues identified in the last month.

Deliverable Schedule

Status reports will be reviewed and approved by NIST

Acceptance Criteria

The NIST Technical Point of Contact (TPOC) will provide comments on each deliverable within 14 calendar days from receipt of a given deliverable. The Contractor shall make any needed changes to the reports within 14 calendar days from receipt of electronic or written comments from the TPOC. The TPOC will provide written notification when a deliverable is accepted.

X. CONTRACTOR KEY PERSONNEL REQUIREMENTS

One (1) scientist/engineer is required as key personnel for this requirement. The proposed key personnel must meet the following minimum qualifications:

• Master’s Degree or higher from an accredited educational institution in Computer Science or Data Science.

• At least one (1) year of experience in the analysis of Internet routing and signaling protocols and Internet topology and routing data sets, including analysis of global BGP trace data and RPKI repository data sets.

• At least one (1) year of experience in the design and development of big data analytics platforms and technologies (e.g., Hadoop, Apache Spark, SQL, and non-SQL databases) and data visualization tools (NodeJS, D3JS, etc.).

XI. NIST VISITOR AND CONTRACTOR PROTOCOLS FOR MITIGATING COVID-19 EXPOSURE

The Contractor shall review and adhere to the requirements of Attachment I titled NIST Visitor and Contractor Protocols for Mitigating COVID-19 Exposure on NIST Campuses. Contractor personnel performing work in NIST workspaces, must provide all information required by Attachment I, in advance of and throughout the period of performance of the purchase order.

While working on the NIST campus, the Contracting Officer's Representative and/or Technical Point of Contact will ensure that Contractor personnel comply with the requirements in place to mitigate COVID-19 exposure.

The Government reserves the right to require that Contractor personnel attend a 1-2-hour training session dedicated to COVID-19 protocols.

Attachment I titled NIST Visitor and Contractor Protocols for Mitigating COVID-19 Exposure on NIST Campuses, is hereby incorporated and made a part of the contract. This document is subject to change and must be monitored by the Contractor via the following NIST websites: https://www.nist.gov and/or https://www.nist.gov/about-nist/visit/nist-visitor-andcontractorprotocolsmitigating- covid-19-exposurenist-campuses