PARIDINE_Industry_Day_Q+A.pdf

2017 DHS S&T Cyber Security Division BAA Industry Day

Predict, Assess Risk, Identify (and Mitigate) Disruptive Internet-scale Network Events

(PARIDINE)

May 25, 2017

Mayflower Hotel 1127 Connecticut Ave.

Washington DC 20036

Question and Answer Discussion – Cyber Security Division and Office of Procurement Operations

1. Are there related NIST standards for NIDE metrics?

Not that I know of, if you know of relevant NIST standards I would be happy to have that information.

2. What are examples of the external tools that you want to share the NIDEs results with?

Certainly the risk assessment and mission impact mitigation tool from the third BAA, but also there are tools like ESRI, which is a first responder tool, where you would want to use that information. This relates to BAA call paragraph 3.1.4 and Figure 3 has been added to the PARADINE BAA call via amendment.

3. Attribution is in all three BAAs; the question is, how does that change?

I expect that the attribution for the PARADINE BAA call will not be as in depth, not as detailed, but the first step in the process. Hopefully that attribution will mature for the second and third BAAs. We’ll just have to see what the proposals are. Attribution is a really hard problem, but I would love to make progress on it.

The attribution will be more accurate, more in depth as the BAAs progress.

However, paragraph 3.2 has the scope for attribution for this BAA call where progress on attribution quality should be made with work toward TTA #2 Goal 2.

4. What kind of attribution are you looking for?

Well first would be network attribution, next device attribution, and third individuals. This relates to BAA call paragraph 3.2, which has been updated to reflect this answer.

5. What does attribution mean if the cause of the event is not a person?

As indicated in BAA call paragraph 3.2, we are looking for root cause analysis, so if it’s a cable cut, you tell us it’s a cable cut, and where it happened, or if a squirrel gets fried on an electrical line then you tell us that’s what happened and where it happened going back to information as to what caused it, that’s what attribution means.

6. Would a framework enabling organizations to jointly mine the data for NIDEs without disclosing sensitive information be in scope for either TTA?

Related to BAA call paragraph 3.1, the answer is yes, that framework would be in scope. It would need to be applied to at least one data set so that we can make sure that it works. Please see BAA call 6.5.2 as well.

7. Can you elaborate more on the control plane disruptions?

See BAA call paragraph 3.1.1. What I had in mind was BGP and BGP hijacking, but there could be other things, software defined network could be in scope.

8. What networks are in scope?

Referring to BAA call paragraph 3.0, proprietary, commercial networks, overlay distributed networks, all networks are in scope. We are looking at large events first, down to more fine-grained events, depending on how the technology develops.

9. What are the main privacy considerations?

Privacy will be dictated by whatever data is used. If the data contains PII (Personally Identifiable Information) then privacy is of great concern and needs to be addressed. If it is open source data, available anywhere, then privacy is of less concern, although there is still the possibility of having privacy concerns when data is aggregated. So privacy should be addressed and be determined by the data that is used. Also, please remember the requirements for the Data Management Plan (DMP) required of each proposal by BAA HSHQDC-17-R-B0001, paragraph

9.6.1 w., which addresses privacy.

10. How much emphasis is on real world prototyping, for example plugging sensors into a smart city versus coming up with models and algorithms?

Looking at BAA call Section 3.0, you see a progression of goals that build on each other as a function of maturity. I look at this as a continuum, you start with models and algorithms and then you move to a real world setting.

11. Will the solicitation provide continuing funding for the Internet Measurement and Attack Modeling (IMAM) program awardees?

BAA call HSHQDC-17-R-00059 is a standalone solicitation with goals defined therein not resulting from IMAM. IMAM, the Internet Measurement and Attack Modeling program, all awards have been done, that program and its associated projects are ending either this year or in 2018. There may be some funding into next year, but that program is over. The Application of Network Measurement Science is the new program, and we look at all proposals as they come in and judge them. There are no pre-conceived winners here.

12. 14. Doug Maughan - Can we involve researchers from countries other than the ones listed in my presentation?

See HSHQDC-17-R-B0002, Section 5 for eligibility specifics, but generally, yes.

13. Can we comment on how teaming arrangements will be considered - is there a team award or an individual award?

I will encourage all of you to look at the overarching solicitation, BAA HSHQDC- 17-R-B0002, that includes the evaluation criteria. The intent of the PARADINE BAA call is to foster the development of technologies to meet the PARADINE BAA call goals. How an offeror chooses to propose, be it as a team or as an individual, is an offeror discretion. Again, all proposals will be evaluated against the four evaluation criteria: the first is the technical topic and how well you have addressed the technical topic. The second is on the team or the performers; what is your background and what capabilities do you have, why should we fund you or the team that you put together. Third is the transition story, so you have to make sure you have a transition story; what is that story, and does it make sense. And fourth, and least important to us, is cost. We will evaluate proposals for cost, to make sure that what you are proposing is reasonable within the dollar amount you are proposing. We encourage you to look at that criteria, it doesn’t matter to us whether it’s a single performer or whether it’s a team of five people, realizing of course that the money only goes so far if you have five people teamed together.

14. Do we have a preference for industry, academia, labs?

No preference for us subject to HSHQDC-17-R-B0002, Section 5. You need to make sure that the team is as strong as it can be, and we don’t care how that team is made up.

15. How much industry vs purely academic participation is expected, do you expect to fund any with no industry partner?

HSHQDC-17-R-B0002, Section 5 defines eligibility requirements. There is no preference for performer type(s) beyond the eligibility requirements. In the past, we have funded numerous universities. We want to make sure the team is solid, whether the team is sole proprietor or a team of five, and don’t care if the team comes from the lab, industry or academia.

16. Is there a requirement to utilize the SWAMP or DETER as part of the performers deliverables?

While HSHQDC-17-R-B0002, Section 3, mentions SWAMP, DETER, and IMPACT, there is no requirement to use SWAMP or DETER, or IMPACT, the three research infrastructures that we make available. You certainly can and they are there for that reason. Our expectation is that some of the data that comes out of this program will be fed into the IMPACT repository. That repository is available, as I mentioned in my remarks, some of the data sets are completely open and do not require any type of an agreement or an MOU, some of them are very sensitive but are still made available by the data provider, and you do have to sign an MOU to get access to them. The expectation is that the data from PARIDINE and from the ANMS program will continue to feed IMPACT, but as far as SWAMP, we are not there yet, but someday in the future, if I have my way, all research code will be required to go through the SWAMP before we will accept it as a deliverable. And if it doesn’t pass the test, you will go back and fix it. But we aren’t quite there yet, so I can’t force you to do that.

17. What’s the minimum and maximum number of awards to be expected?

As indicated in the Industry Day slides, the number of anticipated performers is not known.

18. How does transition timeline work in the case that the technology isn’t useful in itself but depends on the results of later BAAs in the program?

The PARADINE BAA Call will be amended to include a figure that depicts offshoots of this BAA and BAA #2 that didn’t specifically go to the other parts of the program. There are technologies that we expect to come out of the program that will be useful, even though they can feed into BAA #2 and BAA #3, which will also be stand-alone and useful in and of themselves. We strongly believe that outputs from this BAA will be usable in other parts of the program and there are things that we expect to come out of here that will be stand-alone and potentially transition-able as part of that activity.

19. Will the government be open to consuming this data as a service?

Long term my goal is yes, initially, no and therefore the PARADINE BAA call does not include this as a goal. You saw Henning, he was here, and you heard what FCC is doing. We’re working with some other parts of government as well.

In the end the goal is, we want our research to help the government, to be included in operational environments. Right now the best I can give you is that we have FCC signed up and interested, but longer term we expect to have more.

20. Can we get access to S&T network outage data for training the predictive models?

Currently the government is not able to provide outage data specific to government networks, other than what you may have heard from Henning or what he may have available with regard to the FCC. Within DHS S&T we don’t have any outage data to make available at the moment. I would like to, but we are not there yet. Therefore, the PARADINE BAA call paragraph 3.1.2 was amended to explicitly indicate that DHS S&T data will not be provided.

21. Is the BAA call appropriate only for research that has already been done, and is ready to be developed?

The PARADINE BAA call was released because DHS does not believe the solutions to meet the PARADINE BAA call goals exists. As I mentioned in my remarks, we are not funding basic research and initial concept development. We are expecting you to have some level of maturity in what you are bringing to the table. We are willing to fund development and even some amount of research, but we are not going to fund, in DOD world, 6.1 research. Our expectation is at best it’s 6.2, and even into 6.3.

22. The BAA call asks for data sharing and open license, doesn’t that contradict commercialization efforts?

In the case of data sharing it’s our view that it’s good for the community, it’s the type of thing that we like to fund. Data sharing, in my mind, is an expected piece, realizing that in the BAA what we’ve said about data is that we encourage you to use open source. If you have proprietary data you are free to use it, we obviously wouldn’t force you to share proprietary data, but we encourage you, where appropriate, to do so. In the case of open licenses, the way we’ve seen it work and encouraged it with past performers, is even if you do open licenses on your technology you can take your technology and provide it as a service, even though you open source the data associated with it. We don’t believe it contradicts the commercialization, that’s up to you to explain to us how you propose doing that, you are free to convince us. Keep in mind when you do a proposal you also have to include what is called a section K which talks about the intellectual property rights that you are coming to the table with as well as what you are offering to the government as part of the proposal, and that is standard in government contracts.

This relates to overarching BAA HSHQDC-17-R-B0002, paragraph 9.6.1, u and paragraph 12.6.

23. Can an organization submit a single proposal covering both TTAs?

Yes. See PARADINE BAA call paragraph 6.2.2.

24. Can an organization submit separate proposals to the two TTAs?

Yes. See PARADINE BAA call paragraph 6.2.3. The two TTAs are fairly closely linked. We certainly believe you could submit separate proposals, one for TTA 1 and TTA 2, but you are going to have to make a pretty good story about how those are going to tie together. Our view is that a better proposal is one that addresses both TTAs. It is up to you as the offeror how you want to proceed with your offering. You can submit something that only addresses one TTA. I think that’s a bit incomplete, but that is up to you to determine.

25. Do you care about the disruptive events more than the threats?

We are pretty focused on the disruptive events piece. There are a limited number of threats that lead us to network and internet scale disruptive events. Henning talked about them as well. Things like Mother Nature, Katrina, Sandy, those kinds of things, cable cuts, things like that. It’s not an unlimited number of threats. Our focus is on the disruptive events, because when you walk backwards from the disruptive events you come back to a set of a very small number of threats to begin with. So we are not focusing on the threats. What we did say in the BAA is that there are use cases, which we do expect you to think about and talk about, and to line your proposal up with at least two (see PARADINE BAA call paragraph 6.5.1) of those use cases to demonstrate that your technology capability has a broader applicability than just a single topic or single focus area.

26. Should proposals focus on a specific network or internet disruptive event, or a class of NIDEs?

Broader is better, if it’s specific to a single NIDE we do still ask that you talk about how it applies to at least two use cases (see PARADINE BAA call paragraph 6.5.1), and we’ve given you some samples. That is not an unlimited amount of use cases, you can come up with some of your own, but our view is that broader is better.

27. Should the basis of PARADINE research be publically available data only?

The PARADINE BAA call does not prescribe any data source for use in meeting its goals and paragraph 3.2.1 states: “Note: Proprietary data sources may be used if available.” However, open data and open sharing of data amongst the community gets us all further, but if you have access to proprietary data and that helps you produce a better product and gives you the special sauce that perhaps someone else doesn’t have, that is perfectly fine with us. Proprietary data, provided it ties in nicely with your product - we will be glad to work with you on that as well.

28. Global access network, is there any way to obtain a device that allows network capabilities anywhere without global monthly data limits?

The government is not going to provide devices. There will be no government furnished equipment.

29. When do you anticipate releasing the BAA solicitation for BAA #2 and BAA #3?

The idea here between the three BAAs is to give the PARADINE BAA call a little bit of a head start. Our expectation is that BAA #2 will be within 6 – 9 months. By then we will have been able to announce awards, so that people who want to respond to BAA #2 will know who got awards for the PARADINE BAA call. As many of you know, we do public announcement of all awards, so there are press releases and such available off of our web site. So when we go to BAA #2, the PARADINE BAA call awards will have been awarded and there will have been some initial work completed. Just as a consequence of time we did not do white papers for the PARADINE BAA call, but our intent is to do white papers for BAA #2 and BAA #3. BAA #3 we expect to be another 6 – 9 months after BAA #2.

This whole process we are expecting to take 21 – 24 months, depending on how that time rolls out.

QUESTIONS AND ANSWERS PROVIDED BELOW ARE NOT PERTINENT TO

THE PARADINE BAA CALL BUT WERE DISCUSSED IN THE CONTEXT OF

DHS ACTIVITIES

30. Is the Silicon Valley program only available to start-ups and small companies?

Yes, it depends on your definition of start-up and small company. If you look at the Department of Defense, because we are using Other Transaction vehicle, there are some classifications that go along with other transactions. We have extended that and what we are saying is that we are looking at start-up companies and the going number in the market is if you are over a hundred employees you are no longer a start-up, so we are looking at companies that are less than a hundred employees, there’s also a constraint that you can’t have had government contracts totaling over a million dollars within the last twelve months, so that also puts you in some interesting places with a small business. The companies could have had a Small Business Innovate Research (SBIR) contract and still compete for Silicon Valley in that process.

31. Are there separate programs for Universities, or is it integrated into the project?

Within the Directorate of Science and Technology there are other things going on.

For example there is the Office of University Programs, they have ten specific Centers of Excellence, all of those centers have multiple universities participating and they put out their solicitations every year, not just within the current Centers of Excellence, but to the world, and each year they drop some people off and they add some new people on. So if you are a university and you want to participate there are other opportunities. We have one of the Centers of Excellence, it’s called the “Critical Infrastructure Resiliency Institute (CIRI) and the lead organization is the University of Illinois, but again they drop universities and add universities each year and new projects. If you look at our portfolio, it’s probably about 40% funded with university performers, about 50% with industry, and about 10% with national labs and other government agencies. But universities can participate equally and fairly.