STAMP_Industry_Day_QA.pdf

2015 DHS S&T Cyber Security Division Static Tool Analysis Modernization Project (STAMP)

Questions and Answers

The below are responses to questions received both at the December 8, 2015 Industry Day and in the BAA mailbox as of January 15, 2016.

1. Are the funding levels described, the total available funds or the expected award sizes?

Response: See updated diagram – Figure 3 and Figure 4 of the STAMP solicitation, HSHQDC- 16-R-B0002. Also, see Section 6.5.1 of the STAMP solicitation, HSHQDC-16-R-B0002, for proposal funding and ceiling requirements.

2. Is the $10M available, the full scope of the program or the amount estimated for each performer team?

Response: See Section 6.5.1 of the STAMP solicitation, HSHQDC-16-R-B0002, for proposal funding and ceiling requirements which specifically states the solicitation identified funding profile is to be followed for each proposal submitted.

3. How many awards are anticipated?

Response: See Section 6.2.1 of the STAMP solicitation, HSHQDC-16-R-B0002.

4. Is the notional funding profile, the funding anticipated for each awarded TTA team or is it the sum of the funding for all awarded teams?

Response: As described in Section 6.5.1 of the STAMP solicitation, HSHQDC-16-R-B0002, the funding profile and ceiling relates to an anticipated award versus a summation of all awards.

5. Under what solicitation do we apply if we have a prototype, scalable, static analysis engine that has no false positives or false negatives and therefore no residual risk? We have a revolutionary leap ahead approach that we can demonstrate but needs R&D to extend to all languages, address all CWEs etc. Where do we apply for money?

Response: Current DHS S&T BAA solicitations are posted to FedBizOps and are also listed at https://baa2.st.dhs.gov/portal/public/processRequest?eurl=AAAAAAEytBoAAAFSMU6wLAA UQUVTL0NCQy9QS0NTNVBhZGRpbmcAgAAQABAAAQIDBAUGBwgJCgsMDQ4PAAA AUMUP8ssYOu8SxeEfopmq%2F3JxVJpIrI76XzFWBZiTRiyC3RqKuCT7PXvTogAGIO1DQ Hzv0x49kTffIsSS5X%2F3P9hovmbpbs7UHZYN5G1zjpM9ABTMvaS6WMYbGGGcBE3rOxT VJ9g3IQ%3D%3D. Where to propose is at your discretion.

6. Is TTA 3 (Modernization of tools) intended to support vendors? (STAMP)

Response: TTA3 and all other parts of this BAA solicitation/call are intended for capable offerors be they vendors or other types of entities.

https://baa2.st.dhs.gov/portal/public/processRequest?eurl=AAAAAAEytBoAAAFSMU6wLAAUQUVTL0NCQy9QS0NTNVBhZGRpbmcAgAAQABAAAQIDBAUGBwgJCgsMDQ4PAAAAUMUP8ssYOu8SxeEfopmq%2F3JxVJpIrI76XzFWBZiTRiyC3RqKuCT7PXvTogAGIO1DQHzv0x49kTffIsSS5X%2F3P9hovmbpbs7UHZYN5G1zjpM9ABTMvaS6WMYbGGGcBE3rOxTVJ9g3IQ%3D%3D https://baa2.st.dhs.gov/portal/public/processRequest?eurl=AAAAAAEytBoAAAFSMU6wLAAUQUVTL0NCQy9QS0NTNVBhZGRpbmcAgAAQABAAAQIDBAUGBwgJCgsMDQ4PAAAAUMUP8ssYOu8SxeEfopmq%2F3JxVJpIrI76XzFWBZiTRiyC3RqKuCT7PXvTogAGIO1DQHzv0x49kTffIsSS5X%2F3P9hovmbpbs7UHZYN5G1zjpM9ABTMvaS6WMYbGGGcBE3rOxTVJ9g3IQ%3D%3D https://baa2.st.dhs.gov/portal/public/processRequest?eurl=AAAAAAEytBoAAAFSMU6wLAAUQUVTL0NCQy9QS0NTNVBhZGRpbmcAgAAQABAAAQIDBAUGBwgJCgsMDQ4PAAAAUMUP8ssYOu8SxeEfopmq%2F3JxVJpIrI76XzFWBZiTRiyC3RqKuCT7PXvTogAGIO1DQHzv0x49kTffIsSS5X%2F3P9hovmbpbs7UHZYN5G1zjpM9ABTMvaS6WMYbGGGcBE3rOxTVJ9g3IQ%3D%3D https://baa2.st.dhs.gov/portal/public/processRequest?eurl=AAAAAAEytBoAAAFSMU6wLAAUQUVTL0NCQy9QS0NTNVBhZGRpbmcAgAAQABAAAQIDBAUGBwgJCgsMDQ4PAAAAUMUP8ssYOu8SxeEfopmq%2F3JxVJpIrI76XzFWBZiTRiyC3RqKuCT7PXvTogAGIO1DQHzv0x49kTffIsSS5X%2F3P9hovmbpbs7UHZYN5G1zjpM9ABTMvaS6WMYbGGGcBE3rOxTVJ9g3IQ%3D%3D https://baa2.st.dhs.gov/portal/public/processRequest?eurl=AAAAAAEytBoAAAFSMU6wLAAUQUVTL0NCQy9QS0NTNVBhZGRpbmcAgAAQABAAAQIDBAUGBwgJCgsMDQ4PAAAAUMUP8ssYOu8SxeEfopmq%2F3JxVJpIrI76XzFWBZiTRiyC3RqKuCT7PXvTogAGIO1DQHzv0x49kTffIsSS5X%2F3P9hovmbpbs7UHZYN5G1zjpM9ABTMvaS6WMYbGGGcBE3rOxTVJ9g3IQ%3D%3D

7. Are there security vulnerabilities in dynamic languages? (STAMP)

Response: Yes.

8. How will you solicit participation from COTS providers for TTA2 in particular? I.e. COTS vendors of static analysis tools (STAMP).

Response: There are not any provisions of the BAA that are dedicated to soliciting participation from COTS providers specifically. However, COTS providers are encouraged to submit proposals.

9. The STAMP solicitation envisions that during Phase 1, TTA#2 will assess existing tools and prototypes, selecting the best candidates for modernization in Phase 2. The best choice of teammates to modernize a given tool might well be its current developers. However, since the decisions on tools to be modernized are not made until near the end of Phase 1, it would appear that firm decisions on team participation cannot be made until then either. Does DHS anticipate proposals containing teaming structures with tentative participation, depending on the outcome of Phase 1?

Response: DHS expects proposals to include a technical approach that details how an offeror will meet all BAA requirements to include teaming, if an offeror chooses to do so. Where teaming is proposed it will need to be explicit to support both the proposal evaluation process as well as a potential award if the proposal were to be selected.

10. Similarly, does DHS also anticipate that specific technical approaches described for TTA#3, say for a specific programming language and tool, would be tentative as well, depending on the eventual choice of tools to be modernized?

Response: No. Remember the contracting process will not include indeterminate terms.

11. In setting up my organizations proposal to the STAMP BAA, (HSHQDC-16-R-B0002) the portal does not allow me to add a proposal to my worksheet without choosing one of the TTA's.

However, the solicitation requires the proposal to cover all TTA's. I contacted the DHS portal help desk and they suggested I contact this e-mail address to ask how to propose to all TTA's instead of just one. How should I proceed?

Response: Proposals should cover all technical topic areas and phases. Also, the BAA portal was updated to support accepting proposals for all TTAs after the initial posting of the STAMP BAA call.

12. BAA paragraph 9.6.1.q states “Provide a list of any current or pending awards or proposals with DHS that pertain to this work, submitted either as a prime contractor, subcontractor/consultant, or teaming partner.” Do we need to provide this information for the prime offeror only, or for subcontractors and teaming partners also?

Response: This requirement pertains to all entities that are on an offeror team.

13. TTA 3, Option Period 1 Key Deliverable #1 requires a technical report on a “modernization framework for candidate tools”, and TTA 3 Option Period 1 Key Deliverable #3 says “Develop Modernization framework”. What is the government’s evaluation criteria for an acceptable “modernization framework”? What form is the “Develop Modernization framework” deliverable expected to take?

Response: The evaluation criteria are published in BAA HSHQDC-14-R-B0005 11.2 Evaluation Criteria. Related to Section 3.3.1, of the STAMP solicitation, HSHQDC-16-R-B0002, the modernization framework proposed should “encompass new techniques, methods, services and capabilities in the candidate tools.” Presentation of the modernization framework is at the discretion. In this case “framework” is unique to an offeror’s technical approach.

14. TTA 3 Option Period 1 Key Deliverable #5 is “Analysis report of Common Weakness Enumeration (CWE) coverage and mappings to NIST SP 800-53A, Version 1)”, and TTA 3 Option Period 2 Key Deliverable #4 is similar. TTA #3 makes no mention of NIST SP 800-53A.

Are these deliverables supposed to apply to TTA #4? If so, what are their due dates? If these deliverables are expected for TTA 3, please describe the TTA 3 activity to which they apply.

Response: There are three versions of the “Analysis report of Common Weakness Enumeration (CWE) coverage and mappings to NIST SP 800-53A” deliverable required (see Section 4.4 and Section 4.5 of the STAMP solicitation, HSHQDC-16-R-B0002) each is to be reflective of greater maturity with the last to incorporate results from the TTA4 pilot.

15. TTA 3 Option Period 1 Key Deliverable #9 is “Scoring and Benchmarking Tool Version 1”, and TTA 3 Option Period 2 Key Deliverable #7 is also “Scoring and Benchmarking Tool Version 1”. However, the development of a scoring and benchmarking tool is identified as an objective for TTA 4. Are these deliverables supposed to apply to TTA #4? If so, what are their due dates? If these deliverables are expected for TTA 3, please describe the TTA 3 activity to which they apply.

Response: STAMP solicitation, HSHQDC-16-R-B0002, Section 4.4 and Section 4.5 version numbers for the “Scoring and Benchmarking Tool” have been updated in the BAA call via amendment. So, now there are three versions of the “Scoring and Benchmarking Tool” deliverable required (Section 4.4 and Section 4.5 of the STAMP solicitation, HSHQDC-16-R- B0002, each is to be reflective of greater maturity with the last to incorporate results from the TTA4 pilot.

16. TTA 4 specifies the need for an operational pilot evaluation. Is this an operational pilot evaluation of the Tool Scoring and Labeling framework, or of the modernized static analysis tools from TTA3?

Response: As indicated in Section 3.4 of the STAMP solicitation, HSHQDC-16-R-B0002, the pilot is intended to be a comprehensive mechanism “to support transition of the STAMP products.”

17. In Section 3 of the STAMP solicitation, HSHQDC-16-R-B0002, in the bulleted list of STAMP goals is the item “Explore innovation and deliver new techniques and capabilities for vetting mobile applications”. Mobile applications are not mentioned elsewhere in the RFP. Are mobile applications required to be a part of TTAs 1-4?

Response: No; mobile applications not required. The BAA call has been amended to reflect this change.

18. In Section 3 in the bulleted list of STAMP goals is the item “Develop a robust scanning engine that can scale large and complex code bases”. The development of a scalable scanning engine is not mentioned elsewhere in the proposal. Is a scalable scanning engine required to be developed, or is one of the ways in which a candidate tool might be improved?

Response: A scalable scanning engine is not required; however an offeror could propose to use a scalable scanning engine to improve, but technical approach is an offeror decision.

19. TTA 1 Option Period 1 Deliverable #3 – Test Case Generator Prototype is due 6 months after award of option period 1. TTA 2 Base Period deliverables 2, 3, 4 and Option Period deliverable have due dates are incongruous with using the TTA1 Test Case Generator Prototype in the tool evaluations. Can you please clarify these due dates and the expected use of the Test Cases from TTA1 in the TTA2 deliverables?

Response: The due date requirements are as identified.

20. What amount (if any) should be used to estimate registration fees for PI meetings in the travel breakdown for the cost proposal?

Response: Fees for PI meetings are not anticipated in STAMP solicitation, HSHQDC-16-R- B0002, Section 6.6, for Travel requirements and costs should be proposed based on travel origin and destinations.

21. Section 4.4 of the solicitation calls for a "Scoring and Benchmarking Tool Version 1" deliverable in both Option Period 1 and Option Period 2. Should these be renamed to "Scoring and Benchmarking Tool Version 1" and "Scoring and Benchmarking Tool Version 2" (subsequently renaming the one in section 4.5 as Version 3) or is the first mention of "Scoring and Benchmarking Tool Version 1" an error and should be removed?

Response: The version numbers for the “Scoring and Benchmarking Tool” in Section 4.4 and Section 4.5 of the STAMP solicitation, HSHQDC-16-R-B0002, have been updated in the BAA call via amendment.

22. Please clarify the timeline of the deliverables.

a. The diagrams and the discussion at Industry Day state a PoP for TTA1 of 1.5 years.

However, Section 4.2 lists a TTA1 deliverable at 8 months in the Option 1 year (that is at year 1.66).

Response: The STAMP solicitation, HSHQDC-16-R-B0002, has been amended to change the delivery requirement for the “Package of data sets for test cases” deliverable to 6 months after award of option period 1.

b. The diagrams in HSHQDC-16-R-B0002 show a period of performance for TTA3 from year

1.5 through year 3 of the project. However the deliverables call for two reports at 1.5 years (which would be the start of TTA3) and initial versions of modernized tools at 2.0 years, just 6 months into TTA3.

Response: The deliverable requirements are correct.

c. The STAMP solicitation gives a funding profile of $0.5M, 1.5M, 3.5M, and 2.5M for the 4 years of the project. The Q&A session at the Industry Day stated (for a given proposal): $2M for Phase I (first 1.5 years); $3.5M for Phase 2 (next 1.5 years), and 2.5M for the last year. Please clarify, by an amendment if needed, whether the cost proposal is to be broken down by Phase or by Year.

Response: The cost proposal should be broken down by year. See BAA HSHQDC-14-R-B0005, Section 4.5, “Anticipated Period of Performance for Individual Awards”.

d. Are the years calendar years, Gov't fiscal years, or years from project start?

Response: The years depicted in Figure 4 of the STAMP solicitation, HSHQDC-16-R-B0002, are based on date of award.

23. Based on government fiscal years, if the cost proposal is by year, then please clarify how the $2M for Phase I and the $3.5M for Phase 2 is to be divided across the 3 years.

Response: Cost should be proposed by year not phase. See BAA HSHQDC-14-R-B0005, Section 4.5 “Anticipated Period of Performance for Individual Awards”.

24. Are you looking for us to modernize our own tools or “community” tools?

Response: From Section 6.8 of the STAMP solicitation, HSHQDC-16-R-B0002, “…a goal of this BAA solicitation/call (HSHQDC-16-R-B0002) is to develop solutions that are mature enough for deployment or integration into an existing enterprise, the work proposed should be innovative and provide a capability not currently available in the market.” The approach is at offeror’s discretion.

25. Should a proposal target all TTAs?

Response: Yes. See Section 6.2.3 of the STAMP solicitation, HSHQDC-16-R-B0002, “Offerors may provide multiple proposal submissions; however, each submission must be distinct and self-contained without any dependencies on other work of any kind, while providing an approach to meet all of the TTA objectives for every TTA.”

26. What is the contract type?

Response: See BAA HSHQDC-14-R-B0005, Section 4.3 “Anticipated Award Types.”

27. Is test case generation an automation or a methodology for developing/defining test cases?

Response: The “Test Case Generator Prototype” deliverable is intended to automate the “Testcase Development Methodology.”

28. Can you do dynamic testing as well as static?

Response: The scope of the STAMP BAA call is static testing.

29. Must the test generator/test suite be based on Juliet?

Response: From Section 3.1.2, Objective 2, of the STAMP solicitation, HSHQDC-16-R-B0002, “this objective is intended to enhance the capabilities of the Juliet test suite.”

30. How much more complex than the Juliet test cases must the new test cases be?

Response: Technical approach is an offeror’s discretion.

31. Are we interested in new capability that shows functionality that currently does not exist?

Response: From Section 6.8, of the STAMP solicitation, HSHQDC-16-R-B0002, “…a goal of this BAA solicitation/call (HSHQDC-16-R-B0002) is to develop solutions that are mature enough for deployment or integration into an existing enterprise, the work proposed should be innovative and provide a capability not currently available in the market.”

31. Can proposals include all topic areas and just not a single TTA?

Response: See Section 6.2.3 of the STAMP solicitation, HSHQDC-16-R-B0002, “Offerors may provide multiple proposal submissions; however, each submission must be distinct and self-contained without any dependencies on other work of any kind, while providing an approach to meet all of the TTA objectives for every TTA.”

32. Are you strictly focused on tools to detect flaws? Or, are tools used as part of the development process of interest?

Response: Per Section 3.2, of the STAMP solicitation, HSHQDC-16-R-B0002, the STAMP BAA seeks to “…identify the gaps in tool capabilities” that “factors in a comparison of current capabilities with the best capabilities of the state-of-the-art.” Both, we are aimed at fixing some of the tools and improving the development process as part of that activity.

33. What is the definition of modernization?

Response: See Section 3.2 of the STAMP solicitation, HSHQDC-16-R-B0002.

34. Are you expecting a full test suite to be delivered for all relevant languages under TTA1?

Response: Offerors will be expected to choose the languages to use and scope the test suite for

TTA#1.

35. What are your views on static analysis on continuous integration servers after check-in/review?

Response: The STAMP BAA call has been amended to include a discussion of continuous integration in the Go/No-Go metrics. See 6.8.3.2 Go/No-Go #2.

36. Are we going to define the candidate tools?

Response: Proposals are required to respond to Section 3.2.1 of the STAMP solicitation, HSHQDC-16-R-B0002, which states, “3.2.1 Objective 1 - Identification and selection of a list of ten (10) candidate tools to modernize as part of STAMP, which may include free and open-source software analysis tools, as well as commercial tools.”

37. What job are you expecting static analysis tools to do? Are we worried about tools for developer training? Are we worried about productivity when writing secure code? Are we trying to get to a level of higher confidence that the code is secure?

Response: Refer to Section 3 of the STAMP solicitation, HSHQDC-16-R-B0002. STAMP solicitation HSHQDC-16-R-B0002 is expecting to develop static tools to provide coverage for the test cases developed for TTA1.

38. Is the expectation to only enhance free and open source tools or to enhance commercial tools as well? If yes, the commercial, how does this work

Response: See Section 3.2.1 of the STAMP solicitation, HSHQDC-16-R-B0002, which states, “3.2.1 Objective 1 - Identification and selection of a list of ten (10) candidate tools to modernize as part of STAMP, which may include free and open-source software analysis tools, as well as commercial tools.”

39. Your diagram shows TTA2 and TTA3 in a feedback loop (Page 3). TTA1 and TTA3 could also benefit from such feedback. But the TTA Phase and funding diagram appears to preclude TTA1, 2 continuing throughout the 4 years? (See updated Figure 1)

Response: The STAMP BAA call has been amended to update the Option Period 2 deliverables to include “Package of data sets for test cases – Update” and “Technical report on overlapping tool coverage and complimentary tool coverage – Update” to address this question. These deliverables will facilitate the feedback depicted in Figure 1.

40. What is the commercial/ industry application for STAMP?

Response: See Section 6.8.4 of the STAMP solicitation, HSHQDC-16-R-B0002, in particular the bullet mentioning “Transition Plan”. Addressing this requirement is at the offeror’s discretion.

41. What ideas do you have about doing software application security testing at speed for critical infrastructure and others?

Response: Critical infrastructure is not mentioned in this BAA call.

42. Do we have to address all objectives in each technical area? Should we have to address all programming languages? Can we include dynamic analysis tools as part of STAMP?

Response: See Section 6.2.3 of the STAMP solicitation, HSHQDC-16-R-B0002.

43. What languages is STAMP targeting?

Response: Section 3.1.2 of the STAMP solicitation, HSHQDC-16-R-B0002, includes a mention of things to consider to prioritize language choices.

44. Can university and non-profits lead any of these efforts?

Response: Yes. HSHQDC-14-R-B0005, Section 5, “ELIGIBILITY INFORMATION.”

45. Is it possible to get a conflict of interest (COI) determination in advance of submitting a proposal?

Response: No.

46. When you look at how effective static analysis tools are, will you look at how effectively developers can use them?

Response: Pertaining to tools, innovative approaches to TTA2 are sought. See TTA2 objectives.

47. How accurately and efficiently can they triage warnings and correctly fix problems?

Response: Warning handling is a factor that could be addressed by technical approaches to TTA 2 and/or TTA3.

48. How much detail do you have to provide in Phase 1, 2, and 3? Also, how/when can you include COTS vendors of some of the tools?

Response: This question pertains to technical approach and level of detail, which is at the discretion of the offeror.

49. There are numerous important parts such as decision recall to determine whether a tool is useful; there are several other factors as well such as scalability, performance, incrementality, extensibility, usability, etc. How much weight should these other factors be given as part of the proposal?

Response: As a FAR Part 35 solicitation, the STAMP solicitation, HSHQDC-16-R-B0002, solicits solutions to the problems described in the BAA while meeting the BAA requirements and objectives. From Section 6.8 “…a goal of this BAA solicitation/call (HSHQDC-16-R- B0002) is to develop solutions that are mature enough for deployment or integration into an existing enterprise, the work proposed should be innovative and provide a capability not currently available in the market” it is incumbent on the offeror to propose to accomplish this goal. Offeror proposals may make value judgements regarding factors to address in technical approaches when proposing.

50. If an Organizational Conflict of Interest exists, will DHS accept a mitigation plan?

Response: Refer to Section 12.4 of BAA HSHQDC-14-R-B0005, which is the overarching BAA that provides terms and conditions for all solicitations/calls issued against this BAA. Section

12.4 provides specific information regarding handling of any conflict of interest.

51. “DHS has a strong preference for open source licensing of software for all software developed and delivered …”

a. Is the intent of the open source licensing is to ensure that there are no required licensing costs associated with the STAMP solution and that other researchers or vendors can easily expand the system’s capabilities because the source code is available?

Response: Yes

b. Is it acceptable to DHS to have free versions of software (that have not been open-sourced) used in the Go No-Go demonstrations and in the delivered STAMP system? For example, there are several tools that are or could be free but not open source which could be useful to incorporate into both the demonstrations and delivered system.

Response: The technical approach to meet the goals and requirements of the STAMP BAA are at the discretion of the proposer, but the requirements in BAA HSHQDC-14-R-B0005 paragraph

9.6.1 u (Assertion of Data Rights), and 9.6.1 g (2) are required. These BAA requirements are particularly important should an offeror propose any deliverable that the Government would not receive unlimited rights to.