Industry_Day_Questions_and_Answers.pdf

NASA Open Innovation Services 2 (NOIS2) Industry Day Questions and Answers

1. Can you walk us through an example of how you have worked with vendors in a previous challenge?

Once a vendor is selected for task order award based on their proposed and winning plan, the first step is a kick-off meeting with the vendor points of contact, the challenge owner or team, and a Center of Excellence for Collaborative Innovation (CoECI) representative.

Once the task order is awarded, the vendor serves as the project lead and is responsible for challenge execution. The challenge owner provides ongoing support of the technical requirements of the challenge. The CoECI representative serves to ensure effective communication and understanding of the process given many challenge owners may be new to the use of crowdsourcing.

2. What are three main factors that you think will make a successful bid on these upcoming [Request for Proposals] (RFPs)?

It is important to read the RFP in its entirety and follow the directions outlined within it. For a proposal to be considered timely, it must be submitted by the due date and time as stated in the RFP. The draft RFP (if one is released) will contain evaluation criteria as well as instructions to offerors. Potential offerors are encouraged to pay particular attention to those sections of the RFP. NASA anticipates hosting a pre-proposal conference either before or after the final RFP is released. The purpose of a pre-proposal conference is to help industry understand the Government's requirements, provide a high level overview of the RFP, and collect questions from prospective offerors at that time.

3. Will the RFP, when issued, have a mock call order, or order, to give offerors a sense of the technical crowdsourcing activities required to satisfy the Statement of Work

(SOW)?

The NOIS2 procurement strategy is currently in the development phase, therefore NASA cannot discuss this information at this time. Please continue to monitor FedBizOpps (FBO) and the NOIS2 website for additional information.

4. Does CoECI have a maximum number of vendors that can be or will be awarded this

Contract?

The NOIS2 procurement strategy is currently in the development phase, therefore NASA cannot discuss this information at this time. Please continue to monitor FBO and the NOIS2 website for additional information

5. I know that you don’t have insight into the future, but how much do you think healthcare innovation will be part of the problems you are working to solve moving forward?

CoECI lives within the Human Health and Performance directorate at NASA Johnson Space Center. Many of the early challenges were actually focused on human health in spaceflight.

The CoECI website has previous challenges posted and categorized by type. Though no predictor of the future, NASA encourages interested parties to review historical challenges.

6. Historically are there any restrictions and/or conditions or limitations regarding the kind of crowd participation (e.g., US citizens only etc.).

In addition to supporting NASA internal organizations, CoECI provides support to other federal agencies, and different legal authorities are utilized among different agencies.

Participation is dependent upon the legal authority the competition is being conducted under and will be specified on a TO basis.

7. Are there common pitfalls that we should avoid?

There is not enough specificity about what this Question is regarding for the Government to answer.

8. What is the anticipated contract amount or ceiling price?

The NOIS2 procurement strategy is currently in the development phase, therefore the Government cannot discuss this information at this time. Please continue to monitor FBO and the NOIS2 website for additional information

9. What have you learned from your previous innovation challenges?

The CoECI team has learned that innovation challenges work. More specifically, the focus is on obtaining a solution that can be put into operational use. The CoECI team has thus established this crowdsourcing methodology as a part of the requirements development process to ensure the challenge owner has an infusion path for the solution. Something else that the team has learned is that no two challenges are the same. There is consistency to the methodology, but with each effort the team has conducted, something about the challenge itself has been unique.

10. What is most interesting to you moving forward in terms of successfully innovating?

The CoECI team is interested in expanding the use of crowdsourcing. One objective of NOIS2 is to expand the use of crowdsourcing to different methodologies in addition to challenges.

11. How much do you think live events or collaborations with academic institutions enhance traditional innovation challenges? Do you anticipate this need moving forward?

Our research and collaborations with existing academic institutions are important to the current NTL operating model. For example, in reference to the NOIS2 Industry Day chart 14, Harvard University is listed within the current CoECI toolkit. CoECI has an existing cooperative agreement with the Laboratory for Innovation Science at Harvard (LISH) who continue to research the use of crowdsourcing and providing data science expertise.

Academic collaborations are a part of the current NTL operational model.

12. Can you discuss what to expect in the one-on-one meetings? Is there anything in particular that you would like the companies to have available for review?

The Industry Day One-on One meetings are an opportunity for industry to share their companies’ capabilities and to ask questions in a more intimate setting. Please note that all questions that are asked during the one-on-one sessions will be posted publicly to FedBizOps. Each one-on-one session will not exceed 30 minutes in duration.

13. Are any of the current vendors automatically being included in the new contract?

No, current vendors will not be automatically included in the new contract. All interested parties submitting a proposal on time and in compliance with the RFP requirements will be considered for award.

14. When you post challenges, it works best to have all challenges within one system.

Will you only choose one innovation or idea management system vendor?

a. As stated in the Industry Day charts, the anticipated contract type is multiple-award IDIQ which means the government intends to award contracts to multiple offerors.

b. Please review the current process on the NOIS contract, found on chart 19 of the NOIS2 Industry Day charts. In a multiple award contract environment, each task order is competed among the contract incumbents. Once a challenge becomes available, the CoECI team develops a challenge RFP for the proposed Task Order (TO). Each contract holder then has the ability to submit a proposal. Upon the submission deadline, the CoECI team evaluates each proposal and selects the winning vendor. Upon vendor selection and announcement, the selected vendor executes the Task Order.

15. Would you explain what is meant by multimedia within the candidate deliverables? Is this crowdsourcing and executing competitions or challenges for multimedia-type projects?

The multimedia candidate deliverable listed in the NOIS2 Draft SOW is for deliverables to be developed through the use of a crowdsourcing methodology. As the Draft SOW stipulates this is crowdsourcing, competitions and/or challenges for multimedia such as graphic art, logos, animations, video, storyboards, voiceover, music, or writing/copy.

16. Can you give us a sense of the breadth of the questions that are posed in the challenges? Industry? Type of question? Level of question? Advancement of the question? Do you need the science team who would come in and teach the innovation?

NASA encourages all interested companies to visit the CoECI website to explore the current and past challenges. Also please look at the segment of the website dedicated to other agency challenges. A review of the past challenges should provide the breadth of the scope considered. However, the past is not a perfect predictor of future requirements.

17. Does the CoECI website have copies of actual IDIQ requirement solicitations? They would be helpful to review.

The Government is evaluating historical task order RFPs for posting. If any are posted, they will be located in the technical library which can be accessed via the NOIS2 website and

FBO.

18. What will primarily stay the same from the original NOIS contract? Is anything envisioned differently?

Generally speaking, CoECI is hoping to find new approaches to getting value from crowds.

Prizes and challenges have been very successful under the current NOIS contract.

19. Are there specific lessons or advice from NOIS1?

CoECI learned that crowdsourcing works. Federal agencies have the opportunity to stay on the cutting edge and to foster advancement of their missions with the use of crowdsourcing.

CoECI is working to keep users educated on how to most effectively utilize this tool.

20. Can you share what your typical process looks like?

Please see answer under 14b above.

21. Is there any desire to have any use of open source to add more impact?

CoECI has utilized open source solutions in the past and is open to all models that provide value from crowdsourcing.

22. Since the beginning of NOIS, much has been happening in the area of algorithmic responsibility and data ethics. Have there been any considerations in this area in what you’ve seen so far?

The only considerations to date have been in the area of de-identification where CoECI actually launched a challenge focused on that problem space. CoECI is also aware of data bias in machine learning. The strategy in both of these instances was to be aware and stay ahead of it.

23. Have there been any trends in data science activities for your organization?

CoECI has seen activity and growth in data science across NASA and the rest of the federal government. CoECI has had great success with data science related crowdsourcing challenges and that is helping to raise awareness (and demand) of what open innovation can bring to the domain of data science.

24. What is the duration of some of the projects in your past? In 2020, the NOIS2 vendors will be chosen but what are the timelines from there?

All of the task orders and challenges are unique and their length varies. Average challenge length from formulation to solution is approximately 6-7 months, but challenge timelines can range up to multiple years depending on the challenge and how many stages it has.

25. Will deliverables be specifically defined or will there be any flexibility in the deliverables?

CoECI’s experience to date has been in the use of crowdsourcing in the form of public competitions. The NOIS2 Draft SOW focuses on the end-products NASA is seeking. CoECI is interested in determining how crowdsourcing can be used to deliver end products; not by defining the methodology, but instead by relying on vendors to supply the methods that they believe will be most impactful.

26. Will there be a NASA sponsor involved with each challenge?

There is always a challenge owner (sponsor). Please note that may be someone from NASA or from another federal agency.

27. How is NOIS2 different than NOIS1? Are there specific elements that we found lacking in the first that we are looking to augment or offset? Any limitations or problems from

NOIS1?

The primary change from NOIS1 to NOIS2 is the expansion of the SOW to different crowdsourcing methodologies beyond public competitions. Please review the NOIS1 and NOIS2 SOWs, both available on FBO and NOIS2 website, for a complete accounting of the differences. Also note, the posted NOIS2 SOW is a draft and contains pre-decisional information that is subject to change.

28. Do you foresee, or has there been any indication, that the items in section 3 crowd formulation will be used more than it has in the past?

Please see answer to Question 25.

29. For NOIS1, some contractors were able to join up to help each other when bidding for task orders. Will this be allowed for NOIS2?

This information is pre-decisional.

30. Are there different solicitations for micro-purchases? We have the same question for global technology search.

NASA-JSC does not currently have open solicitations for micro-purchases or global technology search. Please monitor FBO for any solicitations that may be issued in the future.

31. For the requests that come through the NOIS program are any of them in support of

Other Transactional Authorities (OTA’s).

All the work that CoECI has done under NOIS has been through Procurement, COMPETES, and/or NASA’s specific challenge authority. NASA’s OTA is tied specifically to the Space Act itself. The only challenge program NASA has active under OTA is the Space Apps challenge.

32. If we are on the contract, can we also be a participant on challenges?

More specificity is required for this Question.

33. After selecting a vendor, can the vendor be part of the solution development, i.e., a participant in the challenge?

This information is pre-decisional and the Government is unable to provide a response at this time.

34. What is the reporting process?

Typical reporting may include the number of participants, start and stop dates, prize amounts, and other basic statistics about the effort. These requirements may change as the NOIS2 SOW is still pre-decisional.

35. In regards to IP is there a particular process that we need to follow?

Intellectual Property (IP) requirements are anticipated to vary on a task order basis, however IP handling requirements are included in the NOIS2 SOW. This SOW is still under development and any information is pre-decisional and subject to change.

36. Regarding the reward amount for prizes, do you provide that or do you ask the vendor to do it?

The approach for determination of prize amounts is anticipated to vary on a task order basis.

37. Are you going to add more contractors for NOIS2?

The Government is unable to speculate on the number of companies who may receive awards for NOIS2.

38. Does each agency have their own group who does crowdsourcing or do you do it for all of them?

CoECI provides access to other federal agencies to use our contracts through a formal interagency agreement (IAA) if they have no internal capability in place. Several agencies have their own challenge and crowdsourcing programs.

39. Do you use challenges in combinations with the SBIR/STTR programs?

The government maintains both SBIR/STTR programs and open innovation programs.

These programs are designed to solve hard problems and mature technologies. The government may choose to try and solve those problems using either or both programs.

40. Is there any information regarding how many companies are interested?

The interested parties list is posted on the NOIS2 website.

41. Will this be open to only small businesses or everybody?

The NOIS2 procurement strategy is currently in the development phase, therefore NASA cannot discuss this information at this time.

42. For the crowdsourcing methodologies, does a contractor need to meet all of them to get on the contract?

The NOIS2 procurement strategy is currently in the development phase, therefore NASA cannot discuss this information at this time.

43. Will the first draft RFP be general to all interested companies?

The government anticipates offering a Draft RFP, which will be available to all interested parties. The Draft RFP will outline the Government’s requirements and will also invite industry comment on the content. The Draft RFP will be posted on FBO and the NOIS2 website.

44. Are you looking for a particular approach or methodology?

Please see the answer to Question 25.

45. Is the SOW in draft or will there be any additions reposted on FBO?

Currently the statement of work is shared in draft format with the issuance of the Requirements Request for Information (RRFI). Any additions or changes to the SOW will be posted on FBO. Please also review the dates to ensure that you are viewing the most current updates.

46. What is the role of the platform after a particular challenge begins? Can the solver be the one to provide the solution or must it come from the vendor?

Please see the answer to Question 33.

47. What would be the next step of this procurement process? Are you looking to define a handful of potential providers or how does this vetting process go? Also, can you explain what the process is like once the vendors are selected?

The NOIS2 Source Selection Process to date has included milestones such as issuance of a Sources Sought Synopsis, a Requirements Request for Information (RRFI), and today’s virtual Industry Day. These milestones represent the market research procurement stage.

The government is currently in the strategy development stage and these milestones intend to culminate the government’s understanding of the industry and marketplace. The anticipated future source selection milestones are outlined on slide 39 of the Industry Day Charts. The government anticipates releasing a Draft RFP as the next major milestone item.

Please refer to Question 1 and reference slide 19 of the Industry Day charts for an outline of the current NOIS1 vendor process.

48. Do we have one system of record to post the challenges to and funnel all of the solutions through?

No, CoECI has found it to be more advantageous to the Government to leverage the vendor’s existing crowdsourcing platforms.

49. When you put out the draft RFP, does it go to everyone that participated in the RFI or a smaller group that you identify?

The Government posts all communications with the public on the FedBizOps website.

Please note that the most recent modifications show up at the bottom of the page.

50. Is it ok if the vendor is located in a country other than the USA?

This information is pre-decisional.. Additional information will be available with the Draft RFP and will be publicly available on FedBizOps and the NOIS2 website.

51. From industry day slide 16 can you explain what is meant by multiple-award, fixed-price, IDIQ?

Indefinite Delivery, Indefinite Quantity (IDIQ) refers to a contract type that allows the Government “flexibility in both quantities and delivery scheduling”. For additional information on contract types please refer to Federal Acquisition Regulation (FAR) 16.

52. Any specific requirements regarding data hosting such as preferred hosting platform

(AWS, Azure, etc.), preferred country, other requirements?

If any specific data hosting requirements are identified, they are anticipated to be to be specified on a task order basis. The NOIS2 requirements are still pre-decisional and this information may be subject to change.

53. What is the typical geographical scope for the engagement campaigns for the challenges?

NASA Tournament Lab (NTL) challenges have the ability to support world-wide participation. The only geographical areas that are off limits are the countries currently designated as unable to do business with the US government.

54. What are the vendor selection criteria and their specific weight?

The procurement strategy is still pre-decisional. The Government is not able to provide an answer at this time.. The Draft RFP will contain that information and will be publicly available on FBO.

55. From your experience, what are some pain points or areas of improvements that you want to see going forward?

Please refer to the NOIS2 SOW for the Government’s requirements.

56. Does NASA do paid outreach? What are the typical channels that you use?

We do not do paid outreach. We rely on the vendor to do the bulk of the outreach for each challenge. Vendors decide in their proposal how much outreach they believe is necessary.

On top of what the selected vendor executes, NASA will often do a web feature on our public website and post several items to various NASA social media accounts.

57. Can the vendor use the NASA logo?

The use of the NASA logo will vary on a task order basis. General guidelines for use of the NASA logo may be viewed at https://nasa.gov/multimedia/guidelines/index.html.

58. What does the handoff of a solution from the vendor to NASA look like?

Deliverable handoff is anticipated to vary on a task order basis. Please reference the NOIS2 SOW for additional information.

59. Where is NASA going in regards to its major goals?

The NASA mission is: Lead an innovative and sustainable program of exploration with commercial and international partners to enable human expansion across the solar system and bring new knowledge and opportunities back to Earth. Support growth of the Nation's economy in space and aeronautics, increase understanding of the universe and our place in it, work with industry to improve America's technologies, and advance American leadership.

60. What tools does NASA use for crowdsourcing and ideation?

NASA uses commercially available platforms selected through traditional Government solicitations.

61. NOIS2 is an effort to go beyond crowdsourcing innovation, can you explain that?

NOIS2 is not going beyond crowdsourcing innovation, it is going beyond challenge-based crowdsourcing innovation. Please refer to the draft SOW for additional information.

62. Have the results of your crowdsourcing innovations been a wide array of products?

Our current contract deliverables range from prototypes, algorithms, software code, CAD models, multimedia products, ideas and concepts. It is a wide array because it varies based on our challenge owners requirements. Some are looking for high level ideas while others may be seeking some very specific highly technical solution.

63. Where do the solutions go once received?

Historically, the IP stance is specified in each task order. Sometimes the Government builds on the solution, sometimes the goal of the challenge is to stimulate industry. This area is something the Government is considering carefully, but is not able to share details at this time.

64. In our work with another agency, we found the paperwork reduction act to be a hurdle in past efforts. Are there any constraints like that for you?

Because the majority of our challenges have been run under procurement authority, CoECI has not had any issue with the Paper Reduction Act.

65. In previous efforts we’ve supported we’ve had to work with the union in developing the campaigns we’ve worked on. Do you have to engage the union in your process?

No. CoECI does not have experience of this nature.

66. Do you guys come up with campaigns on an as-needed basis and when required?

NOIS2 is anticipated to be an IDIQ contract. Task Orders will be issued as requirements are identified.

67. We’re interested in the Citizen Science aspect of your SOW. What methods has NASA used to engage and do outreach?

CoECI has not had much direct engagement with the Citizen Science community within NASA. More information about the active citizen science projects is available on the NASA SOLVE website. There are a range of projects posted and each is responsible for its own outreach activities. As stated in the draft SOW, CoECI is aiming to expand the number of crowdsourcing methodologies available under NOIS2.

68. In the area of prizes and challenge competitions, do you see things moving from this to things more like micro-task and freelance?

Please see the answer to Question 25.

69. Will there be a period for Q&A in the future?

The Government anticipates issuing a Draft RFP. At that time, NASA anticipates hosting a pre-proposal conference where industry can again interact with the Government.

70. Are you committed to staying to the dates that have been laid out?

The schedule that has been shared in the Industry Day Charts is an estimated schedule.

NASA will update the schedule if changes occur, and any changes to the schedule will be shared publicly on FBO.

71. Are the depth and breadth of product offered important? Are designations important like small business, etc.?

Please see the answers to Questions 16 and 41.

72. Have we done any research on vulnerability disclosures or bug bounty programs?

The CoECI team has not directly worked with the NASA’s IT security organization in that area.