Amendment_One_-_Responses_to_Questions.pdf

RFP Number : NIH-OD-OER-2019-16 Amendment 1

- 1 -

OFFICE OF ACQUISITIONS

ACQUISTION SERVICES BRANCH

National Institute of Health

6710B Rockledge Drive

Bethesda, MD 20852

REQUEST FOR PROPOSAL NUMBER: NIH-OD-OER-2019-16

Amendment No.: 1

Date of Issuance: 05/24/2019

The above numbered Request For Proposal (RFP) is amended as set forth below. The hour and date specified for receipt of Offerors remains unchanged.

The purpose of this amendment is to provide Responses to all Questions received in response to this solicitation and to revise the RFP in the manner listed below.

This Amendment revises the RFP as stated below:

• Attachment 4: "Packaging and Delivery of the Proposal" is added to Section J: List of Attachments

• Revise FAR clause on Page 24, Article I.2.,c, to read FAR Clause 52.243-3, Changes-Time-and-Materials or Labor-Hours (Sept 2000)

• Delete the sentence (See SECTION J, Attachment entitled, TECHNICAL PROPOSAL COST SUMMARY) found on page 33, #3, Separation of Technical and Business Proposals. This statement does not apply.

AMENDMENT ONE (1)

“ORIS/NIH Categorization of Alzheimer’s Projects using AI/ML”

Responses to Questions from RFP NIH-OD-OER-2019-16

Amendment One is hereby released to provide answers to the questions received in response to the subject solicitation issued on May 20, 2019. Offerors were instructed to provide all questions no later than 1:00 p.m. on May 22, 2019. In the following document, the Government provides the answers to each question that was received. No further questions will be accepted in regards to this solicitation.

TECHNICAL QUESTIONS

1. Will the government provide the SME and domain expertise to help us understand the research, condition and disease categorization that will help us in building our AI model? No

2. What is the size of the data provided by the government that the team can use to train and tune the model? Equivalent to 30 to 50% of the title, abstract and specific aims contained in the specified categories. A list of the projects reported in each category can be found at the NIH Categorical Spending website.

3. Will the government provide enough labeled data on hand that will enable us to build a good AI model?

NIH can only provide the data specified in the answer to question 2. The vendor will need to choose the appropriate AI algorithm to match the available data (small categories may require different AI approaches than are used for large categories).

4. Is the government looking for us to recommend and propose a COTS solution now or is this something we should identify and recommend after evaluating the requirements? Propose the COTS solution that you have determined will produce the best results.

5. What criteria will the government use to Compare Vendor results to NIH's current RCDC categorization results? Recall and precision.

6. • Since the Information Security Categorization is overall risk level of low with no PII, is there a requirement to obtain a Public trust or any other OPM clearance for this task? Yes, because the Specific Aims contain nonpublic, sensitive data.

• Is there a requirement for the labor categories to be United States Citizens or the United States Permanent Residents? No

7. Can the contractor propose its own proprietary in-house developed system? Yes – if the system can immediately be used for the listed categorizations and can be licensed for potential future use.

8. Does the government have a potential list of possible available systems (commercial or open source) in mind? No

9. Is system identification/market analysis and overview of possible candidate systems part of the response?

No, propose what you think is the best solution.

10. Is system identification/market analysis and overview of possible candidate systems part of the work the contractor will do after the award? No propose what you think is the best solution.

11. In “Statement of work objectives” (page 37) - Does “Review pertinent work” relate to the work done by the contractor or an overview of the work done is the field of categorization in general? Both – please describe work done by the contractor and work done by the field in relation to this specific task.

12. Does the “software and fees” elements in “Other Direct Costs” include system licensing costs and system installation/maintenance costs? Yes

13. In “Statement of Work” attachment (page 3) - “Provide interface to access data in an Oracle database in static objects or loaded through dynamic scheduling capabilities” – will the details of specific table structure of databases be provided? Yes, if necessary, to complete this task but the data for the task can also be provided in standard file formats. Capabilities to interface with an Oracle database will be necessary for possible future use.

14. In “Statement of Work” attachment (page 3) - “Return data (answers), as a list or as an interactive, customizable graphic” – are there more specific requirements of what data should be represented as “graphic”? No – propose what you think will best format to communicate both the results and intermediate results necessary to optimize the categories.

15. Is the Government specifically looking for a cloud based artificial intelligence platform or are you open to other platforms/tools that are not cloud based? Either is acceptable.

16. Page 39, Table for Option to Increase Quantities. Please verify the total hours for #2 – System Identification, Customization, and Implementation. Should this be 649? Yes, the published sum was a typo.

17. How will the contractors get access to the datasets requested? The data can be provided in standard file formats or can be drawn from a standard Oracle database.

18. What kind of volume exists for each potential corpus? Approximately 60,000 projects per year is the total number of potential projects that need to be scanned to determine which projects belong in each category.

19. Will the contractor have access to the RCDC weighting algorithm? No -- this data isn’t needed for this test.

20. From the SOW “(Gold standard sets will be available to train the platform. For example, sets of grants that have been verified and belong in 1 of the 17 categories);”

• Does the algorithm have to cross classify content that are documents that belong in multiple categories (eg Alzheimer’s Disease research on American Indians) Projects need to be categorized to as many categories as they appropriately belong.

• Do the Gold Standard sets have studies that belong in multiple categories? Yes

• How were the Gold Standard sets produced (how were they created/what criteria were used)? The gold standard sets will be selected from those projects that have been verified by subject matter experts as being “scientifically relevant” to the category.

21. Will NIH have documents that do not classify into the 17 groups available for additional AI training to reduce false positives? Yes

22. Can the built-in tutorial refer to a series of independent web pages or does the tutorial have to be self-contained within the application itself? Training materials can be in whatever format is needed to provide the necessary information.

23. • Is the eventual training to be designed for NIH staff with scientific experience? Yes

• Is training to be designed for scientific experts who contribute to RCDC? No.

24. How many staff will be needed for training and how many initial users would need to go through such training? Approximately 25.

25. • Are all of the studies in English? Yes

• In what format are the studies? Title, abstract and Specific Aims as submitted in the format required in an NIH grant proposal – sample forms and instructions to applicants are publicly available at:

https://grants.nih.gov/grants/forms.htm

26. Would the COR facilitate VTC meetings to accommodate out of state work locations for the technical work force or to reduce commuting costs to NIH headquarters. Yes

27. The solicitation states "In-person training". Will the Government accommodate virtual training via VTC or similar for work performed by out of state technical staff? Yes

28. Does the NIH have an existing cloud services relationship and infrastructure that should be used for our work or is the expectation that the offeror will recommend or include cloud services in our bid. If the Offeror proposes a cloud service, it shall be a Fed-ramp cloud rated for moderate security.

29. The solicitation states that the system shall have "Ability to test different machine learning algorithms to determine which provides the best categorizations". Is this testing requirement for the R&D system where the final production system will use the intelligence established in R&D and implement best decisions or is this testing expected in the production system? Given the different categories listed for this task it is expected that a single machine learning algorithm will not return the best results for all of the categories. The vendor is expected to return the best possible categorization results for each category and this may require using different algorithms for different categories. The capability to test different algorithms will be needed for implementation in a full production system if that is the decision supported by the test results.

30. Will the Government consider a base SaaS platform + services offering to speed the way towards developing an interface that the requirements are delivered through? Is this the structure that the example

ODC amount is trying to achieve? The Offeror is to submit their technical and business proposal based on the requirements stated in the SOW. It’s the Offeror’s decision as to whether they want to propose an alternative approach. Proposed alternative approaches will be discussed during negotiations.

31. Given the rapid application of AI in the commercial sector, will the Government accept Past Performance from commercial clients and/ or commercial engagements? Yes

32. Can NIH please clarify the following from the SOW:

a. the definition of an entry (40,000 entries) Title, abstract and specific aims that are submitted for each NIH supported research project.

b. how many papers/documents would be associated with each of the 100,000 projects and on average the page size of each paper/document. See the answer to question 25, second bullet

c. Where are the documents stored today? In the NIH IMPACII document storage system and as structured text entries in the NIH IMPACII Oracle database What format are they in (pdf, word, other). PDF and database element

d. What is stored in the Oracle data base referenced in this statement “Provide interface to access data in an Oracle database in static objects or loaded through dynamic scheduling capabilities”? What is the purpose of this interface? Future use will require access to live data. New projects are entered in the database continually and to capture all projects the categorization system must be able to access live data.

BUSINESS QUESTIONS

1. In 2.1.9.b, the RFP notes that the 35-page limitation in the proposal does not include charts. Please clarify what is meant by charts in this instance. Does this mean, for example, that a full-page schedule included in the approach would not be included in the page count even if it occurred in the middle of the Technical Discussion section? Charts, tables, figures, illustrations or other references can use smaller fonts to fit on the same page with text in the Technical discussion. Otherwise, they are not included in the page count regardless of their location. Offerors can also reference them in the Appendix.

2. In 2.1.9.b, the RFP notes that the 35-page limitation in the technical proposal does not include charts. Should we assume, however, that graphics included within the text for clarification or illustration of our proposal are included in the count. See response to question #1.

3. Can the Government clarify if “charts” pertain to tables and figures within the document, and therefore will not be counted toward total page count? See response to question #1.

4. In Section M, Evaluation Factors, the first evaluation factor is Management and Administration. The technical proposal has no corresponding high level section prior to the technical discussion section, but the SOW places Management and Administration as the first task to be accomplished. Should we assume that the discussion of management and administration should take place as the first subsection of the response to the SOW rather than be a separate top-level item in the Technical Proposal? Management and Administration is both a Task Area and a Technical Evaluation Factor. The Offeror should provide information sufficient to evaluate the Factor Management and Administration. It’s at the Offeror’s discretion on how they layout/format their Technical discussion.

5. In Section M, Evaluation Factors, factor #4 is Past Performance. Although past performance is described in Section 2.1.8 in the document, there is no instruction on where in the response the material should be placed except that it is part of the Technical Proposal. Are we correct in assuming that it should be the last major section in the Technical Proposal? The Government cannot advise on the Offeror’s organization or format of its Technical proposal. The solicitation states “Past Performance” is an evaluation factor and must be submitted as part of their Technical Proposal.

6. Does this solicitation prohibit a US based prime contractor from subcontracting work to a non-US company? If proposing subcontractors, Offerors are to follow the instructions on pg 35, item #8, Past Performance Information and pgs 40-41, item #4, Subcontractors.

7. Does this solicitation prohibit a US based prime contractor from subcontracting work to a non-US company? If it is not allowed, which clause or paragraph in the solicitation covers this question. See response to question #6.

8. Can the Government clarify if this is a new contract or if there is a previous contract? If a previous contract, can you provide the name of the incumbent contractor and if there will be a transition? This is a new requirement.

9. Can the Government provide a not-to-exceed contract value or ceiling value for the pilot project? The contract ceiling or not-to-exceed value in hours is 6,010. The price ceiling will be discussed in the negotiation process.

10. The solicitation states that "The solicitation requires delivery of proposals as stated in Attachment 1:

'Packaging and Delivery of the Proposal'". However, Attachment 1 of the Solicitation is the "Proposal Intent Response Form". Please provide the packaging and delivery instructions. The Government provides Attachment 4: Packaging and Delivery of Proposals to Section J: List of Attachments.

11. Attachment 1 of this document is the “Proposal Intent Response Sheet”; Can the Government provide the attachment containing the “PACKAGING AND DELIVERY OF THE PROPOSAL” information? Yes, the Government provides Attachment 4: Packaging and Delivery of the Proposals to include in Section J:

List of Attachments.

12. Section J is found on Page 26 and does not include any attachments referring to “Packaging and Delivery of Proposal.” Can the Government provide this information? See response to question #10.

13. Can the Government clarify if there are any margin or font restrictions? See instructions in Attachment 4:

Packaging and Delivery of the Proposals which has been included in Section J: List of Attachments.

14. RFP, Section L.2, Instructions to Offerors, Authorized Official and Submission of Proposal, page 33.

The RFP references an “Attachment 1: PACKAGING AND DELIVERY OF PROPOSAL, Part III, Section J,” which was not found in the RFP. Please advise where Offerors should submit proposals and to whom they should be addressed. See response to question #13.

15. (If hard copy submission is preferred), can the Government clarify the number of copies expected? See response to question #13.

16. Can the Government clarify if electronic submission is permitted? See response to question #13.

17. RFP Section L.2, Instructions to Offerors, Authorized Official and Submission of Proposal, page

33. Please confirm if proposals are requested to be submitted only electronically, or is hard copy also required? See response to question #13.

18. (if hard copy submission is preferred), can the Government provide the contact name and mailing address where the responses should be shipped? See response to question #13.

19. There are page limits, but no page format instructions such as line spacing, fonts, and fold-outs. See response to question #13.

20. Will the Government provide an Excel template? The Government does not understand this request for an Excel template.

21. (If hard copy submission is preferred), can the Government clarify if a CD or flash drive containing the Excel file is required? Hard copy submission is not the official format of submission.

22. Attachment 1 is the "Proposal Intent Response Form". Is submission of this form required (a prerequisite

of) in order to submit a final proposal submission? Submission of this form is not required; however, it will assist the Government in planning for proposal evaluation.

23. Does the Government require the Proposal Intent Response Form/ Sheet to be submitted? If so, can the Government provide the submission venue and due date? See response to question #22.

24. The solicitations states "The technical proposal must include direct cost and resources information, such as labor-hours and categories and applicable rates, materials, subcontracts, travel, etc., and associated costs so that the offeror's understanding of the project may be evaluated (See SECTION J, Attachment entitled, TECHNICAL PROPOSAL COST SUMMARY.)" In this solicitation Section J is a list of attachments which does not include the technical proposal cost summary - will NIH please provide that attachment or related guidance. There is no TECHNICAL PROPOSAL COST SUMMARY that should be listed as an attachment. The Government included this language in error.

25. RFP, Section L.3, Instructions to Offerors, Separation of Technical and Business Proposals, pg. 33. The RFP references “SECTION J, Attachment entitled, (TECHNICAL PROPOSAL COST SUMMARY.)” Please confirm that this is not required for a T&M/Labor Hour proposal. See response to question #24.

26. Section J is found on Page 26 and does not include any attachments referring to “Technical Proposal Cost Summary.” Can the Government provide this information? See response to question #24.

27. Both sections reference "information requested in XXX (business or technical) proposal instructions and as specified in Section J, List of Attachments". Section J in the current solicitation does not provide guidance specific to this work. Will a new Section J be provided. A new Section J will not be provided.

The only revisions to Section J is to include Attachment 4: Packaging and Delivery of Proposals.

28. Section J is found on Page 26 and does not include any attachments referring to Section K submission.

The Business Proposal Instructions found on Page 38 does not mention Section K either. Can the Government/ Contracting Officer provide this information? The Offeror is to refer to Part IV – Representations and Instructions, Section K – Representations, Certifications and Other Statements of Offerors on page 27 of the solicitation.

29. The list of cover sheet requirements on Page 28, #2 is not identical to this list. Can the Government clarify if Cover Sheet information should be different between the Technical and Business Proposals (i.e., Technical uses list found on Page 28, Business uses list found on Page 39)? The Offeror is to use the cover page list of instructions found on page 28 of the solicitation.

30. Is there an incumbent for this opportunity? No

31. The Government states that a Time and Materials with Labor Hours contract is anticipated for the award;

however, the Government also included clauses for a Fixed Price contract. Can the Government confirm if a Fixed Priced contract is also an available contract type for this solicitation? No, this requirement is for a T&M contract. The FAR clause on page 24, item c should read 52.243-3 Changes-Time-and- Materials or Labor-Hours (Sept 2000).

32. ARTICLE I.1. GENERAL CLAUSES FOR A TIME AND MATERIAL OR A LABOR HOUR

CONTRACT is an empty section. Please confirm this is intended or is there missing content? Empty section is intended.

Amendment 1 - Responses to Questions