RFP_2016-N-17753__Questions_and_Responses___PART_3.docx

RFP 2016-N-17753

TITLE: PPHF 2016: Diabetes Prevention – Developing and Validating two Computer Simulation Models to Predict the Long-term Health and Economic Outcomes of Interventions for Preventing Diabetes and its Complications – Financed Solely by 2016 Prevention and Public Health Funds

Subject: Questions and Answers Part 3

Cost of travel and stipends for the External Scientific Evaluation Committee should be included in the offeror’s proposal.

The offeror's past performance information will be evaluated after completion of the technical proposal evaluation and determination of the competitive range. Evaluation of past performance will only be conducted on those offerors in the competitive range.

Each offeror within the competitive range shall be evaluated on its performance under existing and prior contracts for similar products or services. Performance information will be used as a responsibility determination. The Government will focus on information that demonstrates quality of performance relative to the size and complexity of the procurement under consideration. The Government will evaluate the quality of the offeror's past performance based on information obtained from references provided by the offeror, as well as other relevant past performance information obtained from other sources known to the Government.

PAST/PRESENT PERFORMANCE REFERENCE QUESTIONNAIRE

Name of Offeror:

Reference Information Table

By submitting a questionnaire to its’ business references, the offeror authorizes the government to discuss past performance with the business reference identified in the table above.

No

THERE SHOULD BE NO SUBMISSIONS OF ELECTRONIC COPIES OF THE PROPOSALS. HARD COPIES ONLY. ELETRONIC COPIES WILL NOT BE ACCEPTED AND WILL NOT BE CONSIDERED FOR EVALUATION.

The offeror should describe the alterative and strength and weakness of each data source and justify the selected data source.

Requirements (Nov 2011) is cited requiring the position sensitivity level (s) of Level II. Does this preclude individuals who are not American Citizens from performing work under this effort? This question was asked previously but not answered by the CDC (question 33).

No, it does not. However, the expectation is that the offeror’s personnel who have primary responsibility for managing the requirements of the contract will be U.S. based.

Clause: I.4 HHSAR 352.231-70 Salary Rate Limitation (December 18, 2015) Yes

If it is the CDC’s intent for the offerors to factor this pricing in and being that the ESEC members will not be identified until after award, can the contracting office provide recommended budget amounts for travel and honoraria, so all offerors provide consistent values in their budget for this?

The offeror is responsible for convening the members, coordinate available dates and times for the meetings and develop agendas. CDC will discuss with panel members honorariums, travel, etc. when/and if applicable.

Will the contracting office please provide more guidance on the response requirements for the cited “reference” section and “past performance sections”? Do these sections differ from the “similar experience” section that is 30% of the evaluation criteria? What information is required in these sections?

Please see the response to question 31 above.

The Past Performance information is more general while the similar experience information is more specific to building simulation models of chronic diseases/diabetes that are similar in complexity and size to the anticipated diabetes models in chronic disease epidemiology especially in type 1 and type 2 diabetes epidemiology studies

1. Sol#: 2016-Q-65241: Application of the four-part compartment models for the estimation of rates and trends in diabetes incidence and detection in the U.S. adult population (sole source to Institute for Biometry and Epidemiology, German Diabetes Center)

2. Sol.# 2016-Q65242: Global and country-specific rates and trends in incidence of diagnosed type 2 diabetes (sole source to Baker IDI Heart and Diabetes Institute, Melbourne, Australia)

Does the CDC anticipate data from either of these contracts being relevant to the Diabetes Simulation Model? If so, can the data from these two sole-source contracts be made available (post award), since the CDC has already obtained it?

No, the projects are not related.

I. Quality Assurance Surveillance Plan (QASP)/Performance Matrix Questions

A. Category: Work Quality Context: Regarding the final work plan, reports, software product, source code, documentation, user manuals, and manuscripts… the solicitation indicates that 100% of these work products must be technically correct with no correction required by the government. Also, you indicated that the method of surveillance is primarily based on “the professional judgment of the technical monitors and expert panel reviewers.”

Question 1: Is the method of surveillance primarily based on the professional judgment of the External Scientific Evaluation Committee (ESEC), or another body that the CDC chooses outside of the purview of the contractor?

ANSWER: Professional judgment of the External Scientific Evaluation Committee (ESEC) is one of several ways that CDC will use to judge the quality of the deliverables. Professional judgment from the CDC modeling team or CDC pointed outside of the purview may also be used. Other measurements such as publishing manuscripts in top-tired professional journals, acceptance of the models by professional modeling community will also be used.

Question 2: When the ‘Standards – Criteria for Acceptance’ states that “100% of the work products are technically correct,” are we to assume for example that each software product is 100% correct (e.g. have zero defects)? Or, are we to assume that there is an expectation of high quality (low defects) based on objective measures for all (100%) of the work products in the list?

ANSWER: 100% correct, no defects - deliverable time frame accounts for correction of defects if any.

Question 3: If we are to assume that all of the work products are expected to have an objectively high quality, what are the objective measures and may we propose a minimal Acceptable Quality Level (AQL) as per the FAR Part 46 Quality Assurance guidance on performance-based contracting (e.g. 95, 90, 85%) with a sliding scale for the incentives associated with that level of quality?

ANSWER: No

B. Category: Compliance Context: Regarding work reports, data acquisition, software product, source code, documentation, technical reports, user manuals, web user interface, manuscripts, and meeting management… the solicitation indicates that 100% of these work products must adhere to respective CDC and Federal government regulatory and statutory guidelines. Also, you indicated that the method of surveillance is primarily based on “the professional judgment of the technical monitors and expert panel reviewers.”

Question 1: When the ‘Standards – Criteria for Acceptance’ states that “100% of all work products adhere to… regulatory and statutory guidelines,” are we to assume that these guidelines are sourced from Section C.6 (Special Considerations) of the Solicitation? Are there any other rating or evaluation criteria that will be considered or used?

ANSWER: Yes please see Section C.6 of the RFP. No, there are no other rating or evaluation criteria to be considered.

Question 2: Is the method of surveillance primarily based on the professional judgment of the ESEC, or another body that evaluates regulatory and statutory compliance outside of the purview of the contractor?

ANSWER: Surveillance will be primarily based on the professional judgment of the technical monitors and expert panel reviewers as stated in the performance matrix (C.9). CDC may also consult outside expert is needed.

Question 3: What is the minimum Acceptable Quality Level (AQL) for initial compliance, with an opportunity to correct within a certain timeframe when it could be identified that a work product may unintentionally not meet that initial compliance AQL threshold? Can there be a sliding scale for incentives associated with rates of initial compliance and minimized timeframes for compliance correction if a work product could be identified as out of compliance?

ANSWER: There is not a minimum AQL and there will not be a sliding scale.

C. Category: Accuracy Context: Regarding literature review, equations, algorithms, software product, source code, documentation, technical reports, user manuals, and manuscripts… the solicitation indicates that 100% of these work products must be accurate in presentation and adherence to accepted elements of style with no errors. Also, you indicated that the method of surveillance is primarily based on “the professional judgment of the technical monitors and expert panel reviewers.”

Question 1: When the ‘Standards – Criteria for Acceptance’ states that “100% of all work products must be accurate in presentation and adherence to accepted elements of style with no errors,” are we to assume that any and all work products must be ready for journal publication or other academic dissemination upon submission to the CDC? Will the government accept a draft submission, review and then revise and set to final version, with the evaluation tied to the final version?

ANSWER: Please see the deliverable table Section (F.2) of the RFP.

Question 2: If so, then will the ESEC determine publication and dissemination readiness or will it be a separate body outside of the purview of the contractor?

ANSWER: Please see Deliverable #20 -Deliverable Table (F.2) of the RFP

Question 3: What is the minimum AQL for publication and dissemination readiness? Can there be a sliding scale for incentives based on minimized review and revision upon submission to the CDC?

ANSWER: Please see Performance Matrix (Section C.9) Of the RFP. There will be no sliding scale.

D. Category: Format Context: Regarding literature review, equations, algorithms, software product, source code, documentation, technical reports, user manuals, and manuscripts… the solicitation indicates that 100% of these work products must be in appropriate correspondence format, or as specified by the customer. Also, you indicated that the method of surveillance is primarily based on “the professional judgment of the technical monitors and expert panel reviewers.”

Question 1: When the ‘Standards – Criteria for Acceptance’ states that “100% of all work products must be in appropriate correspondence format,” are we to assume that these guidelines are sourced from Section C.6 (Special Considerations) of the Solicitation? Are there any other rating or evaluation criteria that will be considered or used?

ANSWER: Assumption is correct, no other rating or evaluation criteria will be used.

Question 2: If so, then will the ESEC approve correspondence formatting for all disseminated products?

ANSWER: Final dissemination if applicable will be determined by the CDC COR with input from the ESEC

Question 3: What is the minimum AQL for correspondence formatting? Can there be a sliding scale for incentives based on minimized review and revision upon submission to the CDC?

ANSWER: Please see Performance Matrix (Section C.9 of the RFP). There will be no sliding scale.

E. Category: Customer Satisfaction

Context: Regarding work reports, data acquisition, software product, source code, documentation, technical reports, user manuals, web user interface, manuscripts, and meeting management… the solicitation indicates that 100% of the time the contractor staffing demonstrates reasonable and cooperative behavior, a commitment to customer satisfaction and a concern for the interest of the customer. Also, you indicated that the method of surveillance is primarily based on “the professional judgment of the technical monitors and expert panel reviewers.”

Question 1: When the ‘Standards – Criteria for Acceptance’ states that “100% of the time the contractor staffing demonstrates reasonable and cooperative behavior…,” what is the minimum AQL for customer satisfaction? Can there be a sliding scale for incentives based on objective criteria so as to mitigate subjective judgment over differences in communication style?

ANSWER: There is not a minimum AQL and there will not be a sliding scale.

Question 2: Are there specific approaches or mechanisms being considered for gathering the customer satisfaction inputs (e.g. surveys, focus groups, etc.) and if not, would the government be amenable to the contractor conducting such approaches?

ANSWER: No, there are not any specific approaches or mechanisms being considered for gathering the customer satisfaction inputs (e.g. surveys, focus groups, etc.). Please follow the instructions of the RFP

F. General QASP Question Would the government be open to us submitting a revised QASP, based on responses to the above questions and utilizing a QASP best practice approach that has been successful with other similar clients?

ANSWER: No, the government cannot accept a revised QASP

II. General Questions

1. The requirements to adequately provide FFP cost for the design, build and tailor of two diabetes models will be highly dependent on outcomes from each subsequent tasks. Given these dependency and challenges government may have in comparing costs from different vendors would the government be open to an FFP LOE or T&M as the contract type? Please provide clarification of the computation environment that the models should be developed within.

ANSWER: No, the Government is not open to FFP LOE or T&M as the contract type.

1. In order to adequately price Task 7 & 9 FFP, please provide more specificity of the acceptance criteria/requirements for the web-based models including:

1. Acceptable defect/error rates

ANSWER: Low defect/error rates are preferred. As there are many unknown factors that may affect the defect rate, the minimum defect rate will be determined after the award based on the agreement between CDC and the contractor, but should be lower than the standard as recommended by various simulation guidelines.

1. User-friendly

ANSWER: The interface shall be simple and straight forward and easily understood by users and the model can be run by users who are not involved in developing models easily using the user manual without special training.

1. Number of stakeholders and/or sessions involved to finalize computer simulation requirements

ANSWER: This will be determined after award based on the mutual agreement between CDC and the contractor.

1. Total number (and relative size) of data sources the models will be able to handle

ANSWER: There is no specific requirement on this. But the model have to have all functionalities and meet all the technical requirements as listed in RFP.

1. What is the number of concurrent sessions that will access the web based model?

ANSWER: We expect no less than 10.

1. Is the contractor expected to host the web based model?

ANSWER: No

1. Is there a performance expectation on how quickly the model will be required to return results?

ANSWER: As quickly as possible. Although there is no minimum time requirement at this time but must be reasonable, and agreed between CDC and the contractor at the time the model development is completed.

1. Will the model utilize a fixed or variable number of cores?

ANSWER: This can be determined after the model development is completed, depending on the CDC computation environments and cost associated with each selection.

1. Is the contractor required to supply the computational environment as part of this contract? If so, will the computational environment be required to be included in the bid amount?

ANSWER: The contractor is not required to supply the computational environment as part of this contract. If the contractor’s the computational environment differs from the CDC computational environment at the time of developing the web-based interface, CDC and the contractor will negotiate for a solution to make the software operational at CDC computational environment.

Are there any latency and/or throughput requirements that must be met?

ANSWER: No

1. Should the system support batch runs of many simulations with different data inputs? If so, can you provide an estimate on the number of different configurations for a single run?

ANSWER: Yes. The number of different configurations for a single run will be determined after award based on the agreement between CDC and the contractor.

1. What operating systems must be supported by the simulation platform (e.g. Windows, Linux)? Please differentiate between the back-end processing environment and the web-based user interface.

ANSWER: Given that the model is to be housed at CDC, the models have to be compatible with the CDC IT infrastructure. This will be determined after award based on the CDC IT infrastructure at the time the model development is completed.

1. Are there any requirements or constraints on a database management system for intermediate data storage?

ANSWER No. This will be determined after award based on the agreement between CDC and the contractor.

1. Please clarify the page limitations requirements as the RFP states 25 pages (Page 82, Section L8e) vs. 30 pages (Page 82, Section L.7). Can you clarify how many pages the proposal should be?

ANSWER: This questions has been previously answered. Also see Amendment 00001 that was previously posted.

1. Can you please clarify whether the members of the External Scientific Evaluation Committee (ESEC) are prohibited from compensation for their time and expertise?

ANSWER: The offerors are responsible for convening the members, coordinate available dates and times for the meetings and develop agendas. ESEC will be formed after the award and appointed by CDC with inputs from the contractor. The CDC will discuss with panel members honorariums, travel, etc. when/and if applicable.

1. Can you please also clarify whether we need to submit written commitments from the members of the ESEC as well as their compensation rates if compensation is not prohibited?

ANSWER: Members of External Scientific Evaluation Committee (ESEC) will be formed after the award and appointed by CDC with inputs from the contractor. There is no need to contact any potential experts or get written commitments prior to award.

1. Due to the holiday and potential length of time for the Question and Answer period, would the Government be amenable to a submission due date of 2:00 p.m., Friday, July 15, 2016?

ANSWER: THE DUE DATE CANNOT BE EXTENDED TO JULY 15, 2016. HOWEVER, THE DUE DATE FOR PROPOSALS HAS BEEN EXTENDED FROM JULY 7, 2016 TO JULY 13, 2016.

1. Would CDC release the code, equations, and algorithms developed by the contractor as open source code, for public use and modification?

ANSWER: Not at this point. This will be decided after award.

1. Regarding the transfer of software and programming code to the CDC as sole owner, would the CDC permit the contractor continue and ongoing use of the code so that additional academic papers could be developed based on the code? If yes, could these additional papers be supported by other funders (e.g., NIDDK)? Would the CDC as sole owner permit the contractor continued use of the code to address responses to journal requests for revision to the 5 papers submitted as part of the contracted work?

ANSWER: No, in general. Regarding the papers, this will be determined by CDC COR and the contractor depending on the specific situation at the time.

1. Could the contractor develop the models in a programming language other than C++, Python, and Java? Specifically, would CDC consider the use of other open source languages, such as R.?

ANSWER: Given the model complex, using programming language is likely to speed up the model computation process. Other languages shall be considered only if the contractor can justify the proposed language perform much better than listed language in the RFP. In addition, the model is to be housed at CDC, the models have to be compatible with the CDC infrastructure.

1. For the type 2 diabetes model, will models be used to simulate intervention effects only in patients with manifest disease or will models also need to examine interventions in patients in the pre-diabetic disease state?

ANSWER: Both

1. Are we correct that hosting and maintenance of model/web interface would migrate to CDC?

ANSWER: Yes

1. Is our understanding correct that labor will be fixed price but travel cost will be reimbursed? What about other direct costs, such as travel for steering committee members, space for offsite project meetings, etc.? Are these billed and reimbursed, or incorporated in the proposed budget?

ANSWER: other direct costs, such as travel for steering committee members, space for offsite project meetings should be incorporated in the proposed budget.

1. Regarding “Mobile Application Development”, does the CDC intend for models to be accessible via mobile devices?

ANSWER: This will be determined at a later date after consultation with CDC IT department.

1. Regarding “Records Retention”, does ‘data created’ also refer to simulated model results?

Yes