FedBizOpps_Pre-Sol_Notice.docx

Presolicitation Notice Program Evaluation of NIH Peer Review Processes: The Role of Anonymization

HHS-NIH-NHLBI-SBSS-RG-16-111-JF

Contracting Office Address

Department of Health and Human Services, National Institutes of Health, National Heart, Lung and Blood Institute, Rockledge Dr. Bethesda, MD, Office of Acquisitions 6701 Rockledge Dr RKL2/6145 MSC 7902, Bethesda, MD, 20892-7902, UNITED STATES Description

This is a set-aside Presolicitation Notice for Woman-Owned Small Businesses (WOSB). This is NOT a solicitation for proposals, proposal abstracts, or quotations.

The proposed program evaluation project will test the effect on reviewer ratings of removing investigator and institutional information from grant applications. Specifically, the goal is to determine if anonymity in grant applications will reduce any effects on application scores attributable to reviewers’ perception of race, gender, seniority, and institutional affiliation of the applicant. Support services are needed to (1) anonymize grant applications by removing identifying information, (2) assign original and anonymized applications for review by CSR reviewers, (3) analyze resulting scores to determine the effect of anonymizing on overall impact scores, and (4) evaluate the accuracy of reviewers’ perceptions of applicant characteristics.

The designated NAICS code for this work effort is 541611. An organization that is not considered a small business under NAICS code 541611 should not submit a response to this notice.

Background

The NHLBI, on behalf of the Center for Scientific Review (CSR), is seeking woman-owned small businesses with the ability to serve the purpose of conducting a program evaluation project on the effect of anonymizing grant applications on reviewer scoring behavior. The data on this topic of interest are not currently available from any other source.

The Center for Scientific Review (CSR) is the portal for NIH grant applications and their review for scientific merit. We receive all research grant applications sent to NIH and handle the review of more than 70% of those by organizing peer review groups (study sections) to evaluate research grant applications. CSR has major responsibility for the quality of NIH’s investments to advance science and health through research. Our mission is to see that NIH grant applications receive fair, independent, expert, and timely reviews – free from inappropriate influences – so NIH can fund the most promising research. Each year, CSR handles over 85,000 grant applications, employs nearly 500 staff (more than 50% PhD level scientists) and uses more than 17,000 reviewers.

CSR has engaged in a new initiative to closely examine the peer review process. Aims include the identification of procedures and practices that are most beneficial in accomplishing the Center’s mission as well as identifying any aspects that might make the review process vulnerable to bias. Should any potential weaknesses be identified, CSR will work with experts in the research community to develop policies that mitigate any potential problems. The proposed work will be a first step in this process.

Recent data (Ginther et al, 2011; 2012) have shown that African American investigators are less likely than White investigators to receive NIH R01 grant funding by up to 10 percentage points. Although the highest scoring applications had equal likelihood of funding regardless of race of applicant, black applicants were less likely than other groups to receive scores that allow them to be discussed by study sections and, therefore, were not advanced for funding consideration. This raises the question of possible bias in the peer review process.

Given that demographic information is not included in NIH grant applications, the notion of bias in peer review is based on the assumptions that (1) reviewers can infer or have direct knowledge of the race, ethnicity, and/or gender of grant applicants and (2) this information influences their ratings. To investigate these possibilities, reviewer perceptions of applicant demographics must be assessed. In addition, a comparison will be made of reviewer scores on grant applications submitted by black and white investigators where their identities are either masked or unmasked. If all other applicant characteristics are equivalent, a difference in scores for masked and unmasked applications from black applicants that is greater than the difference in scores for masked and unmasked applications from white applicants would provide support for the hypothesis of bias due to race of applicant. In addition to race, the effect on awards of gender, seniority and institutional affiliation is also of interest.

Project Requirements

The proposed project will test the effect on reviewer ratings of removing investigator and institutional information from grant applications. Specifically, the goal is to determine if anonymity in grant applications will reduce any effects on application scores attributable to reviewers’ perception of race, gender, seniority, and institutional affiliation of the applicant. Services are needed to (1) anonymize grant application by removing identifying information from grant applications, (2) assign original and anonymized applications for review by CSR reviewers (3) analyze resulting scores to determine the effect of anonymizing on overall impact scores, and (4) evaluate the accuracy of reviewers’ perceptions of applicant characteristics and assess the perceived grantsmanship of each application. CSR will supply 1200 grant applications to be reviewed in whole and anonymized versions. Each version will receive three independent reviews by reviewers that will judge between 4 and 6 applications each and then electronically submit their reviews.

A critical component of this proposed project is data security; specifically, the security of grant and investigator related information and maintenance of the confidentiality of grant reviewers. Requirements include assurance of secure communication with potential reviewers, confidential transmission of grant related materials and related critiques, and the security of personally identifiable information. Potential vendors must have appropriate equipment, software, policies/procedures, and personnel to maximize data security. They must be prepared to work with applications and data permanently residing on NIH servers. In addition, a firm understanding of the NIH peer review processes, practices and regulations is needed so that procedures for selection of reviewers and assignment of applications for review can mirror NIH practices.

Evidence of data analytic experience is required including measure development and testing of psychometric properties. Relevant data analytic skills are also needed to design an anonymizing experiment and answer the following research questions: (1) To what degree do reviewers believe they have awareness of the demographic characteristics of grant applicants? (Can grant applications be anonymized?) (2) How accurate are their assessments of the demographics of grant applicants? (3) To what degree do scores provided for anonymized applications differ from their original versions that include investigator and institution information? (4) Are the differences in scores between anonymous and original applications different for black versus white applicants? and (5) Are there any differences in grantsmanship ratings for original and anonymized applications for black and white Principal Investigators (PIs)?

The solicitation release date is pending. The Government intends to negotiate one time-and-materials type contract, for a period of two years with an approximate award date of September 15, 2016.

Point of Contact

Jenn Frazier, Contract Specialist, NHLBI COAC Services Branch, Phone (301) 496-6092, Fax (301) 480-3338, Email: jenn.frazier@nih.gov Janet Mattson, Contracting Officer, NHLBI COAC Services Branch, Phone (301) 435-0357, Fax (301) 480-3338, Email: mattsonjm@nhlbi.nih.gov Department of Health and Human Services, National Institutes of Health, National Heart, Lung and Blood Institute, Rockledge Dr. Bethesda, MD, Office of Acquisitions 6701 Rockledge Dr RKL2/6145 MSC 7902, Bethesda, MD, 20892-7902, UNITED STATES