Guidelines_for_Preparing_Technical_Documentation_for_NCSES_Surveys.docx

National Center for Science and Engineering Statistics Guidelines for Preparing Technical Documentation for NCSES Surveys

Singpurwalla, Darius

Contents

Survey Descriptions Purpose: The survey description[footnoteRef:1] provides a high level overview of the survey’s purpose, design, collection, and data quality measures. Survey descriptions are, for the most part, cycle independent thus, they need not be re-written for each cycle unless there has been a significant change to the survey design. [1: The SED survey description can be found by clicking on the link: http://www.nsf.gov/statistics/srvydoctorates/ and then expanding the “Survey Description” section.]

Audience: The intended audiences for the survey description are (a) data users who are trying to determine if this is an appropriate source of data for their research and (b) respondents who want to determine if this is a legitimate survey.

This suggested redesign for the survey description maintains most of the topics that are currently included but significantly reduces the level of detail given for each topic. In essence, the survey description would be closer to a bulleted list rather than detailed passages. The proposed survey description topics are listed below and a sample survey description follows in the next section.

1. Survey Overview ([Year] survey cycle) (no more than ½ page in length)

a. Purpose

b. Data collection authority (including OMB control number and expiration date)

c. Major changes to recent survey cycle

2. Key Survey Information

a. Frequency

b. Initial survey year

c. Reference period

d. Response unit (i.e. individual, establishment)

e. Sample or census

f. Population size

g. Sample size

h. Key variables: Key variables of interest are listed below.

· A listing and description of some of the key data elements collected in the survey.

3. Survey Design (no more than ½ page in length)

a. Target population

b. Sample frame

c. Sample design (if applicable)

4. Data Collection and Processing (no more than ½ page in length)

a. Data collection

b. Data processing

c. Estimation techniques

5. Survey Quality Measures (no more than one page in length)

a. Sampling error

b. Coverage error

c. Nonresponse error

d. Measurement error

6. Data Availability and Comparability

a. Data Availability: A brief statement on the years that the survey data is available.

b. Data Comparability: A brief discussion on using the data for trend analysis across the years.

7. Data Products

8. Contact Information

Sample Survey Descriptions An example of the revised survey description for the Survey of Postdocs at Federal Funded R&D Centers and Business Research and Development and Innovation Survey (BRDIS) are presented below. The example illustrates the intended structure, length and level of detail for the survey descriptions based on the proposed guidelines.

Survey of Postdocs at Federal Funded R&D Centers

1. Survey Overview (2013 survey cycle)

a. Purpose: The Survey of Postdocs at Federally Funded Research and Development Centers is intended to identify all FFRDCs that employed postdoctoral researchers (postdocs) and to collect aggregate information on the demographic characteristics and fields of research of the postdocs employed at each FFRDC. The survey is conducted as part of the Survey of Graduate Students and Postdoctorates in Science and Engineering, which is sponsored by the National Science Foundation and the National Institutes of Health.

b. Data collection authority: National Science Foundation Act of 1950, as amended, and the COMPETES Reauthorization Act of 2010 collected under Office of Management and Budget control number 3145-0077, expiration date 9/30/2014.

c. Major changes to recent survey cycle: None.

2. Key Survey Information

a. Frequency: Periodic.

b. Initial survey year: 2005.

c. Reference period: 1 October of the survey year.

d. Response unit: Establishment.

e. Sample or census: Census.

f. Population size: 40 FFRDCs.

g. Sample size: Not applicable.

h. Key variables: Key variables of interest are listed below.

· Source of financial support, sex, and citizenship

· Ethnicity and race for U.S. citizens and permanent residents

· Field of research and citizenship

· Administrative type of the FFRDC (university, nonprofit, industry)

3. Survey Design

a. Target population: The survey target population includes all FFRDCs that employed postdocs as of 1 October 2013.

b. Sample frame: The Master Government List of FFRDCs in the United States, maintained by NSF (http://www.nsf.gov/statistics/ffrdclist/)

c. Sample design: Not applicable.

4. Data Collection and Processing

a. Data collection: The FFRDC Postdoc Survey data are collected via a Web instrument. Before data collection begins, past survey respondents at each FFRDC are contacted via e-mail to confirm whether or not they are still the most appropriate contact person. The FFFRDC Web survey access information is then e-mailed to the appropriate contact persons.

b. Data processing: All data submitted by the FFRDCs are reviewed to ensure that all data fields are complete and that data are internally consistent. Data from any FFRDC that are more than 20% above or 20% below that FFRDC’s corresponding prior-year data are flagged for review by the survey staff. If additional information or data corrections are needed, FFRDC respondents are contacted by telephone or e-mail and are asked to provide additional information or to correct data.

c. Estimation techniques: No estimation is necessary as the survey is a census of all eligible FFRDCs and has a 100% response rate.

5. Survey Quality Measures

a. Sampling error: Not applicable.

b. Coverage error: There is no coverage error for this survey. FFRDCs were identified through the master list of FFRDCs maintained by the National Science Foundation.

c. Nonresponse error: None.

d. Measurement error: Potential sources of measurement errors include incomplete administrative data or data misreported by the respondent.

6. Data Availability and Comparability

a. Data availability: Postdoc data from FFRDCs are available for 2005–07, 2009, 2010, 2012 and 2013. The FFRDC survey data were first published in 2010.

b. Data comparability: The FFRDC postdoc counts in 2012 and 2013 are not comparable with the counts reported in 2010 and prior years. The Frederick National Laboratory for Cancer Research (FNLCR) revised their postdoc reporting to include only FFRDC contract employees after it was reorganized and renamed from the National Cancer Institute of Frederick in 2011. Consequently, the total number of postdocs reported by the FNLCR dropped from 286 in 2010 to 25 in 2012 and 18 in 2013. All comparisons of FFRDC postdoc data spanning this period need to account for this difference.

7. Data Products

a. Publications: NSF releases the data from this survey through InfoBriefs and Detailed Statistical Tables in the National Center for Science and Engineering Statistics publication series Postdoctorates in Federally Funded R&D Centers.

b. Electronic access: Data files for the 2010, 2012, and 2013 surveys are available on request. For the data from prior years, contact the project officer.

8. Contact Information For additional information about this survey or the methodology report, contact the Project Officer listed below.

Kelly H. Kang Project Officer Human Resources Statistics Program National Center for Science and Engineering Statistics National Science Foundation 4201 Wilson Boulevard, Suite 965 Arlington, VA 22230 Phone: (703) 292-7796 E-mail: kkang@nsf.gov

Business Research and Development and Innovation Survey

1. Survey Overview (2011 survey cycle)

a. Purpose: The Business R&D and Innovation Survey (BRDIS) is the primary source of information on R&D expenditures and the R&D workforce of for-profit, nonfarm businesses with five or more employees operating in the United States.

b. Data collection authority: The information is collected under Title 13, United States Code. Title 13, United States Code, requires that responses be seen only by persons sworn to uphold the confidentiality of Census Bureau information and may be used only for statistical purposes. BRDIS is collected under the Office of Management and Budget, control number 0607-0912, expiration date 1/31/15.

c. Major Changes to the Recent Survey Cycle: None.

2. Key Survey Information

a. Frequency: Annual.

b. Initial survey year: 2008. The predecessor to BRDIS, the Survey of Industrial R&D (SIRD), began in 1953.

c. Reference period: Calendar Year 2012.

d. Response unit: Establishments.

e. Sample or census: Sample survey.

f. Sample size: Approximately 45,000 companies.

g. Population size: Approximately 2,000,000 companies.

h. Key variables:

· R&D performance (U.S. and worldwide)

· Total and R&D employment (U.S. and worldwide)

· Sources of R&D funding

3. Survey Design

a. Target population: The target population consists of all for-profit nonfarm companies that have five or more paid employees in the United States and have at least one establishment that is classified in the in-scope sectors based on the North American Industry Classification System (NAICS), is in business during the survey year, and is physically located in the United States.

b. Sample frame: The Business Register, maintained by the Census Bureau, serves as the primary source of establishments used to create the sample frame from which the sample is selected.

c. Sample design: The BRDIS has a stratified probability sampling design that uses both simple random sampling and probability proportional to size (PPS) sampling within strata. Stratification is based on R&D activity and a NAICS-based industry code. For companies with known R&D activity, PPS sampling is based on R&D performance, for companies with unknown R&D activity, PPS sampling is based on annual payroll. Companies known to perform large amounts of R&D and companies with large amounts of payroll are selected with certainty.

4. Data Collection and Processing

a. Data collection: BRDIS uses multimode data collection by paper booklet or Web reporting instruments. Mail (48%) and Web reporting (52%) are used about equally.

b. Data processing: Given the size and complexity of BRDIS, a large number of survey responses included errors that require correction or unusual patterns that required validation. Several hundred automated edit checks were programmed to improve the efficiency of analyst data review and correction.

c. Estimation techniques: The general methodology used to produce estimates from BRDIS involves sums of weighted data (reported or imputed), in which the weights are the product of the sample weight and the unit nonresponse adjustment factor. Any exceptions to this process can be read in the methodology report.

5. Survey Quality Measures

a. Sampling error: Estimates based on the total sample have relatively small sampling errors. Estimates of sampling errors associated with each cell in the detailed statistical tables are available by request.

b. Coverage error: The BRDIS has minimal coverage error.

c. Nonresponse error: The unweighted response rate for this survey is approximately 75%. The coverage rate based on weighted R&D for known R&D performers is approximately 90%. The coverage rate based on payroll for companies not known to perform R&D is about 75%. The total quality response rate (TQRR) measures the proportion of the weighted estimate reported by units in the sample. TQRR estimates are available by request for each cell in the detailed statistical tables. The TQRR rate for total R&D performed in the United States is approximately 70%.

d. Measurement error: Variations in respondent interpretations of the definitions of R&D activities and variations in accounting procedures are of particular concern, specifically, the characterization and reporting of R&D activities by large defense contractors funded by the U.S. federal government; the reporting of R&D activities by companies classified in the R&D services industry, NAICS 5417; and how companies, in general, count and report numbers of employees in various categories, including whether they work full-time or part-time on R&D. Although no measure of measurement error is produced, ongoing efforts to minimize measurement error include questionnaire pretesting, improvement of questionnaire wording and format, inclusion of more cues and examples in the questionnaire instructions, in-person and telephone interviews and consultations with respondents, and post-survey evaluations.

6. Data Availability and Comparability:

a. Data availability: Data collected are for R&D related activities during the calendar year and for innovation activities during the past three years.

b. Data comparability: BRDIS is a cross-sectional survey designed to produce annual estimates of R&D performance and related statistics (as was SIRD). However, many of the companies that perform large amounts of R&D are included in the survey each year. Thus, there is a significant longitudinal component to the survey. Because of this and the generally low sampling variability of the annual level estimates, estimates of year-to-year changes are generally precise. Estimates for changes covering a longer time span will generally be less precise.

Users should be aware that the redesign of the SIRD to create BRDIS introduced substantial changes in scope and methodology. Chiefly, changes in the assignment of companies to industry strata, the inclusion of worldwide data, changes in the measurement of employment, and the use of a modular survey questionnaire may have introduced discontinuities in the data series. Although significant efforts were made to preserve the comparability of the data series, trend estimates that bridge the 2007–08 periods should be reviewed carefully.

7. Data Products:

a. Publications: BRDIS data are published in NCSES InfoBriefs and Detailed Statistical Data Reports in the Business and Industrial R&D series. Data from BRDIS are also used in the biennial report Science and Engineering Indicators.

b. Electronic access: The BRDIS contains confidential data that are protected under Title 13 and Title 26 of the U.S. Code. Restricted microdata can be accessed only at 1 of the 15 secure Research Data Centers administered by the Center for Economic Studies (CES) at the Census Bureau. Researchers interested in analyzing microdata can apply for data use by submitting a proposal to the CES, which evaluates proposals based on their benefit to the Census Bureau, scientific merit, feasibility, and risk of disclosure. To learn more about the Research Data Centers and how to apply, please visit the CES page on research opportunities. For additional information about the application process, including how to initiate a project, please contact the administrator at the primary site where the research will be conducted.

8. Contact Information:

For additional information about this survey or the methodology report, contact the Project Officer listed below.

Raymond M. Wolfe Senior Analyst Research and Development Statistics Program National Center for Science and Engineering Statistics National Science Foundation 4201 Wilson Boulevard, Suite 965 Arlington, VA 22230

Phone: (703) 292-7789 E-mail: rwolfe@nsf.gov

Technical Notes

Purpose: Technical notes are intended to help data users interpret and use the data. They are provided with all sets of data tables disseminated through the NCSES website and provide a general understanding of the data and data quality. They are not intended to allow readers to reproduce survey results or use microdata for statistical research.

The technical notes contain some of the same information as the survey description, but at a deeper level. The technical notes may be accompanied by technical tables. Information that is well suited for data table display should be included as a technical table (e.g., unit response rates, item response rates, item imputation rates).

Note that these guidelines address the topics that technical notes cover. Please refer to the Word templates that ITSP has developed for mechanical guidance on technical notes for both DSTs and e-tables.

Intended Audience: Technical notes are written for data users interested in using our data as disseminated through data tables.

The technical notes will contain the following sections:

Survey Overview (XXXX survey cycle) Purpose Data collection authority (including OMB control number and expiration date) Survey contractor Survey sponsor(s)

Key Survey Information Frequency Initial survey year Reference period Response unit (i.e. Individual, establishment) Sample or census Population size Sample size

Survey Design Target population Sample frame Sample design (name of sample design, name strata)

Data Collection and Processing Methods Data collection. Include an overview on the survey’s collection process.

Mode. Type and percent respondents by mode Response rates. Include a summary of response rate information and, if applicable, a sentence indicating that detailed response information can be found in accompanying technical tables.

Data editing. Include a summary statement about the survey’s editing process, including any cautions related to editing that users need to be concerned about. Call out any variables that have undergone heavy editing.

Imputation. Include a statement about the survey’s imputation processes and, if applicable, a sentence stating that detailed imputation information can be found in accompanying technical tables. In this statement, alert the reader of any known limitations of the imputation process.

· Note: If the imputation rates for all imputed variables are less than 3%, then only report the imputation rate range. Variables with imputation rates higher than 3% must be reported separately.

Weighting. Include a statement if the survey data is weighted for either sampling or non-response.

Variance estimation. Include a statement discussing any variance estimation techniques used for the survey and, the basis for using this method, and if applicable, a sentence stating that detailed variance information can be found in accompanying technical tables.

Survey Quality Measures Sampling error Coverage error Nonresponse error Measurement error

Data Comparability Include changes that have a significant impact on the dissemination or interpretation of data and/or trends, but only for the years (and variables for those years) covered by that report’s data tables (include the years covered by trend tables). Reporting only high-impact changes should prevent swamping this section with minute detail. Organize changes as follows:

· Changes in survey coverage and population

· Changes in questionnaire

· Changes in reporting procedures or classification Definitions Notes

Survey Overview Purpose. The National Science Foundation (NSF) Higher Education Research and Development (HERD) Survey is the primary source of information on separately accounted for R&D expenditures within higher education institutions in the United States and outlying areas.

Data collection authority. The information is solicited under the authority of the National Science Foundation Act of 1950, as amended. The Office of Management and Budget (OMB) control number is 3145-0100 and expires on 30 September 2016.

Survey contractor. ICF International.

Survey sponsor. The HERD Survey is sponsored by NSF.

Key Survey Information Frequency. Annual.

Initial Survey Year. In 2010, the HERD Survey replaced a previous annual collection, the NSF Survey of Research and Development Expenditures at Universities and Colleges (Academic R&D Expenditures Survey), which was conducted from FY 1972 through FY 2009.

Reference period. FY 2013. Response unit. Establishment. Sample or census. Census.

Population size. A total of 891 research-performing academic institutions.

Sample size. Sample is 891; the survey was a census of all known eligible universities and colleges.

Survey Design Target Population. The FY 2013 HERD Survey target population consisted of public and private nonprofit postsecondary institutions in the United States, Guam, Puerto Rico, and the U.S. Virgin Islands that granted a bachelor’s degree or higher in any field, expended at least $150,000 in separately budgeted R&D in FY 2013, and were geographically separate campuses headed by a president, chancellor, or equivalent.

The survey population was reviewed before data collection began to ensure that each institutional classification was accurate. Characteristics of the schools were reviewed before and during the course of the survey to determine whether changes had occurred (e.g., name; highest degree granted; school openings, closings, or mergers). Table A-1 shows all institution name changes between the FY 2012 and FY 2013 surveys.

After data collection closed, institutions were reviewed to verify that only those reporting at least $150,000 in separately budgeted R&D were included in the population. Of the 917 institutions surveyed, 26 completed the survey but reported total R&D expenditures of less than $150,000. These institutions were excluded from the population, and their data are not included in the FY 2013 survey totals. The total and federally financed R&D expenditures for these 26 institutions are listed in table A-2.

Sample frame. The frame for the FY 2013 HERD Survey included (1) all institutions considered in scope for the FY 2012 survey, (2) institutions that granted a bachelor’s degree or higher and reported greater than $0 for research on the Integrated Postsecondary Education Data System 2011 Finance Survey, and (3) all U.S. service institutions that granted a bachelor’s degree or higher and were not already part of the HERD Survey population. When FY 2012 R&D expenditures were not known, institutions in the frame were sent a brief questionnaire asking whether the institution had R&D expenditures during FY 2012 and whether those expenditures were less than $150,000, were $150,000 to $999,999, or were $1,000,000 or more.

The population review screener was sent to 261 institutions. The overall number of academic institutions in the population decreased from 893 in FY 2012 to 891 in FY 2013 (table A-3). A total of 33 institutions were added to the survey population during the population review, but 35 were subsequently removed from the population after they indicated via their survey response or a special contact that their R&D expenditures were less than $150,000 for FY 2013.

Sample design. The FY 1997 survey was the last one conducted as a sample survey. Since FY 1998, the survey has been a census of all known eligible universities and colleges.

Data Collection and Processing Methods Data collection. The FY 2013 survey questionnaires were sent by e-mail in November 2013. Respondents could choose to submit an Adobe PDF questionnaire downloaded from the Web or use a Web-based data collection system to respond to the survey. Every effort was made to maintain close contact with respondents in order to preserve both the consistency and continuity of the resulting data. Questionnaires were carefully examined for completeness upon receipt. Survey data reports were then prepared for each institution; these showed comparisons between the current and 2 prior years of data and noted any substantive disparities. Respondents were sent personalized e-mail messages asking them to provide any necessary revisions before final processing and tabulation of data. These e-mail messages included a link to the HERD Survey Web-based collection system, allowing respondents to view and correct their data online.

Respondents were asked to explain significant differences between current-year reporting and established patterns of reporting verified for prior years. They were encouraged to correct prior-year data, if necessary. When respondents updated or amended figures from past years, NSF made corresponding changes to trend data in the 2013 data tables and to the underlying microdata. For accurate historical data use only the most recently released data tables.

Mode. Respondents could choose to submit an Adobe PDF or Microsoft Excel questionnaire downloaded from the Web, or use the Web-based data collection system to respond to the survey. Only six institutions submitted data using the PDF or Excel form.

Response rates. By the survey’s closing date in May 2014, forms had been received from 874 universities and colleges out of a population of 891, a response rate of 98.1%. Responses were received from 98.5% of all doctorate-granting institutions. The R&D expenditures reported by these doctoral institutions constituted 98.6% of the estimated national R&D expenditures for FY 2013. Table A-4 displays a detailed breakdown of response rates by survey form and highest degree granted, and table A-5 displays a breakdown of response rates for each survey question.

Data editing. The HERD Survey was subject to very little editing. Respondents were contacted and asked to resolve possible self-reporting issues themselves. Questionnaires were carefully examined by survey staff upon receipt. Reviews focused on unexplained missing data and explanations provided for changes in reporting patterns. If additional explanations or data revisions were needed, respondents were sent personalized e-mail messages asking them to provide any necessary revisions before the final processing and tabulation of data.

Imputation. Missing values were imputed based on the previous year’s data and the reported data of peer institutions in the current cycle. For the 13 institutions that had not responded by the closing date of the survey and had been included in the FY 2012 HERD Survey population, R&D expenditures were imputed by applying inflator and deflator factors to the prior year’s key totals. The key totals for FY 2013 included total R&D expenditures, federal R&D expenditures, expenditures received as a subrecipient from higher education sources, expenditures received as a subrecipient from non-higher education sources, expenditures passed through to higher education entities, and expenditures passed through to non-higher education entities. Imputation factors were ratios derived from the 2-year-trend data of responding institutions with similar characteristics, including highest degree granted, type of institutional control (public or private), and level of total R&D expenditures. Other values that were not identified as key totals were imputed by applying ratios from the previous year’s data.

For four institutions that were new to the survey population, no past-year data were available. For these institutions, total R&D expenditures were assumed to be $150,000 or $1 million, depending on the institution’s response to the population review screener. Other values were then imputed as a proportion of total R&D expenditures based on the data of institutions with similar characteristics. Data for partial nonresponse were imputed using similar techniques.

Tables A-6 through A-20 present imputed amounts for each applicable survey variable. The dollar amount imputed is displayed, along with the percentage it represents of the national estimate for universities and colleges in a particular field. The imputed total R&D was $63 million, or 0.1%, of the $67.2 billion in total R&D expenditures (table A-6).

A number of surveyed institutions have responded intermittently in past years. For years in which no response was received, data have been imputed as previously described. Although the imputation algorithm accurately reflects national trends, it cannot account for specific trends at individual institutions. For this reason, a re-imputation of institutional data for prior years is also performed. For each institution, previously imputed values from the HERD Survey (FYs 2010–12) were recomputed to ensure that the imputed data are consistent with reporting patterns from the FY 2013 survey. These procedures result in much more consistent reporting trends for individual institutions but have little effect on aggregate figures reflecting national totals. In the data tables, the letter i is used to identify imputed data.

R&D expenditures from unspecified federal agencies (Question 10) and from the American Recovery and Reinvestment Act of 2009 (ARRA) (Question 11) and capitalization thresholds for software and equipment (Question 14) were not imputed. Response summaries for these questions can be found in Tables A-21 through A-23.

Weighting. Survey data were not weighted.

Variance estimation. No variance estimation techniques were used.

Survey Quality Measures Sampling error. Because the FY 2013 survey was a survey distributed to all institutions in the universe, there was no sampling error.

Coverage error. Coverage of large research institutions is excellent, because they are easily identified using the NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions. However, institutions with smaller amounts of R&D expenditures have been more difficult to identify because they often do not receive federal funding for science and engineering R&D.

As part of the expanded HERD Survey collection, a special effort was undertaken in 2010 to screen 1,715 4-year and higher postsecondary institutions that were not currently included in the survey to identify additional institutions meeting the $150,000 threshold. These institutions were given detailed instructions regarding what to include as R&D expenditures and were asked to respond to Question 1, total R&D expenditures by source of funds, of the HERD Survey. This short form survey achieved a 64.4% response rate and identified 187 additional institutions that met the $150,000 threshold. The total R&D expenditures reported by these institutions in FY 2010 was $249 million, or 0.42% of the $61.2 billion reported by FY 2010 HERD Survey respondents. More details on this effort are included in the methodology report and technical notes for the FY 2010 survey. NSF continues to annually screen all 4- year and above institutions reporting non-zero amounts of research expenses to the Department of Education Integrated Postsecondary Education Data System to determine if new institutions qualify for inclusion in the survey.

Nonresponse error. Seventeen universities and colleges did not respond in FY 2013, out of a total of 891 eligible institutions, for a nonresponse rate of 1.9%. Table A-4 displays a detailed breakdown of response rates by survey population and highest degree granted.

The item nonresponse rates varied from 0% for Question 1, total R&D expenditures by source of funds, to 17.3% for Question 6, R&D expenditures by character of work (basic research, applied research, and development). The other question with a nonresponse rate greater than 10% was Question 17, headcount of R&D postdocs (10.7%). Table A-5 displays a breakdown of response rates for each question in each of the two surveys. Tables A-6 through A-20 present imputed amounts for each applicable survey variable.

Measurement error. The most likely source of measurement error is institutional records containing categories different from those on the survey. For example, institutions were asked to report all R&D expenditures by field. The NSF-designed fields do not always translate to an institution’s departmental structure, and adjustments must be made by the institution in order to complete the survey. A crosswalk between the NSF fields of R&D and the National Center for Education Statistics Classification of Instructional Programs codes was provided with the survey in order to mitigate this source of measurement error.

Another source of error is NSF’s category of institutionally financed research. The survey requested that institutions report discretionary internal funds used for research. NSF discovered through debriefings conducted at the conclusion of the FY 2010 survey that there are varying interpretations of what should be included on the HERD Survey as institutionally funded research. Some institutions include all expenditures from separate accounts designated for research; others include only internal R&D projects that are competitively awarded and have detailed budgets. A workshop was held in summer 2012 to discuss these differences in definitional interpretations. Based on the findings from the workshop, the FY 2012 survey questionnaire was modified to clarify that all expenditures designated for research can be included within this category. This includes expenditures separately budgeted for organized research and expenditures of other funds designated for research but not categorized as organized research. A checklist question (Question 1.1) was also added to encourage inclusion of all eligible expenditures and to determine the full extent of the variation in reporting across institutions. An analysis of Question 1.1 responses from FY 2012 indicated that the majority of institutions that reported some institutionally funded R&D were including funds that would not be considered organized research (e.g., 70% included startup funds, bridge funding, or seed funding, and 71% included other departmental funds designated for research). However, many institutions reported that they still could not report institutionally funded research that was not organized research because those funds were not separately accounted for. Therefore, survey totals are missing expenditures for R&D that come from multipurpose accounts, and as such, they represent an undercount of the total amount of funding institutions make available to conduct R&D.

The reporting of unrecovered indirect costs is another known source of error. The survey requested that the total amount of indirect costs associated with a research grant or contract be calculated and reported, including costs that were not reimbursed by the external funding source. The unrecovered indirect cost is calculated by multiplying the institution’s negotiated indirect cost rate by the corresponding base and then subtracting the actual indirect cost recovery, preferably on a project-by-project basis. In FY 2013, 7.5% of respondents reported that these data were unavailable.

It should also be noted that because institutions were asked to include funds passed through to higher education institutions as well as subrecipient funding from higher education institutions, there is double-counting included in national and group totals. For example, Institution A’s survey included the $2 million passed through to Institution B, and Institution B’s survey also included that $2 million in subrecipient funding that it received from Institution A. Overall, institutions reported $3.0 billion in expenditures from subrecipient funding received from other universities and $3.1 billion in funds passed through to higher education subrecipients.

Data Comparability (Changes) Annual data are available for FYs 1972–2013. When the review for consistency between each year’s data and submissions in prior years reveals discrepancies, it is sometimes necessary to modify prior years’ data. This is especially likely to affect trends for certain institutions that fail to report every year, because current-year data are used to impute prior-year data. For accurate historical data, use only the most recently released data tables. Individuals wishing to analyze trends other than those in the most recent data tables are encouraged to contact the Project Officer for more information about comparability of data over time.

Changes in survey coverage and population. Before FY 2010, the population included only institutions with R&D expenditures and degree programs in S&E fields. Institutions that performed R&D in only non-S&E fields were excluded from the population. Although not a change in the coverage or population, each campus headed by a campus-level president, chancellor, or equivalent began completing a separate survey in 2010 rather than combining its response with the other campuses in a university system. As a result of this step, the overall number of academic institutions in the population increased from 711 in FY 2009 to 742 in FY 2010. For data users wishing to compare HERD Survey data across university systems by aggregating member campuses, table 84 shows all institutions in the FY 2013 population, including short form survey institutions, by state, institutional control, and system.

Changes in questionnaire. Tables include data from the Academic R&D Expenditures Survey (FYs 1972–2009) and the HERD Survey (FYs 2010–13). Analysts should be cautious when examining trend data. Although many variables are similar across the two surveys, because of clarification of which funds are to be included in the definition of R&D and the inclusion of non-S&E expenditures, exact comparisons may be misleading. In prior years, the Academic R&D Expenditures Survey collected expenditures for S&E and non-S&E fields separately. Institutions were not always able to provide non- S&E expenditures, and those data were not imputed previously. Also, revisions to the instructions on what types of activities are included as R&D in 2010 may have influenced reported values to varying degrees, depending on the numbers of clinical trials and training grants at a particular institution.

• For the FY 2013 collection, the instructions were revised to clarify that funds from foreign and U.S. universities and colleges should be reported under All other sources (Question 1, row f). The instructions also specified that gifts designated by donors for research should be included in Question 1, row f.

• For the FY 2012 data collection, NSF modified the survey instructions to clarify what types of institutionally funded actives should be included in reported data. The instructions explained that all expenditures for R&D from an institution’s current operating funds that are separately accounted for should be reported. This includes expenditures separately budgeted for organized research and expenditures of other funds designated for research but not categorized as organized research. The instructions also specified that funds from an institution’s 501(c)3 foundation should be reported under institutionally financed research.

Changes in reporting procedures or classification.

• In order to reduce the burden for institutions with minimal amounts of R&D expenditures, NSF introduced a shorter version of the HERD Survey beginning with the FY 2012 collection. The short form included only a few core questions. For the FY 2013 cycle, the short form included 246 institutions that reported R&D expenditures between $150,000 and $1 million during FY 2012. The remainder of the institutions (645) were included in the full version of the survey.

• Short form survey data for FY 2012 and FY 2013 appear only in those tables that specify in their title that the data presented include the short form version of the survey. Data from the short form survey population are included in the year totals prior to FY 2012, aggregated under “all other surveyed institutions.” The total FY 2013 R&D expenditures reported by institutions in the short form survey population ($132 million) represent 0.2% of the expenditures reported by all institutions ($67.2 billion).

Definitions

• Equipment expenditures by source and field. In Question 15, institutions were asked for expenditures for capitalized equipment (defined below under “Expenditures by type of cost”) by field of study and source of funds (federal or nonfederal).

• Expenditures from contracts. In Question 3, institutions were asked how much of the total R&D expenditures reported in Question 1 came from contracts rather than grants, reimbursements, or other agreements. Contracts were defined as legal commitments in which a good or service was provided by the reporting institution and benefited the sponsor. The sponsor specified the deliverables and gained the rights to the results.

• Expenditures by character of work. In Question 6, institutions were asked for the amount of federal and nonfederal R&D expenditures by character of work, as defined below:

Basic research. Research that is undertaken primarily to acquire new knowledge without any particular application or use in mind.

Applied research. Research that is conducted to gain the knowledge or understanding to meet a specific, recognized need.

Development. Development is the systematic use of the knowledge or understanding gained from research directed toward the production of useful materials, devices, systems, or methods, including the design and development of prototypes and processes.

• Expenditures for clinical trials. In Question 5, institutions were asked to report the amount of total R&D expenditures that were expended for Phase I, Phase II, and Phase III clinical trials with human patients. Clinical trials were defined as research studies designed to answer specific questions about the effects of drugs, vaccines, medical devices, tests, treatments, or other therapies for patients. Clinical trials are used to determine safety and effectiveness.

• Expenditures by federal agency and field of research. In Question 9, institutions were asked for R&D expenditures in 36 fields of study by federal funding agency. If an individual project involved more than one field of R&D, respondents were asked to prorate expenditures to report an amount for each field involved. Subrecipient funding was reported under the agency that sponsored the original award. Institutions were asked to report expenditures funded by six specific agencies (the Department of Agriculture; Department of Defense; Department of Energy; Department of Health and Human Services, including the National Institutes of Health; National Aeronautics and Space Administration; and National Science Foundation). Any expenditures funded by other federal agencies were reported under Other. The names of agencies included in the Other category were requested in Question 10. A list of the 36 fields of study reported on can be found on the survey questionnaire. In the data tables, the fields are grouped into 10 major areas: engineering, physical sciences, environmental sciences, mathematical sciences, computer sciences, life sciences, psychology, social sciences, other sciences, and non-science and engineering.

• Expenditures funded by the American Recovery and Reinvestment Act (ARRA). In Question 11, institutions were asked for the amount of federal expenditures funded by ARRA.

• Expenditures funded by foreign sources. In Question 2, institutions were asked how much of the total R&D expenditures reported in Question 1 came from foreign sources. Foreign sources included foreign governments, businesses, universities, nonprofit organizations, and any other entity sending funds to the United States from a location outside the United States and its territories. Also included were funds from international governmental organizations located in the United States, such as the United Nations, World Bank, and International Monetary Fund.

Projects sponsored by a U.S. location of a foreign company were not considered foreign.

• Expenditures within medical schools. In Question 4, institutions were asked to report expenditures for R&D projects in their medical schools. A medical school was defined as one that awards MD or DO degrees. Expenditures from projects assigned to the medical school or to research centers that were organizationally part of the medical school were included.

• Expenditures by nonfederal source and field of research. In Question 12, institutions were asked for R&D expenditures by field of study and nonfederal sources of funds (see the definitions below under “Expenditures by source”).

• Expenditures passed through to subrecipients. In Question 8, institutions were asked for the amounts of federal and nonfederal expenditures passed through to subrecipients from four types of pass-through entities (U.S. higher education institutions, businesses, nonprofit organizations, and other). Expenditures from vendor relationships were not included. Institutions were asked to report expenditures as federal or nonfederal based on the original source of funds.

• Expenditures received as a subrecipient. In Question 7, institutions were asked for the amounts of federal and nonfederal expenditures received as a subrecipient from four types of pass-through entities (U.S. higher education institutions, businesses, nonprofit organizations, and other). The subrecipient for an award carries out the work but receives the funds from a pass-through entity rather than directly from the original funding source. Subrecipients tend to be the coauthors of publications, writers of technical reports discussing findings, inventors, and similar. Expenditures from vendor relationships were not included. Institutions were asked to report expenditures as federal or nonfederal based on the original source of funds.

• Expenditures by source. In Question 1 of the survey, institutions were asked to report their total R&D expenditures by funding source, as defined below:

U.S. federal government. Any agency of the U.S. government. Federal funds that were passed through to the reporting institution from another institution were included.

State and local government. Any state, county, municipality, or other local government entity in the United States, including state health agencies. State funds that supported R&D at agricultural and other experiment stations were included. Public institutions reported state appropriations restricted for R&D activities in this category.

Business. Domestic or foreign for-profit organizations. Funds from a company’s nonprofit foundation were not reported here; they were reported under Nonprofit organizations.

Nonprofit organizations. Domestic or foreign nonprofit foundations and organizations, except universities and colleges. Funds from the reporting institution’s 501(c)3 foundation were reported under Institutional funds. Funds from other universities and colleges were reported under All other sources.

Institutional funds. Included institutionally financed research (all R&D funded by the institution from accounts that are only used for research), cost sharing (committed), and unrecovered indirect costs (the portion of indirect costs associated with a sponsored project that was not reimbursed by the sponsor in accordance with the institution’s negotiated indirect cost rate).

All other sources. Sources not reported in other categories, such as funds from foreign governments, foreign or U.S. universities, and gifts designated by the donors for research.

• Expenditures by type of cost. In Question 13, institutions were asked for expenditures by type of cost, as defined below:

Salaries, wages, and fringe benefits. Included compensation for all R&D personnel whether full time or part time, temporary or permanent, including salaries, wages, and fringe benefits paid from institution funds and from external support.

Software purchases, noncapitalized and capitalized. Included payments for all software, both purchases of software packages and license fees for systems.

Capitalized equipment. Included payments for movable equipment exceeding the institution’s capitalization threshold, including ancillary costs such as delivery and setup.

Pass-throughs to other organizations. See the definition above.

Other direct costs. Other costs that did not fit into one of the above categories, including (but not limited to) travel, tuition waivers, services such as consulting, computer usage fees, and supplies.

Indirect costs. Includes both recovered and unrecovered indirect costs.

• Fiscal year. Institutions were asked to report data for their fiscal year (or financial year).

• Headcount of R&D principal investigators (PIs) and all other R&D personnel. Institutions were asked to report the number of PIs and other personnel paid from the R&D salaries, wages, and fringe benefits reported in Question 13. A PI is designated by the institution to direct the R&D project or program and be responsible for the scientific and technical direction of the project. Co- principal investigators (co-PIs) were reported as PIs. Institutions were asked to count each person only once. If a person served as a PI or co-PI on one project and as other personnel on another project, that person was counted as a PI. The headcount included all personnel and students paid from R&D accounts, regardless of how much they received.

• Headcount of postdocs working on R&D. In Question 17, institutions were asked for the number of PIs or other personnel categorized as postdocs. NSF defined a postdoc as someone who holds a recent doctoral degree, generally awarded within the last 5 years, and has a limited-term appointment, generally no more than 5–7 years.

• Research and development (R&D). R&D is defined as creative work conducted systematically to increase the stock of knowledge (research) and to use that knowledge to devise new applications (development). R&D includes basic research, applied research, and development (see the definition of Expenditures by character of work for additional information). R&D does not include public service or outreach programs, curriculum development (unless included as part of an overall research project), or non-research training grants. Respondents were also asked to exclude capital projects (i.e., construction or renovation of research facilities) from reported expenditures.

• R&D expenditures: Institutions were asked to report R&D expenditures from the institution’s current operating funds that were separately accounted for.