3.7.G. Data Collection Guidance (2).docx

Draft: July 3, 2018

DATA COLLECTION, MANAGEMENT, AND REPORTING GUIDANCE

Goals

1. Set requirements that help implementers become more data-driven

2. Facilitate the flow of data from programs back to CSO to inform future analysis Data Sources Programs generally fall into two categories: research/analysis programs and non-research. Data collection is more extensive in research / analysis programs.

Data may be generated from program activities themselves and/or from program monitoring.

Common methods of generating data include surveys, focus groups, key individual interviews, direct observation, and document reviews.

Note: Direct observation: Watching, taking notes, and recording specific actions within a target community, such as communications, spatial interaction, or exclusion (SFCG Designing for Results, Chapter 12) Selection of Methodological Tools and Definitions Implementers will need to work with the relevant regional CSO team and AA and DME to select the appropriate methods and/or tools for data collection. CSO will approve any questionnaires and methodologies, to include the Monitoring and Evaluation plan that is generally required after award.

CSO will collaborate with the implementer to determine:

· Data definitions (e.g. violent election = 10 + deaths falling within 6 months before or 3 months after elections)

· Population of interest (e.g. What demographics will participant information include), sampling strategy, and appropriate sample size

· Data metrics (e.g. ???)

· Data collection methodology

· Data cleaning and quality control techniques

· Strategies to evaluate the validity and reliability of the data

· Data disaggregation: What is the unit of analysis?

· Survey design and instrument (i.e. How will the individual questions be phrased? What content will they cover?)

· Data Collection Parameters Data collected should be responsive to the research question or program goal.

Note: This includes collecting data from the correct sample (e.g. if you’re looking at increases in religious tolerance, don’t just collect data from people who believed in religious tolerance before the program).

For monitoring purposes, baseline data should be collected prior to program implementation.

Demographic information collected should be tailored to the specific program:

· Usually relevant: age, gender, and town of current residence

· For town of current residence, provide a specific town (or if possible, latitude/longitude)

· Possibly relevant: ethnicity, religion, birthplace, education level, education type, income level Data needed by source type:

· Surveys: description of survey goal; implementors and sub-implementors involved in the survey; survey dates; survey method (in person, telephone, online); survey locations (if in person); target population; sample size and explanation for why the sample size is appropriate for the question/population of interest; sample frame; sampling strategy (noting whether it is probabilistic or not); relevant demographic information about participants; questions asked and question order (think careful about the impact of question and response option order. Will often want to randomize response options); response options (if applicable); responses to questions; assessment and precautions taken for participant safety and interviewer safety (if applicable); steps taken to protect PII; data weighting strategy (if appropriate); response rate; instrument pre-testing; compensation (if any) to participants

· Number of demographic questions partly depends on survey size – it might not make sense to have 10 demographic questions for a 10-question survey

· Focus groups: description of focus group goal; what implementer conducted the focus groups; focus group date; focus group location; target population; number of participants; participant selection criteria and recruitment strategy; relevant demographic information about participants; questions asked and question order; notes on responses; interviewer information (number and demographic characteristics—ideally the same interviewers would conduct the focus groups if there are multiple); assessment and precautious taken for participant and interviewer safety; questionnaire pretesting; compensation (if any) to participants.

· If possible, collect the full responses to the questions, but transcribing (and translating) those may be too much of a burden

· Key informant interviews: description of interview goal; what implementer conducted the interview; interview date; interview location; selection criteria for the interviewee and recruitment strategy; relevant demographic information; questions asked and question order; notes on responses; interviewer information (number and demographic characteristics—ideally the same interviewers would conduct the focus groups if there are multiple); assessment and precautious taken for participant and interviewer safety; questionnaire pretesting; compensation (if any) to participants.

· If possible, collect the full responses to the questions, but transcribing (and translating) those may be too much of a burden.

· Observation: description of observation goal; what implementer conducted the observation; population of interest, sample, number of observations, unit of analysis; quality control steps; robustness checks; codebook and coding rules; intercoder reliability (if applicable) observation dates; observation locations; notes from observations, including any checklists or guidelines that guided the note-taking.

· Document review: explicitly stated research question; sources used, both in the text and as a bibliography at the end; date of review completion; what role the review played in program decision-making; document language and any translation techniques.

· If the document review’s findings will be shared outside CSO, such as with Post, require an executive summary or one-pager summarizing the review’s findings.

Data Reporting Expectations As the NOFO language states, implementers should share raw data with CSO in a clear, structured manner to the extent that is possible, while preserving safety for participants.

· In the past, implementers have said they own the raw data, so this expectation must be clear.

· Some organizations will need more support than others with structuring their raw data in a format CSO can use, so it’s best to identify any support needs early.

For any program that involves collecting information about multiple units of observation, (like people or CSOs) the ideal reporting tool is a spreadsheet with a row for every unit of observation and a column for every trait of the observation. That enables easier disaggregation based on traits. APPL can provide templates as needed.

· Example: a row for each participant in the focus group and columns for traits of those participants, like town of residence and gender

· Participants can be identified by a random ID number rather than by name All data should include a data dictionary, (codebook) that explains what any codes or shorthand / imprecise terms refer to.

· Example code dictionary: for gender, 0=male, 1=female; for religion, 0=Buddhist, 1=Muslim, 2=Christian

· Example term dictionary: “Location” = city the participant lives in, “Household” = number of individuals living in the participant’s home