47QFPA23R0001_Task_Order_0001_Attachment_1_DEOCS5.0 Phase 2 Validation Analysis Plan Outline V2 2023-02-28 .pdf

DEOCS 5.0 Phase 2 Validation Analysis Plan (Outline)

Background & Aims

Background

• DEOCS provides military commanders and other DoD leaders with feedback about current climate within their unit/organization.

• The DEOCS was redesigned and launched in January 2021

• With over a year of data collected, we’re continuing the validation of DEOCS 5.0 factors o Conducted in two phases, the first phase aimed to:

Test the validity of the defined factors using DEOCS-only data Use these data to identify possible items for exclusion o The second (current) phase of validation will focus on external validation and exploring confidential and multi-year data. The current phase involves two broad goals:

Using new, reduced factor scales, generate thresholds for alerting commanders about the risks associated with their factor scores Generate a risk composite/typology for each STO

Aims:

1. Generate thresholds for alerting commanders about the risks associated with their factor scores

2. Generate risk indices for each STO so a commander has a complete score and what that score means for their risk of each STO o Racial/Ethnic Harassment/Discrimination o Readiness o Retention o Sexual Assault o Sexual Harassment o Suicide

Core Activities

Data Source Discovery (2022-2023 under TO94)

• Search strategy

• Inclusion/exclusion criteria

• Already known/identified variables (from Climate Drivers)

• Establishing access to data

Identify Subsets of the Data that may be more Viable for Aggregate-to-aggregate Matching (e.g., unit-to-unit, installation-to-installation)

• Army UICs tend to be more reliable

• Rollups at the installation level o Some installations have been found to have less noise o Identify contacts with information on more reliable installations

DEOCS 5.0 Phase 2 Validation Analysis Plan Outline o Utilizing Military Service Academy DEOCS data in conjunction with SAGR results o Matching 2101 DEOCS data with 2022 SAGR results to assess SA/SH o In January, 2023 all MSAs will have administered a full confidential DEOCS

Offers potential for pre-post analysis (i.e., DEOCS 2021SAGR 2022

DEOCS 2022/23)

o Triangulating unit data with additional unit details (e.g., commander name, zip code, etc.)

• Limitations:

o Limiting to specific installations may introduce systematic biases in analyses e.g., more “reliable” installations may be made up of specific people like higher ranking officers, males, etc.

When calculating STO risk thresholds, demographics will matter, as we know certain outcomes are highly associated with factors such as gender, race, etc.

Limited variation may limit our ability to generalize beyond specific unit/installation types

Using Roster Data for Analyses with External Data Sources (2023 upon award of next TO)

• 2101 Roster Information o Roughly two thirds (67%) of emails listed on DEOCS rosters match to known EDIPIs on administrative data (approximately 2.8 million) o With the large number of matched roster records, we assume that we can leverage unit-level DEOCS results to link unit/organization members to completed (non- DEOCS) individual-level data to conduct the following analyses:

Unit level analyses:

• Use unit level DEOCS scores to predict individual level outcomes o These analyses would tell us how the climate in one’s unit at

Time 1 relate to their risk of experiencing the outcome of interest at Time 2

• Use unit level DEOCS scores to predict unit level outcome aggregations?

o These analyses would tell us how the climate in one’s unit at Time 1 relate to that same unit’s risk of experiencing the outcome of interest within the unit at Time 2 o Caveat: aggregating unit scores from limited OPA survey responses from external data may be extremely inaccurate

Individual level analyses:

• Use individual DEOCS scores to predict individual level outcomes

• This would require confidential data o Experiment in 2021 o Full confidential after 9/29 2022

• Experimental confidential: ~32k completed surveys with EDIPI

• Proposed Matching Process:

1. Identify DEOCS roster records that match to administrative data containing EDIPI

2. Using OPA bridge files, identify EDIPI from external data source survey IDs and merge

3. Identify matching completed surveys on external data with DEOCS roster data EDIPI

a. Caveat: we cannot identify which people on a roster actually completed a DEOCS survey, so we would be matching unit-level DEOCS scores to individual-level outcomes.

b. We may also match unit-level DEOCS scores to aggregated unit-level STO scores/rates from the other surveys by identifying folks who match to the same DEOCS roster and aggregating their scores, but this aggregation may be extremely unreliable.

4. Merge DEOCS data and outcome data by unit ID

Figure 1. Illustrative Example of DEOCS Roster Mapping: Binge Drinking and Sexual Assault (SA) STO Outcome from WGR

• Requirements:

o Getting bridge files for OPA survey IDs to EDIPIs o Potential HRPP Approval Processes

Reaching out to Rhonda to confirm required processes given DEOCS non-human subjects determination (will update with more information)

• Potential Datasets (See “Data Sources” section for details on data fielding populations):

o OPA Survey Data Sources:

Workplace Gender Relations Survey Workplace Equal Opportunity Survey Status of Forces Survey Service Academy Gender Relations Survey o Administrative Data Sources:

ADMF, DSAID, SDR

o Other Data Sources:

OSIE

• Limitations:

o No known, consistent procedure for mapping of aggregate- level outcomes (e.g., unit or installation) o Unknown overlap between survey populations o Potential issues with timing of outcome data

WGR presents issues with temporal order of fielding

Activities Requiring Full Confidential DEOCS Data (2024)

• DEOCS full confidential data information o Force wide collection push:

o Earliest expected data access:

o Expected completes:

o Unit level analyses:

Use unit level DEOCS scores to predict individual level outcomes

• These analyses would tell us how the climate in one’s unit at Time 1 relate to their risk of experiencing the outcome of interest at Time 2 Use unit level DEOCS scores to predict unit level outcome aggregations?

• These analyses would tell us how the climate in one’s unit at Time 1 relate to that same unit’s risk of experiencing the outcome of interest within the unit at Time 2

• Caveat: Aggregating unit scores from limited survey responses from external data may be extremely inaccurate o Individual level analyses:

Use individual DEOCS scores to predict individual outcomes

• These analyses would tell us how one’s own experience of climate in their unit at Time 1 relate to their risk of experiencing the outcome of interest at Time 2

• Proposed Matching Process:

o Using OPA bridge files identifying EDIPI on external OPA data sources, match outcome data to EDIPI o Using this overlapping EDIPI, match to individual DEOCS responses

• Requirements:

o Getting bridge files for OPA survey IDs to EDIPIs o Do we need to have IRBs for use of each OPA survey?

o Do we need IRBs to pursue seeing number of matches?

Request for information to Rhonda, HRPP

• Potential Datasets (See “Data Sources” section for details on data fielding populations):

o OPA Survey Data Sources:

Workplace Gender Relations Survey Workplace Equal Opportunity Survey Status of Forces Survey Service Academy Gender Relations Survey o Administrative Data Sources:

ADMF, DSAID, SDR

o Other Data Sources:

OSIE

Data Sources

2101 Anonymous DEOCS 5.0 Data

• Analyses conducted in 2023 will use DEOCS data from registrations that closed in the 2101

DEOCS survey version fielding period

• Individual-level data files will be merged to create an individual-level analytic dataset used for individual-level analyses o At this time, we are not creating annual files, so we'll be using the same process to merge by month as DEOCS 5.0 Validation Phase 1 o This dataset could also be used to create a unit-level aggregated analytic dataset, restricted to include only units with at least a 10% response rate o For individual-level analyses, responses would be reduced to only those containing the required EDIPI to link to external DoD data sources

Confidential DEOCS 5.0 Data

• Analyses conducted in 2024 will use DEOCS data from confidential registrations that closed in 2023 (exact cutoff timing TBD)

• Individual-level data files will be merged to create an individual-level analytic dataset used for individual-level analyses o This dataset could also be used to create a unit-level aggregated analytic dataset, restricted to include only units with at least a 10% response rate o For individual-level analyses, responses would be reduced to only those containing the required EDIPI to link to external DoD data sources

Non-DEOCS DoD Data

Data Type Data Title Outcomes Measured

Survey Last

Fielded

Most Recent Fielding

Survey Next Field Sample Population(s)

OPA

Survey

Workplace Equal Opportunity Survey of Active Duty

Members (WEO-A) (Confidential)

Racial/ Ethnic Harassment/ Discrimination 2017 Opened Sept.

‘22 2024 80,301 Active Duty

OPA

Survey

Workplace Equal Opportunity Survey of Reserve Members

(WEO-R) (Confidential)

Racial/ Ethnic Harassment/ Discrimination 2019 Opened Sept.

‘22 2024 203,697 Reserve

OPA

Survey Workplace Gender Relations

Survey (WGR) (Confidential) Sexual Harassment, Sexual

Assault Dec. ‘21 to

March ‘22 Jul. ‘23 to Oct. ‘23

994,826

WGRA:

746,987

WGRR:

247,839

Active Duty and Reserve

OPA

Survey Status of Forces Survey

(SOF[A/R])

Suicidal Ideation, Suicide Attempts, Retention

(Intentions), Readiness?

2020 Closing Oct.

‘22 Spring 2023 125,765 (A) 125,009 (R) Active Duty

OPA

Survey Service Academy Gender

Relations Survey (SAGR) Sexual Harassment, Sexual

Assault 2020 2022 2024 15,500

Military Service Academy

Cadets and Midshipmen

Other On-Site Installation Evaluation (OSIE) Data Suicide Rates N/A N/A N/A N/A DoD wide

Admin Active Duty Master File (ADMF) Data Retention N/A N/A N/A N/A Active Duty

Admin Defense Sexual Assault Incident Database (DSAID)

Data Sexual Assault

N/A N/A N/A N/A DoD wide

Admin Suicide Data Repository (SDR) Suicide Deaths

N/A N/A N/A N/A Active Duty/Reserve, Veterans

Data Preparation & Cleaning

Anonymous DEOCS Data

• Inclusion Criteria

• Treatment of Missing Data

• Weighting o Analyses will be conducted using unweighted data

• Factor Scale Scores

• Variable Recodes

Confidential DEOCS Data

• Inclusion Criteria

• Treatment of Missing Data

• Weighting o Using weighted vs. unweighted data will be further explored

• Factor Scale Scores

• Variable Recodes

External DoD Data

• Inclusion Criteria

• Treatment of Missing Data

• Weighting

• Variable Recodes

Planned Analyses

Empirically Derived Factor Alert Thresholds

1. Generate thresholds for alerting commanders about the risks associated with their factor scores o This analysis would provide a data-driven process for alerting commanders to high risk of experiencing specific STOs

i. Replacing the factor alerts that are currently based on percentiles within service branches o Using discriminant analysis (e.g., logistic regression), we can identify meaningful cut points in risk of STO X

i. Ex:

1. log(odds[SA]) = a + BX[binge drinking] + e

2. 𝐷𝐷𝐷𝐷 = a + b*x[binge drinking]

a. Where 𝐷𝐷𝐷𝐷 is predicted SA (discriminant function score)

b. x is the predictor

c. b is discriminant coefficient

ii. Interpretation:

1. Option 1: At value X of binge drinking, the predicted probability of SA occurring is at or above 0.5, meaning we can expect this level of binge drinking to cause SA among one or more members of a unit to experience SA—assuming strong models (i.e., models with high goodness of fit that do not overfit the data).

2. Option 2: At value X of binge drinking, the predicted probability of SA occurring is above the 3rd quartile of SA (i.e., high risk of SA can be defined as above the inter-quartile range of SA), meaning we can expect this level of binge drinking to cause high SA risk among one or more members of a unit.

3. Option 3: At value X of binge drinking, the predicted probability of SA occurring is above the historic Service average of SA, meaning we can expect this level of binge drinking to cause higher than average SA risk among one or more members of a unit.

4. Limitation: other factors not measured could increase or decrease the likelihood of SA o Absolute vs. relative risk o Through discriminant analyses, we will identify 3 options for establishing cut points

i. From these options, OFR can weigh in on desired method

ii. Methods will likely vary by:

1. Number of flagged units we want

2. How conservative/liberal our thresholds are

Process/Timeline

1. Data shaping & prep. Requirements:

o Data mapping across surveys Merge files o Outcome variables:

Identify desired rate variables for SA, SH, RH, Suicide?

Identify appropriate analytic variable for retention & readiness *Requires discussion & decision making—who should weigh in?

o Determining thresholds for inclusion What sample size are we comfortable with?

What proportion of unit representation are we comfortable with?

• We may not even have enough to be picky in this way Will require some discussion/ad hoc analysis—who should weigh in?

2. Model type(s):

o Binary logistic regression

Binary rate outcomes o Depending on the nature of each STO analytic variable, there is potential need for different models to account for the outcome variable(s)

3. List of models:

o Sexual Assault ~ [Protective/Risk Factor X] + [control(s)] o Sexual Harassment ~ [Protective/Risk Factor X] + [control(s)] o Racial/Ethnic Harassment ~ [Protective/Risk Factor X] + [control(s)] o Retention ~ [Protective/Risk Factor X] + [control(s)] o Readiness ~ [Protective/Risk Factor X] + [control(s)] o Suicide ~ [Protective/Risk Factor X] + [control(s)] o Options for control variables & implications of inclusion:

Unit controls

Individual Controls Little of both?—Requires discussion & decision making—who should weigh in?

4. Results interpretation & assessment of significance o There may be opportunities to cut down on models here based on significance

5. Model fit diagnostics (e.g., LRT, Pseudo R2, AIC, BIC, etc.)

o There may be opportunities to cut down on models here based on model fit o We may also adjust models based on fit.

6. Postestimation to determine cut point options o Identify 3 Options for threshold creation

7. Assess the ability of the 3 cut point options to “correctly” classify observations for flagging o Present results to OPA, OFR for feedback

8. Have OPA/OFR weigh in on 3 options o Requires discussion/decision. Who should weigh in?

STO Factor Indices

2. Generate risk indices for each STO o Use principal components analysis (PCA) to create 3 indices (Protective, Risk, & Composite) for each STO

i. Conduct PCA to determine loadings of individual factors on overall risk (protective, risk, or composite)

1. Identifying relevant factors inputs for specific STOs can be done in multiple ways:

a. All factors in one model (Risk & Protective)

b. Using a subset of factors identified in the STO Factor

Alignment (DEOCS 5.0 Redesign)

c. Using a subset of factors that are significantly associated with the STO identified in the discriminant analyses (above)

d. Caveat: The reason we assume we cannot conduct models that include the actual outcomes is because doing so would require us to have outcomes in the future to re-calculate loadings for each of the typologies

ii. Weight factors according to loadings & sum for overall score o Using these scores, we’ll predict each STO to assess the predictive power of established risk score

i. Given results of these predictive models, we adjust the weighting based on predictive power of determined scores

ii. Identify flagging method for scores which indicates increased risk for experiencing STO

1. Establish a range of scores that represent higher risk o Future work could involve identifying thresholds for composite scores that indicate high risk of experiencing STOs

Process/Timeline

1. Data shaping & prep. Requirements:

o Data mapping across surveys Merge files o Outcome variables:

Identify desired rate variables for SA, SH, RH, Suicide?

Identify appropriate analytic variable for retention & readiness

2. Model type(s):

o Principal Components Analysis

PCA is a variable reduction tool, similar to typical factor analysis models.

What’s unique with PCA is that, unlike factor analysis, PCA retains all elements within the model. In other words, PCA does not “throw out” the elements that don’t correlate strongly.

3. List of models:

o 3 PCA Models:

i. 1 Protective

ii. 1 Risk

iii. 1 Composite

iv. These models will be constrained to 1 “component” to produce scores for each unit in the model (1 protective-only score, 1 risk-only score, and 1 composite score) o 18 Regression models

i. Once the three PCA component scores are created, we will use them to predict each STO using regression models

ii. With each model, we’ll assess the relationship between each of the three index scores and each of the 6 STOs (18 relationships)

1. Protective score Sexual Assault

2. Risk score Sexual Assault

3. Composite score Sexual Assault

4. Etc. for each STO

4. Results interpretation & assessment of significance o Once we identify those scores that are significantly predictive of specific STOs, we will assess how these scores should be presented, and the thresholds we’ll use in order to flag these scores for Commanders.

i. The generation of thresholds will follow a similar process as the one outlined above.

ii. We’ll have to discuss where control variables may fit in. Given our findings from threshold creation (above), we may find that there is a set of control variables, at a specific level, that give us the best indicators for our purposes that we’d want to use when we predict STO scores and identify thresholds for flagging.

5. Assess the performance of the metrics o How many units would be flagged?

o Case studies on specific set of units?

i. Ones with undesirable DEOCS scores

ii. Ones exhibiting high performance/success?

Quality Control

• Syntax o Syntax will be double-checked by a statistician and/or senior researcher with substantial expertise in the specific analytic technique for:

Suitability for answering the research question Accuracy (e.g., of specifications) o Changes will be checked again prior to delivery of annotated syntax to OPA o Changes requested by OPA will be implemented and double checked in same manner

Appendix

• TBD