FY2020 Longitudinal Study Technical Appendices.docx

U.S. Department of Veterans Affairs Vocational Rehabilitation and Employment (VR&E) Longitudinal Study (PL 110-389 Sec. 334) Annual Report 2021 for FY 2020 Appendices

Contract No. VA101V-16-P-3700

Submitted by:

Economic Systems Inc. (Prime Contractor) 3120 Fairview Park Drive, Suite 500 Falls Church, VA 22042 703.642.5225 www.econsys.com

Westat, Inc. (Subcontractor) 1600 Research Boulevard Rockville, MD 20850

301.251.1500 www.westat.com i i Contents

Appendix A – Section 334 of Public Law 110-389 PUBLIC LAW 110–389—OCT. 10, 2008. SECTION 334. LONGITUDINAL STUDY OF DEPARTMENT OF VETERANS AFFAIRS VOCATIONAL REHABILITATION PROGRAMS

(a) Study Required.— Chapter 31 is amended by adding at the end the following new section:

‘‘§ 3122. Longitudinal study of vocational rehabilitation programs “(a) Study Required.—(1) Subject to the availability of appropriated funds, the Secretary shall conduct a longitudinal study of a statistically valid sample of each of the groups of individuals described in paragraph (2). The Secretary shall study each such group over a period of at least 20 years.

“(2) The groups of individuals described in this paragraph are the following:

“(A) Individuals who begin participating in a vocational rehabilitation program under this chapter during fiscal year 2010.

“(B) Individuals who begin participating in such a program during fiscal year 2012.

“(C) Individuals who begin participating in such a program during fiscal year 2014.

“(b) Annual Reports.—By not later than July 1 of each year covered by the study required under subsection (a), the Secretary shall submit to the Committees on Veterans' Affairs of the Senate and House of Representatives a report on the study during the preceding year.

“(c) Contents of Report.—The Secretary shall include in the report required under subsection (b) any data the Secretary determines is necessary to determine the long-term outcomes of the individuals participating in the vocational rehabilitation programs under this chapter. The Secretary may add data elements from time to time as necessary. In addition, each such report shall contain the following information:

“(1) The number of individuals participating in vocational rehabilitation programs under this chapter who suspended participation in such a program during the year covered by the report.

“(2) The average number of months such individuals served on active duty.

“(3) The distribution of disability ratings of such individuals.

“(4) The types of other benefits administered by the Secretary received by such individuals.

“(5) The types of social security benefits received by such individuals.

“(6) Any unemployment benefits received by such individuals.

“(7) The average number of months such individuals were employed during the year covered by the report.

“(8) The average annual starting and ending salaries of such individuals who were employed during the year covered by the report.

“(9) The number of such individuals enrolled in an institution of higher learning, as that term is defined in section 3452(f) of this title.

“(10) The average number of academic credit hours, degrees, and certificates obtained by such individuals during the year covered by the report.

“(11) The average number of visits such individuals made to Department medical facilities during the year covered by the report.

“(12) The average number of visits such individuals made to non-Department medical facilities during the year covered by the report.

“(13) The average annual income of such individuals.

“(14) The average total household income of such individuals for the year covered by the report.

“(15) The percentage of such individuals who own their principal residences.

“(16) The average number of dependents of each such veteran.”.

(b) Clerical Amendment.— The table of sections at the beginning of such chapter is amended by adding at the end the following new item:

“3122. Longitudinal study of vocational rehabilitation program

Appendix B – Survey Methodology Survey methodology. Each of the three VR&E cohorts of participants will be followed annually for 20 years. A stratified random sample of cohort members will be invited to complete the annual survey. At the end of the full 20-year study period, each cohort must have a survey sample size that will provide enough statistical power to detect significant changes over time, as well as to detect statistically significant differences between specific subpopulations of interest. A final survey sample of 1,190 Veterans in each cohort that are still participating in the study at the end of the 20 years will yield a statistically valid sample with enough power to conduct meaningful analyses among subpopulations of interest. To ensure a statistically valid survey sample at the end of the 20-year period, the survey design requires that a minimum of 3,500 participants in each cohort complete the survey during the first year of administration. Table B-1 presents the schedule of survey data collections VA has planned beginning in 2012.

Reporting Year (EconSys)

FY 2010/

FY 2011

Cohort I n=10,792

Cohort II n=15,397

Annual Survey

Cohort I n≈3,710

Cohort II n≈3,636 Initial Survey

Cohort III n=21,082 n≈3,710

Cohort II n≈3,636

Annual Report Due to Congress:

July 1, 2015 n≈3,710

Cohort II n≈3,636 n≈4,102

July 1, 2016 n≈3,710

Cohort II n≈3,636 n≈4,102

July 1 n≈9,928

Cohort II n≈14,604 n≈10,856

July 1, 2021

The new sample will continue based on the FY2020 update

July 1, 2022

Sample

The new sample will continue based on the FY2020 update

Annual Report Due to Congress:

July 1

Source: EconSys Study Team Sampling strata. Survey estimates can be for the entire population or for subpopulations, referred to as domains of interest. As the survey has continued, we have monitored the recruitment and enrollment process to ensure we have recruited a sufficient number of Veterans with specific characteristics that will allow us to compare subpopulations of interest at the end of the 20-year study period. Figure B-1 lists the domains of interest used to stratify the sample for the VR&E Longitudinal Survey, namely gender, age, and the program track being pursued while in rehabilitation.

Figure B-1. Sampling Strata for VR&E Longitudinal Survey

Source: Westat After identifying a decline in responses over the past few iterations of the survey, we determined that a refresh of the sample was needed to ensure a final survey sample of 1,190. In order to refresh the sample, the following Veterans were added to each cohort:

Cohort I (FY 2010): Invite again the 6,430 Veterans who did not respond to the initial survey, but did not explicitly refuse to participate, increasing the total to 9,928 Veterans.

Cohort II (FY 2012): Invite 1,502 Veterans who were not invited to participate in the initial survey and invite again 9,914 Veterans who did not respond to the initial survey, but did not explicitly refuse to participate, increasing the total to 14,604 Veterans.

Cohort III (FY 2014): Invite the 7,082 Veterans who were not invited to participate in the initial survey, increasing the total to 10,856 Veterans.

Survey response rates for ninth year of data collection. Findings reported in the first two years of the VR&E Longitudinal Study were based solely on VBA administrative data. The initial survey for the study was administered to Cohort I and Cohort II in the fall of 2012, and annual follow-up surveys were fielded every subsequent year. For this current report, data collection for the eighth annual follow-up survey began in the winter of 2020 for Cohorts I and II. All Veterans and Servicemembers in Cohorts I and II who completed the initial survey in 2012 were encouraged to participate this year for the seventh annual follow-up.[footnoteRef:2] The initial survey for Cohort III was administered early in 2015, and the sixth annual follow-up was administered in the winter of 2020. [2: During the first year of survey data collection in 2012, a $20 incentive was offered to Veterans who completed the initial survey. The incentive was not offered for the first annual follow-up administered in 2013, however a $20 incentive was reinstituted for the 2014 survey. For 2015, a $10 incentive was provided to survey respondents. For this current survey year, no incentive was provided to survey respondents.]

Cohort members with email addresses were first invited to complete an online web survey starting in November 2020 for all three Cohorts. A letter was also mailed to cohort members (both with and without email addresses) inviting them to complete the web survey. Weekly email reminders were sent to non-respondents with email addresses. Cohort members who did not complete a web survey by week 3 of the data collection field period, including those cohort members without email addresses, were then sent a paper-pencil copy of the survey through the mail. At the start of week 5, non-respondents of both the web and mail survey were then contacted by telephone and invited to complete the survey over the phone with an interviewer. Data collection ended in March 2021.

The sampling plan for the survey was designed to yield 3,500 completed surveys from each cohort during the initial year of administration. This was achieved with 3,710 Cohort I members, 3,636 Cohort II members and 4,102 Cohort III members responding to the initial survey. As stated previously, it was determined that the initial sampling plan would not meet the end of study requirements and was modified this year. Table B-2 shows this year’s responses rates including the additional Veterans who were added to the study.[footnoteRef:3] [3: Appendix E includes details on the procedures used for survey non-response weighting.]

Non-response weighting of survey data. Although the survey design for the longitudinal study emphasized a specific number of completed surveys, as opposed to reaching a target response rate, non-response bias is still a concern, as with any scientific study that uses survey data. Rigorous, sound research practice dictates that studies based on survey data employ the use of survey weights to adjust for non-response, and thereby reduce the potential for bias. Therefore, we examined response rates by strata and constructed survey weights to account for non-response. When analyzing and reporting on findings from the survey data, survey weights allow us to adjust the survey data to represent the population from which the same was drawn. Hence, although the survey was administered to sample of cohort members, analysis of the survey data using survey weights allow us to generalize the findings to the entire cohort population. See Appendix E for more details on the procedures used for non-response weighting.

Cohort I

(FY 2010)

Cohort II

(FY 2012)

Cohort III

(FY 2014)

1 Eligible potential respondents include cohort members who completed the initial survey and were still alive at the start of the field period for the third annual follow-up.

2 Eligible potential respondents include cohort members who were not deceased and cohort members who did not refuse to be contacted for follow-up surveys.

3 Final refusals include those respondents who indicate that they do not want to participate and want no further contact about the study in future years.

Source: VR&E Longitudinal Survey

Appendix C – VR&E Longitudinal Survey

PIN:

Appendix D – Administrative Variables

Track 2 – Rapid Access to Employment Track 3 – Self-Employment Track 4 – Employment through Long-Term Services Track 5 – Independent Living

Female

High school Some college Four-year degree Graduate training

Diseases of the eye, ear, or other sensory organs Other physical health conditions or diseases Neurological conditions and convulsive disorders (excluding mental health conditions or disorders)

PTSD

Major depression Other mental health conditions Dental and oral conditions

Post World War II Korean Conflict Post Korean Conflict Vietnam War Post Vietnam War era Gulf War era I Gulf War era II

Navy Air Force Marine Corps Coast Guard Reserves/Guard or Other

W1 – W5 (warrant officer) O1 – 08

College, non-degree Undergraduate Graduate school Farm cooperative Non-pay work experience in a Federal, state, or local agency Apprenticeship Nonvocational Chapter 31 (includes Independent Living and Extended Evaluation) Non-pay on-job training in a Federal, state, or local agency; training in the home; vocational courses in a sheltered workshop or rehabilitation facility; and independent instructor On-job training (OJT) Special rehabilitative services; improvement of rehabilitation potential; special evaluation procedures; adult basic education

Appendix E – Weighting Procedure When research studies collect survey data from a set of individuals sampled from a specific population, it is important that the survey respondents are representative of the population being studied. When analyzing data collected from survey respondents, analysis weights are needed to correct for imperfections in the respondent data that might lead to systematic deviations between the respondent data and the reference population. One such imperfection is possible lack of representativeness due to sampling. Another imperfection is possible bias due to different types of respondents having different response rates.

This appendix describes how we computed the weights for respondents to the FY 2020 data collection effort, which contacted Veterans in cohorts for FY 2010, 2012 and 2014. Cohort I (FY 2010) and Cohort II (FY 2012) were first contacted in the FY 2012 data collection effort and then re-contacted during the FY 2013, FY 2014, FY 2015, FY 2016, FY 2017, FY 2018 and FY 2019 data collection effort for annual follow-up interviews. Cohort III (FY 2014), on the other hand, was contacted for the first time with the FY 2014 data collection effort and then re-contacted during the FY 2015, FY 2016, FY 2017, FY 2018 and FY 2019 data collections. A cohort’s first year of data collection is referred to as Cycle 1. For the FY 2020 data collection, all originally sampled records were contacted, minus out of scope records (deceased, refusals to the study, etc.).

To develop the FY 2020 weights, we executed the following procedures: The final Cycle 1 weights were adjusted for Cycle 6 (or Cycle 8) non-response using cell weighting classes defined by the Cycle 1 sampling strata.

First, base weights equal to the reciprocal of the probability of selection were calculated for each sampled Veteran.

Next, the base weights were adjusted for nonresponse using weighting classes defined by the sampling stratum.

The non-response-adjusted weights were then adjusted to the same population counts as Cycle 1, determined from VA-provided administrative data.

Details of these weighting procedures are described in the following sub-sections.

Non-response adjustments. To compensate for losses due to nonresponse, we adjusted the base weights of the respondents to account for the weights of the non-respondents. Specifically, we made the adjusted weight for nonrespondents equal to zero and adjusted the base weight of respondents by multiplying it by an adjustment factor. For all cases in FY 2020, these adjustment factors were the reciprocals of the response rates calculated from all cases in the same sampling stratum as the given case and there was no need for collapsing cells due to a small number of respondents.

Adjustment to control totals. The final step in the FY 2020 weighting process is the same as the final step in the Cycle 1 weighting process – the further adjustment of the weights so that they sum to control totals computed from the Cycle 1 administrative data provided by VA. Four sets of control totals were computed, by:

Gender (male and female), Age category (<30, 30-44, 45-54, and 55+), Program track (Tracks 1-3, Track 4, and Track 5), and Serious employment handicap status (yes and no).

Appendix F – Understanding Tables and Figures Descriptive Univariate Analysis Most figures and tables throughout the body of the report present descriptive data for those cohort members who are still persisting, have discontinued, and have been rehabilitated (as of the end of the previous fiscal year, unless noted otherwise). Summary data that are based on categorical variables present distributional information (e.g., frequencies and percentages) of all possible values for the specific variable. Averages (e.g., means and medians) are presented for continuous variables.

Cohort population versus survey sample. Two sources of data are used for all analyses presented in this report: (1) VBA administrative data available for the entire cohort population, and (2) survey data available for those eligible cohort members who responded to the VR&E Longitudinal Survey. All tables and figures in the report presenting descriptive data indicate if the data are based on VBA administrative data available for the cohort population or survey data that has been weighted up to reflect the cohort population. As discussed in Appendix E, weights were constructed for survey respondents to adjust for possible nonresponse bias so that the survey data are representative of the cohort population, and not just those cohort members who responded to the survey. Hence, all analyses presented in the body of the report that are based on survey data use weighted survey data rather than unweighted survey data. Appendix G and H presents both weighted and unweighted data for cohort characteristics that are based on survey data.

By the start of data collection for FY 2016, individuals from all three cohorts had passed away. These individuals were removed from the survey sample (i.e., coded as ineligible), and as a result, the survey weights do not sum to the initial cohort population total. Another important point to note is that the weighted survey data are not calibrated to represent the subgroups of persisting, discontinued and rehabilitated, as these statuses can change over time. Therefore, population counts by subgroups differ between the survey and administrative data.

VBA administrative data were not always available for every cohort member for every administrative variable, and are noted in the tables as ‘Data Unavailable.’ Additionally, survey respondents were allowed to refuse or skip specific questions within the survey, resulting in a small number of missing observations for some survey items, which have been noted as ‘Survey Item Skipped.’

Mean versus median. Mean and median are statistical terms that are used to describe the central tendency or average of a group of data. A mean is traditionally used to measure the average or midpoint of a sample, however means are subject to being affected by extreme values in the sample (e.g., values that are very high or very low compared to the rest of the sample). In such cases, a median is a better measure of the sample average because it is much more robust and not affected by outliers. A mean is calculated by adding up all values and dividing that sum by the number of values. A median is calculated by listing all values in ascending order and then locating the number in the center (e.g., middle) of that distribution. For an even-numbered list of values, the median is calculated by locating the two middle numbers, summing those two values, and then dividing by two. Because the distribution of some measures is so heavily skewed toward zero (e.g., annual earnings), and thus, the data are not normally distributed, tables within the report may report both the unconditional (includes zero values) and conditional (excludes zero values) mean and/or median.

Standard deviation versus standard error. The standard deviation is a measure of variability. The standard deviation of a sample provides an estimate of the variability of the population from which the sample was drawn. In contrast, the standard error provides an estimate of how much sample means will vary from sample to sample. When calculating the sample mean, we are usually interested in, not the mean of that particular sample, but instead the mean for the population upon which the sample is based. A sample mean will vary from sample to sample, and the amount of that variation is estimated by the standard error of the estimate of the mean. In other words, the standard error provides a measure of the precision of the sample mean. So if we want to determine how widely scattered or variable some measurements are, we use the standard deviation. If we want to indicate the uncertainty around the estimate of the mean measurement, we calculate the standard error of the mean. The standard error can be used to calculate a confidence interval around the mean estimate. A 95% confidence interval is obtained by multiplying 1.96 by the standard error and then subtracting and adding that product to the mean.

Explanatory Multivariate Analysis Multivariate analysis is used throughout the report to identify factors that affect or explain a certain outcome. Regression analysis identifies how a given independent variable (or predictor) affects a dependent variable (or outcome), while holding other variables fixed. For example, female Veterans tend to be younger than male Veterans. When we examine differences in outcomes by gender alone, we may be also picking up an age affect as gender and age are correlated. If we include both gender and age in a regression analysis, we can identify the independent effect each of these variables have on their own on the outcome of interest. Therefore, regression analysis helps to identify the independent contribution of each factor to the outcome of interest.

All of the tables throughout the body of the report that present the result of regression analysis list the statistically significant factors (e.g., predictors) that affect the outcome variable indicated. Additionally, the tables in the report body indicate if that factor has a positive or negative affect on the specific outcome measure, noted in the table by a (+) or (-). For instance, if the outcome of interest is household income, and the table indicates that age has a positive (+) effect on income, this means that as age increases, income increases. Appendix K presents more detailed results from the multivariate regression analyses summarized in Appendices G, H and I of the report. This additional detail provides information that describes the extent to which the statistically significant factors affect the outcome. The paragraphs below describe how to understand this additional detail for each type of regression estimation included in the report.

Logistic regression. Logistic regressions are used for dichotomous outcome variables where a 1 indicates one value of interest and a 0 indicates the other value of interest. For example, a logistic regression was used to determine the probability of rehabilitation, where a 1 indicates rehabilitation and 0 indicates persisting. Appendix K presents the odds ratio and associated p-value (or statistical significance level) for each factor included in the regression model. Odds ratios larger than a value of 1 indicate a positive association between a factor and the outcome. Odds ratios smaller than 1 indicate a negative association between a factor and the outcome. Larger odds ratios indicate a stronger association.

Survival regression. Survival regressions are used to determine the association of the amount of time that passes before some event occurs with the factors that may be associated with that quantity of time. Survival regression estimation was used to determine the factors associated with the amount of time it takes cohort members to rehabilitate. Appendix K presents the hazard ratio and associated p-value for each factor included in the regression model. Hazard ratios larger than a value of 1 indicate a negative association between a factor and the outcome. Hazard ratios smaller than 1 indicate a positive association between a factor and the outcome. Larger hazard ratios indicate a stronger association.

Linear regression. Linear regressions are used for continuous outcome variables. Appendix K presents the coefficient and associated p-value for each factor included in the regression model. Positive coefficient values indicate a positive association between a factor and the outcome. Negative coefficient values indicate a negative association between a factor and the outcome. When looking at the absolute value of coefficients, larger absolute values indicate a stronger association.

Negative binomial regression. Negative binomial regressions are used for ordinal count variables, such as the number of months worked during the past year, a count variable ranging from 0 to 12. Appendix K presents the coefficient and associated p-value for each factor included in the regression model. Positive coefficient values indicate a positive association between a factor and the outcome. Negative coefficient values indicate a negative association between a factor and the outcome. When looking at the absolute value of coefficients, larger absolute values indicate a stronger association.

Appendix G – Characteristics of Study Cohorts by Program Participation Status This appendix presents the demographic, military-related, and program-related characteristics of each study cohort in tabular format. The characteristics are displayed for those cohort members who are still active, have discontinued and have been rehabilitated (as of the end of the previous fiscal year).

Tables G-1a through G-12 are based on VBA administrative data that is available for the entire cohort population. Tables G-13a through G-17 are based on self-report survey data that is available for survey respondents only, but has been weighted up to reflect the entire cohort population (see Appendix E for more details on weighting procedures). All tables based on survey data present the unweighted survey data that sums to the survey sample, in addition to weighted data that sums to the cohort population. Administrative data on current case status was not available for two Cohort I (FY 2010) members, one Cohort II (FY 2012) and for three Cohort III (FY 2014) members. Hence, the cohort total for all tables and figures that report findings by current case status is 10,789 for Cohort I (instead of 10,791), 15,395 for Cohort II (instead of 15,396) and 21,079 for Cohort III (instead of 21,082).

Table G-1a. Program Track Selection of Cohort I (FY 2010) by Participation Status as of end of FY 2020

Participants All Rehabilitated Participants

Participants Total

Cohort I (FY 2010)

Note: Frequencies (N) and percentages (%) reported in table are based on VBA administrative data available for cohort population.

Table G-1b. Program Track Selection of Cohort II (FY 2012) by Participation Status as of end of FY 2020

Participants All Rehabilitated Participants

Total

Cohort II (FY 2012)

Source: VBA Administrative Data

Table G-1c. Program Track Selection of Cohort III (FY 2014) by Participation Status as of end of FY 2020

Participants All Rehabilitated Participants

Total

Cohort III (FY 2014)

Source: VBA Administrative Data

Participants All Rehabilitated Participants

Total

Cohort I (FY 2010)

Participants All Rehabilitated Participants

Total

Cohort II (FY 2012)

Participants All Rehabilitated Participants

Total

Cohort III (FY 2014)

Source: VBA Administrative Data

Participants All Rehabilitated Participants

Total

Cohort I (FY 2010)

Participants All Rehabilitated Participants

Total

Cohort II (FY 2012)

Participants All Rehabilitated Participants

Total

Cohort III (FY 2014)

Table G-4. Age Ranges of Study Cohorts by Participation Status as of end of FY 2020

Participants All Rehabilitated Participants

Total

Cohort I (FY 2010)

Participants All Rehabilitated Participants

Total

Cohort II (FY 2012)

Table G-4. Age Ranges of Study Cohorts by Participation Status as of end of FY 2020 (continued)

Participants All Rehabilitated Participants

Total

Cohort III (FY 2014)

Source: VBA Administrative Data

Participants All Rehabilitated Participants

Total

Cohort I (FY 2010)

Participants All Rehabilitated Participants

Total

Cohort II (FY 2012)

Table G-5. Length of Service in Military of Study Cohorts by Participation Status as of end of FY 2020 (continued)

Participants All Rehabilitated Participants

Total

Cohort III (FY 2014)

Source: VBA Administrative Data

Participants All Rehabilitated Participants

Total

Cohort I (FY 2010)

Participants All Rehabilitated Participants

Total

Cohort II (FY 2012)

Participants All Rehabilitated Participants

Total

Cohort III (FY 2014)

Source: VBA Administrative Data

Participants All Rehabilitated Participants

Total

Cohort I (FY 2010)

Participants All Rehabilitated Participants

Total

Cohort II (FY 2012)

Participants All Rehabilitated Participants

Total

Cohort III (FY 2014)

Source: VBA Administrative Data

Participants All Rehabilitated Participants

Total

Cohort I (FY 2010)

Participants All Rehabilitated Participants

Total

Cohort II (FY 2012)

Participants All Rehabilitated Participants

Total

Cohort III (FY 2014)

Source: VBA Administrative Data

Participants All Rehabilitated Participants

Total

Cohort I (FY 2010)

Participants All Rehabilitated Participants

Total

Cohort II (FY 2012)

Participants All Rehabilitated Participants

Total

Cohort III (FY 2014)

Source: VBA Administrative Data

Participants All Rehabilitated Participants

Total

Cohort I (FY 2010)

Participants All Rehabilitated Participants

Total

Cohort II (FY 2012)

Participants All Rehabilitated Participants

Total

Cohort III (FY 2014)

Source: VBA Administrative Data

Participants All Rehabilitated Participants

Total

Cohort I (FY 2010)

Participants All Rehabilitated Participants

Total

Cohort II (FY 2012)