Attachment 3 finalrpt05.pdf

OSHA Data Initiative Collection Quality Control:

Analysis of Audits on CY 2005 Employer

Injury and Illness Recordkeeping

Task Order No. 22 Option Year 4 Contract No. J-9-F-3-0015

FINAL REPORT

September 5, 2008

Prepared for:

Office of Statistical Analysis Occupational Safety and Health Administration Washington, DC

Prepared by:

Eastern Research Group, Inc.

Lexington, MA National Opinion Research Center Chicago, IL

Final Report, September 2008

CONTENTS

SUMMARY

Purpose and Objectives of the Audits

Audit Methodology and Analytical Approach

Findings

Conclusions

APPENDICES

A. Data Tables on Injury and Illness Recordkeeping Accuracy from Audits on CY 2005 Employer Records

B. List of OSHA Data Initiative Collection Quality Reports and Related Studies

C. Background on the OSHA Injury and Illness Recordkeeping Audit Program

D. Tracking Status Codes Used in Processing Data for ODI Submissions

E. OSHA Directive (coversheet and url only): Audit and Verification Program of Occupational Injury and Illness Records (CPL 02-00-138)

Final Report, September 2008 1

SUMMARY

Purpose and Objectives of Audits

• Purpose. Under the audit program, OSHA annually conducts onsite audits of employer injury and illness records as a key method of verifying the accuracy of data submitted for the OSHA Data Initiative (ODI). OSHA conducts an analysis of the annual audits and estimates the extent of employer compliance with OSHA recordkeeping requirements defined in 29 CFR 1904. The audit program has been operational since 1998.

For data tables from the analysis of audits on CY 2005 recordkeeping, see Appendix A.

For a list of audit analysis reports and others related to ODI data quality, see Appendix B.

For general background on the audit program, see Appendix C.

• Objectives. The primary objectives of this year’s analysis were to:

o Estimate CY 2005 employer injury/illness recordkeeping accuracy nationwide based on OSHA recordkeeping audits conducted according to an established protocol at a sample of non-construction establishments drawn from the standard ODI universe.

o Continue implementing a three-year analysis cycle for audits on employer recordkeeping that includes two interim-year summary analyses followed by a comprehensive reporting-year analysis for submission to OMB. This year’s analysis addresses the second interim year of a three-year cycle.

Audit Methodology and Analytical Approach

• Sample selection from a standard ODI universe. This was the seventh year in which OSHA implemented the audit program by selecting a sample of audit establishments from a standard ODI universe. The standard universe covers all years of the ODI to include establishments either in OSHA Federal jurisdiction States or in State Plan States participating in the ODI, with total employment of 40 or more, and in any SIC code ever selected for the ODI collection with the exception of SIC 53 (General Merchandise Stores) and SIC 806 (Hospitals); construction is not included.

• Summary of OSHA’s sample selection steps:

1. Selected an initial sample from a standard ODI universe that included 109,393 establishments.

2. Included 400 establishments from the initial sample in the CY 2005 ODI collection universe.

3. At completion of the ODI collection for CY 2005, eliminated 127 establishments from the sample that did not meet audit program requirements (e.g., establishment is in a State Plan State not participating in the particular year’s audit program, or

Final Report, September 2008 2 establishment’s CY 2005 submission was not rated by OSHA as OK-verified).

(See Appendix D for a list of tracking codes used in processing ODI submissions.)

4. Assigned the remaining sample of 273 establishments for an audit.

5. Assigned 45 replacements for establishments where audits could not be conducted

(e.g., establishment had gone out of business, or establishment is a headquarters location with office workers only).

6. Eliminated 7 of the completed audits due to divergence from the protocol in conducting the audit. (21 of the assigned audits were not conducted.)

• State Plan State participation. The participating State Plan States in the tenth-year of the audit program were Arizona, California, Indiana, Iowa, Kentucky, Maryland, Minnesota, New Mexico, North Carolina, Utah, and Virginia.

• Audit protocol. Audits on CY 2005 records were conducted under CPL 02-00-138, which did not change the protocol for conducting the audits over past years. As of the CY 2004 audits, the audit program CPL has been upgraded from a Notice to an Instruction. (As a result, any year-specific changes to the protocol will be communicated to field staff by the National office in a memo.) Also, no changes were made to the ORAA software that have significance for how the audits were conducted.

The audit protocol (CPL 02) is presented as Appendix E (cover sheet and url only).

• Analytical approach. OSHA implemented the same general approach for analyzing the results of the establishment audits as was used for previous years of the program (e.g., comparing characteristics of the sample to the universe, calculating universe estimates for overall accuracy, performing a case-level analysis, and comparing submitted and onsite OSHA Log data).

Findings

• Audits available for the analysis. After following the sample selection steps listed above, a total of 245 audits were available for use in the universe estimates, case-level analysis, and comparison of onsite and submitted data. (This year OSHA did not reach the methodology’s target of 250 establishment audits available for conducting the analysis.)

• Distribution of audit establishments across the standard universe. Overall, the sample of audited establishments closely matched the distribution of establishments in the standard ODI universe by size group and at the 2-digit SIC level, reflecting the effect of implicit stratification. OSHA applied Pearson’s Chi-Square test for goodness of fit to the industry match and observed no significant deviations from fit (Chi-Square = 18.63, df = 30, n.s.).

The test was not applied to the comparison by size groupings because of the small number of grouping to compare.

See Appendix A for details:

Final Report, September 2008 3

Table 1 (size group comparison) and Table 2 (industry comparison).

• Recordkeeping accuracy universe estimates. The universe estimates analysis focuses on the types of recording errors that affect an employer’s injury and illness rate—that is, the underrecording of total recordable cases and the underrecording or misrecording of DART cases (days away from work, restriction, or transfer injury/illness cases).

Generalizing from the sample of establishments audited for CY 2005 recordkeeping, the percent of establishments classified with accurate recordkeeping (at-or-above the 95 percent threshold) for the standard ODI universe is above 90 percent for both total recordable injury/illness cases (94.69%, SE 1.43 %) and DART injury/illness cases (93.47%, SE 1.57 %). Further, based on 95 percent confidence intervals for the two estimates, the percentages for total recordable and DART are not statistically different. Overall these results are consistent with the level of recording accuracy observed previously. (For instance, accuracy estimates for audits on CY 2004 recordkeeping were 95.70% (SE 1.26 %) for total recordable and 95.31% (SE 1.32 %) for DART.)

o Note on accuracy estimates across the two most recent years. OSHA also applied a statistical test to the accuracy estimates from this year and last year’s analysis.

The test found no significant difference in the means for either total recordable or DART cases.

o Note on universe estimates. Consistent with the definition of the standard ODI universe, the accuracy estimates from the sample of audits can be generalized to all ODI establishments nationwide, except those in SICs 53 and 806. (The estimates should not be generalized to all of the nation’s workplaces because the ODI focuses on selected high-rate industries and excludes establishments with fewer than 40 employees. Also, not all State Plan States participate in the ODI or the audit program.)

As a separate step in the universe estimates analysis, OSHA also examines the overrecording of cases. Overrecorded cases are those cases found on the employer’s Log that the auditor has determined are non-recordable based on a review of employee records during the audit.

Overall, this year’s results on the overrecording of cases are consistent with the level observed previously.

Table 3 (accuracy estimates by TR and DART from audits on CY 2005 recordkeeping).

Item on Overrecording of Cases (p. A-10)

Conclusions

• Consistency of the findings. The interim year analysis indicates that recordkeeping accuracy

(as defined for the audit program) for CY 2005 employer records is consistent with the high level of accuracy found since the beginning the audit program. This finding of a reasonable

Final Report, September 2008 4 level of accuracy supports OSHA’s continued use of the ODI data for targeting enforcement and compliance assistance interventions and for measuring Agency performance in reducing workplace injuries and illnesses.

Table 4 (accuracy estimates across the most recent past 2 years of the audit program).

Final Report, September 2008 A-1

Appendix A

Data Tables on Injury and Illness Recordkeeping Accuracy from Audits on CY 2005 Employer Records

Final Report, September 2008 A-2

Table 1 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Number and Percent of Establishments in the Recordkeeping Audit Sample and the Standard ODI Universe by Establishment Size Group

Audit Samplea Establishments

Standard ODI Universeb

Establishment Size Group (average number of employees)

Number

Percentc of Sample

Number

Percentc of Universe

All Small (40-99)d 124 50.61 60,444 55.25

Small (40-49)e 30 12.24 17,632 16.12

Medium (100-249) 87 35.51 34,212 31.27

Large (≥250) 34 13.88 14,737 13.47

All Sizes 245 100 109,393 100

Note on representativeness of sample: Overall, the sample of audited establishments is similar to the distribution of establishments in the standard ODI universe by size group, reflecting the effect of implicit stratification. OSHA did not assess further this apparent relationship, noting (as pointed out by Hays, W. L. [1994]. Statistics [5th ed.].

Orlando, FL: Harcourt Brace & Co.) that Pearson’s Chi-Square test would not provide a reliable assessment of goodness of fit, given that only three size categories are available (i.e., the establishment size grouping of “all small,” medium, and large). This test provides a reasonable approximation only when the number of categories available for conducting the comparison—size or industry categories in this analysis—is reasonably large.

a. The audit sample is limited to establishment audits that OSHA assigned from the original sample of establishments, as drawn from the standard ODI universe, and that OSHA determined were usable for the analysis after confirming that the audits were conducted according to established recordkeeping audit procedures (see CPL 2 in Appendix E). Establishments in the original sample were assigned for an audit if they were under the OSHA Federal jurisdiction or in one of the 11 State Plan States that voluntarily participated in the audit program, and if their CY 2005 OSHA Data Initiative (ODI) submission was OK verified. For the comparison in Table 1, establishment size group information for the audit sample establishments was derived from the employer-submitted 2005 ODI data.

b. The standard ODI universe includes all establishments that are in States participating in the ODI, have 40 or more employees, and are in one of the SICs selected for any of the ODI collectionsCexcept SIC 53 (General Merchandise Stores ) and SIC 806 (Hospitals). Because OSHA has not collected ODI data from all establishments in the standard ODI universe, for the comparison in Table 1, establishment size group information for establishments in the standard ODI universe was derived from Dun & Bradstreet data.

c. Because of rounding, percentages may not add to 100.

d. The “all small” (40-99) and “small” (40-49) establishment size groups include 2 establishments with an average employment of less than 40 employees. These establishments had total employment of 148 and 41 for the year, but average employment during 2005 was determined to be 4 and 34 respectively.

e. The “small” size group is a subset of the “all small” size group.

Final Report, September 2008 A-3

Table 2 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Number and Percent of Establishments in the Recordkeeping Audit Sample and the Standard ODI Universe by Industry (2-digit SIC) Sorted by Number of Establishments in the Universe

Standard ODI Universeb Establishments

SIC Code (2-digit level) and Industry

Number Percentc of Sample Number Percentc of Universe

80 Health Services 32 13.06 13,692 12.52

42 Trucking and Warehousing 21 8.57 9,102 8.32

35 Machinery, Except Electrical 20 8.16 8,817 8.06

34 Fabricated Metal Products 21 8.57 7,960 7.28

20 Food and Kindred Products 13 5.31 6,171 5.64

36 Electric and Electronic Equipment 12 4.9 6,038 5.52

30 Rubber And Misc. Plastics Products 9 3.67 5,163 4.72

51 Wholesale Trade-Nondurable Goods 8 3.27 4,714 4.31

28 Chemicals and Allied Products 9 3.67 4,630 4.23

37 Transportation Equipment 10 4.08 3,702 3.38

50 Wholesale Trade-Durable Goods 8 3.27 3,638 3.33

52 Building Materials & Garden Supplies 13 5.31 3,478 3.18

38 Instruments and Related Products 5 2.04 3,450 3.15

24 Lumber and Wood Products 11 4.49 3,424 3.13

27 Printing and Publishing 7 2.86 3,269 2.99

26 Paper and Allied Products 6 2.45 3,175 2.9

32 Stone, Clay, and Glass Products 8 3.27 2,939 2.69

33 Primary Metal Industries 9 3.67 2,816 2.57

25 Furniture and Fixtures 4 1.63 2,125 1.94

23 Apparel and Other Textile Products 5 2.04 2,124 1.94

39 Misc. Manufacturing Industries 3 1.22 2,115 1.93

22 Textile Mill Products 3 1.22 1,698 1.55

49 Electric, Gas, and Sanitary Services 2 0.82 1,479 1.35

45 Transportation by Air 2 0.82 879 0.8

Final Report, September 2008 A-4

Establishments

Standard ODI Universeb Establishments

SIC Code (2-digit level) and Industry

Number Percentc of Sample Number Percentc of Universe

01 Agricultural Production-Crops 2 0.82 522 0.48

29 Petroleum and Coal Products 0 0 501 0.46

02 Agricultural Production-Livestock 0 0 498 0.46

31 Leather and Leather Products 0 0 305 0.28

44 Water Transportation 0 0 305 0.28

43 United States Postal Service 0 0 297 0.27

07 Agricultural Services 1 0.41 155 0.14

47 Transportation Services 1 0.41 124 0.11

21 Tobacco Manufacturers 0 0 88 0.08

All Manufacturing SICs 155 63.27 70,378 64.34

All Non-Manufacturing SICs 90 36.73 39,015 35.66

All SICs 245 100 109,393 100

Note on representativeness of sample: Overall, the sample of audited establishments appears representative of the standard ODI universe by industry, reflecting the effect of implicit stratification. OSHA further evaluated and supported this finding with Pearson’s Chi-Square test for goodness of fit, using the many more categories available for this comparison than for the size category comparison. In applying the test, no significant deviations from fit were observed (Chi-Square = 18.63, df = 30, n.s.).

a. The audit sample is limited to establishment audits that OSHA assigned from the original sample of establishments, as drawn from the standard ODI universe, and that OSHA determined were usable for the analysis after confirming that the audits were conducted according to established recordkeeping audit procedures (see CPL 2 in Appendix F). Establishments in the original sample were assigned for an audit if they were under the OSHA Federal jurisdiction or in one of the 11 State Plan States that voluntarily participated in the audit program, and if their CY 2005 OSHA Data Initiative (ODI) submission was OK verified. For the comparison in Table 2, establishment industry information for the audit sample establishments was derived from the employer-submitted 2005 ODI data.

b. The standard ODI universe includes all establishments that are in States participating in the ODI, have 40 or more employees, and are in one of the SICs selected for any of the ODI collectionsCexcept SIC 53 (General Merchandise Stores ) and SIC 806 (Hospitals). Because OSHA has not collected ODI data from all establishments in the standard ODI universe, for the comparison in Table 2, industry information for establishments in the standard ODI universe was derived from Dun & Bradstreet data.

c. Because of rounding, percentages may not add to 100.

Final Report, September 2008 A-5

Table 3 Universe Estimates for OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Number and Percent* of Establishments Classified as Accurate in Recording the Number of Total Recordable and Days Away, Restriction, or Transfer (DART) Injury and Illness Cases with the Standard Error of the Estimate

2005 AUDIT RESULTS

Type of Case Number of establishments classified with accurate recording (at-or-above the 95% threshold of accuracy)

Percent of establishments classified with accurate recording (at-or-above the 95% threshold of accuracy)

Standard error of the estimate (percent)

232 / 245 (13 below) 94.69% 1.43% Total Recordable

229 / 245 (16 below) 93.47% 1.57% DART

* The percent of establishments “at or above” the 95% threshold of accuracy calculated from the sample also provides an estimate that can be extrapolated to the standard ODI universe (i.e., establishments nationwide that are in States participating in the ODI, have 40 or more employees, and are in one of the SICs selected for any of the ODI collections—except SIC 53 (General Merchandise Stores ) and SIC 806 (Hospitals)).

Note: The standard error of the estimate was calculated using the simple random sampling variance estimator.

Final Report, September 2008 A-6

Table 4 Universe Estimates for OSHA Audits on CY 2003 and CY 2004 Recordkeeping:

Number and Percent* of Establishments Classified as Accurate in Recording the Number of Total Recordable and Days Away, Restriction, or Transfer (DART) Injury and Illness Cases with the Standard Error of the Estimate

2003 AUDIT RESULTS** 2004 AUDIT RESULTS

Type of Case

Number of establishments classified with accurate recording (at-or-above the 95% threshold of accuracy)

Percent of establishments classified with accurate recording (at-or-above the 95% threshold of accuracy)

Standard error of the estimate

(percent)

Number of establishments classified with accurate recording (at-or-above the 95% threshold of accuracy)

Percent of establishments classified with accurate recording (at-or-above the 95% threshold of accuracy)

Standard error of the estimate

(percent)

232 / 251 245 / 256 (19 below) 92.43% 1.66% (11 below) 95.70% 1.26% Total

Recordable

226 / 251 244 / 256 (25 below) 90.04% 1.88% (12 below) 95.31% 1.32% DART

* The percent of establishments “at or above” the 95% threshold of accuracy calculated from the sample also provides an estimate that can be extrapolated to the standard ODI universe (i.e., establishments nationwide that are in States participating in the ODI, have 40 or more employees, and are in one of the SICs selected for any of the ODI collections—except SIC 53 (General Merchandise Stores ) and SIC 806 (Hospitals)).

** The standard ODI universe was further refined as of the CY 2001 program by adding SIC 43 (U.S. Postal Service), with 297 establishments.

Note: The standard error of the estimate was calculated using the simple random sampling variance estimator.

Final Report, September 2008 A-7

Table 5 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Number and Percent of Establishments Below the Threshold of Accuracy for Total Recordable Cases by Establishment Size Group and Manufacturing vs. Non-Manufacturing

Industry Category

Manufacturing Non-Manufacturing

Establishment Size Category (average number of employees)

Number Percent Number Percent

All Small (40-99)* 4 / 76 5.26 0 / 48 0.00

Small (40-49)** 1 / 20 5.00 0 / 10 0.00

Medium (100-249) 4 / 50 8.00 3 / 37 8.11

Large (≥250) 1 / 29 3.45 1 / 5 20.00 9 / 155 4 / 90

Total (146 pass / 155)

5.81 (86 pass / 90)

4.44

* The “all small” (40-99) and “small” (40-49) establishment size groups include 2 establishments with an average employment of less than 40 employees. These establishments had total employment of 148 and 41 for the year, but

** The “small” size group is a subset of the “all small” size group.

Final Report, September 2008 A-8

Table 6 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Number and Percent of Establishments Below the Threshold of Accuracy for Days Away, Restriction, or Transfer (DART) Injury and Illness Cases by Establishment Size Group and Manufacturing vs. Non-Manufacturing

Industry Category

Manufacturing Non-Manufacturing

Establishment Size Category (average number of employees)

Number Percent Number Percent

All Small (40-99)* 5 / 76 6.58 0 / 48 0.00

Small (40-49)** 1 / 20 5.00 0 / 10 0.00

Medium (100-249) 2 / 50 4.00 5 / 37 13.51

Large (≥250) 2 / 29 6.90 2 / 5 40.00

9 / 155 7 / 90 Total

(146 pass / 155) 5.81

(83 pass / 90) 7.78

* The “all small” (40-99) and “small” (40-49) establishment size groups include 2 establishments with an average employment of less than 40 employees. These establishments had total employment of 148 and 41 for the year, but

Final Report, September 2008 A-9

Item on Overrecording of Cases from OSHA Audits on CY 2005 Injury and Illness Recordkeeping

• Overall. A total of 99 entries (93 injuries, 4 illnesses, and 2 entries that did not indicate either) were found on employers’ Logs for incidents that are not considered OSHA-recordable cases. These overrecorded cases were distributed across 60 establishments. At 37 of these 60 establishments, only one instance of overrecording was found.

Only 14 of these 99 overrecorded cases were classified as DART cases by employers.

These 14 overrecorded DART cases were distributed across 11 establishments.

• Total recordable cases. Overall, 229 of 245 (96.48%) establishments were at-or-above the

95 percent threshold of accuracy with respect to overrecording.

Of the 232 establishments at-or-above the 95 percent threshold of accuracy with respect to underrecording of recordable cases, 218 (93.97%) were found to be at-or-above the threshold with respect to overrecording.

Two of the 13 establishments below the 95 percent threshold of accuracy with respect to underrecording also tested below the 95 percent threshold of accuracy for overrecording (84.62% accurate).

• DART cases. Overall, 243 of 245 (99.18%) establishments were at-or-above the 95 percent threshold of accuracy with respect to overrecording.

Of the 229 establishments at-or-above the 95 percent threshold of accuracy with respect to underrecording of DART cases, 227 (99.13%) were found to be at-or-above the threshold with respect to overrecording.

None of the 16 establishments below the 95 percent threshold of accuracy for DART underrecording tested below the 95 percent threshold of accuracy for DART overrecording (100% accurate).

Final Report, September 2008 A-10

Table 7 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Number and Percent of Recordable Injury and Illness Cases Identified by Auditors by Type of Recordkeeping Errors*

Recordable Cases Type of Recording Error

Number Percent**

Not Recorded 60 / 631 9.51

DART Recorded as Non-DART 48 / 631 7.61

Non-DART Recorded as DART 2 / 631 0.32

Total Recording Errors (above) 110 / 631 17.43

Total Cases with None of the Above Errors 521 / 631 82.57

Total 631 100

* The frequencies in this table are unweighted and should not be used to draw broad conclusions about the recordkeeping audit universe.

** Because of rounding, percentages might not add to 100.

Final Report, September 2008 A-11

Table 8 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Number and Percent of Recordable Injury and Illness Cases Identified by Auditors and Not Recorded on the Employer’s Log*

Injury/Illness Category

Type of Case Number of Cases Not Recorded

Number of Cases Discovered by Auditor

Percent of Category Not Recorded

Percent of All Cases Not Recorded

Non-Days Away, Restriction, or Transfer (DART) Cases 22 167 22 / 167 = 13.17 22 / 60 = 36.67

Days Away From Work (DAFW) Only 13 123 10.57 21.67

Restricted Work Activity or Transfer (RWA) Only 13 214 6.07 21.67

DAFW and RWA 3 75 4.00 5.00

Injuries

All Types for Injuries (Total) 51 579 8.81 85.00

Non-DART Cases 6 26 6 / 26 = 23.08 6 / 60 = 10.00

DAFW Only 2 11 18.18 3.33

RWA Only 1 11 9.09 1.67

DAFW and RWA 0 4 0 0

Illnesses

All Types for Illnesses (Total) 9 52 17.31 15.00

Non-DART Cases 28 193 28 / 193 = 14.51 28 / 60 = 46.67

DAFW Only 15 134 11.19 25.00

RWA Only 14 225 6.22 23.33

DAFW and RWA 3 79 3.80 5.00

Injuries and Illness Combined

All Types (Total) 60 631 9.51 100

* The frequencies in this table are unweighted and should not be used to draw broad conclusions about the recordkeeping audit universe.

Final Report, September 2008 A-12

Table 9 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Number and Percent of Recordable Days Away, Restriction, or Transfer (DART) Injury and Illness Cases Identified by Auditors and Recorded on the Employer’s Log as Non-DART Cases*

Injury/Illness Category Type of Case

Number Cases Recorded as Non-DART

Cases

Number Cases Discovered by

Auditor

Percent of Category Not Recorded as DART Case

Percent of All DART Cases Recorded as Non-DART

Cases

Days Away from Work (DAFW) Only 12 123 12 / 123 = 9.76 12 / 48 = 25.00

Restricted Work Activity or Transfer (RWA) Only 29 214 13.55 60.42

DAFW and RWA 6 75 8.00 12.50

Injuries

All Types for Injuries (Total) 47 412 11.41 97.92

DAFW Only 0 11 0 / 11 = 0 0 / 48 = 0

RWA Only 1 11 9.09 2.08

DAFW and RWA 0 4 0 0 Illnesses

All Types for Illnesses (Total) 1 26 3.85 2.08

DAFW Only 12 134 12 / 134 = 8.96 12 / 48 = 25.00

RWA Only 30 225 13.33 62.50

DAFW and RWA 6 79 7.59 12.50 Injuries and Illnesses Combined

All Types (Total) 48 438 10.96 100

* The frequencies in this table are unweighted and should not be used to draw broad conclusions about the recordkeeping.

Final Report, September 2008 A-13

Table 10 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Results of the Comparison of Total Recordable Injury and Illness Cases and Days Away, Restriction, or Transfer (DART) Injury and Illness Cases Submitted to OSHA for the Data Collection with Data on the Employer’s Log as Found During Audits by Establishment Size

Establishment Comparison Results Establishment Size Group Total Recordable Injury and Illnesses Cases DART Injury and Illness Cases

Audit Less Audit Same Audit More Audit Less Audit Same Audit More (average number of employees)

Number Percent Number Percent Number Percent Number Percent Number Percent Number Percent

All Small (40-99)* (124 establishments) 7 5.65 110 88.71 7 5.65 7 5.65 110 88.71 7 5.65

Small (40-49)** (30 establishments) 2 6.67 27 90.00 1 3.33 2 6.67 27 90.00 1 3.33

Medium (100-249) (87 establishments) 9 10.34 70 80.46 8 9.20 6 6.90 74 85.06 7 8.05

Large (≥250) (34 establishments) 1 2.94 24 70.59 9 26.47 0 0 21 61.76 13 38.24

ALL SIZES

(245 establishments) 17 6.94 204 83.27 24 9.80 13 5.31 205 83.67 27 11.02

* The “all small” (40-99) and “small” (40-49) establishment size groups include 2 establishments with an average employment of less than 40 employees. These establishments had total employment of 148 and 41 for the year, but average employment during 2005 was determined to be 4 and 34 respectively.

Final Report, September 2008 A-14

Table 11 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Results of the Comparison of Hours Worked Data Submitted to OSHA for the Data Collection with Hours Worked Provided During Recordkeeping Audits by Establishment Size

Establishment Size Group Hours Worked

Audit Less Audit Same Audit More (average number of employees)

Number Percent Number Percent Number Percent

All Small (40-99)* (124 establishments) 14 11.29 90 72.58 20 16.13

Small (40-49)** (30 establishments) 1 3.33 24 80 5 16.67

Medium (100-249) (87 establishments) 11 12.64 64 73.56 12 13.79

Large (≥250) (34 establishments) 2 5.88 30 88.24 2 5.88

ALL SIZES

(245 establishments)

27 11.02 184 75.1 34 13.88

* The “all small” (40-99) and “small” (40-49) establishment size groups include 2 establishments with an average employment of less than 40 employees. These establishments had total employment of 148 and 41 for the year, but average employment during 2005 was determined to be 4 and 34 respectively.

Final Report, September 2008 A-15

Table 12 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Results of the Comparison of Total Recordable Injury and Illness Cases and Days Away, Restriction, or Transfer (DART) Injury and Illness Cases Submitted to OSHA for the Data Collection with

Data on the Employer’s Log as Found During Recordkeeping Audits by Industry Type (Manufacturing vs. Non-Manufacturing)

Total Recordable Injury and Illnesses Cases DART Injury and Illness Cases

Audit Less Audit Same Audit More Audit Less Audit Same Audit More Industry Type

Number Percent Number Percent Number Percent Number Percent Number Percent Number Percent

All Manufacturing SICs (155 establishments) 10 6.45 136 87.74 9 5.81 7 4.52 134 86.45 14 9.03

All Non-Mfg SICs (90 establishments) 7 7.78 68 75.56 15 16.67 6 6.67 71 78.89 13 14.44

ALL SIZES

(245 establishments)

17 6.94 204 83.27 24 9.80 13 5.31 205 83.67 27 11.02

Final Report, September 2008 A-16

Table 13 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Results of the Comparison of Hours Worked Data Submitted to OSHA for the Data Collection with Hours Worked Provided During Recordkeeping Audits by Industry Type (Manufacturing vs. Non-Manufacturing)

Hours Worked Audit Less Audit Same Audit More

Industry Type

Number Percent Number Percent Number Percent

All Manufacturing SICs (155 establishments) 15 9.68 121 78.06 19 12.26

All Non-Mfg SICs (90 establishments) 12 13.33 63 70.00 15 16.67

ALL SIZES

(245 establishments) 27 11.02 184 75.10 34 13.88

Final Report, September 2008 A-17

Table 14 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Primary Reasons for Differences Between the Injury and Illness Data Submitted to OSHA for the Data Collection and Injury and Illness Data on the Employer’s Log Provided During the Recordkeeping Audits

Primary Reason for Difference* Reason(s) Given for Difference(s) in Injury and Illness Data

Number Percent**

Log change(s) or correction(s) made after the data were submitted, reflecting new information brought to the attention of recordkeeper(s) pertaining to cases on the Log

23 41.07

Checkmark error(s) 9 16.07

Error(s) associated with reporting data from the wrong facility or facilities 7 12.50

Other reasons 6 10.71

Survey processing edit(s) (employer’s Log was otherwise the same as the submitted data) 5 8.93

Clerical error(s) (e.g., typo or transposition) 4 7.14

Addition error(s) 2 3.57

Blank or auditor could not determine reason 0 0

Error(s) associated with omitting reporting components (e.g., temporary labor, salaried employees) 0 0

Establishment Totals*** 56 100

* The audit software also provides fields for noting any secondary reasons given to explain the differences. This analysis considers only the primary reasons.

** Because of rounding, percentages might not add to 100.

*** Although 56 establishments provided a primary reason for a difference (as noted in this table), the difference resulted in a change in total recordable injury and illnesses case counts for only 41 establishments (see total of Audit Less and Audit More for Total Recordable Injury and Illnesses Cases in Table 10). In the 15 instances where there was no impact on the total case count, Log column differences in effect canceled each other out.

Final Report, September 2008 A-18

Table 15 OSHA Audits on CY 2005 Injury and Illness Recordkeeping:

Primary Reasons for Differences Between the Data on Hours Worked Submitted to OSHA for the Data Collection and Data on Hours Worked

Provided During the Recordkeeping Audits

Primary Reason for Difference* Reason(s) Given for Difference(s) in Hours Worked Data

Number Percent**

Error(s) associated with omitting reporting components (e.g., temporary labor, salaried employees) 20 35.09

Other reasons 16 28.07

Estimated value instead of actual value 13 22.81

Error(s) associated with reporting from wrong facility or facilities 8 14.04

Blank or auditor could not determine reason 0 0.00

Establishment Totals 57 100

* The audit software also provides fields for noting any secondary reasons given to explain the differences. This analysis considers only the primary reasons.

** Because of rounding, percentages might not add to 100.

Final Report, September 2008 B-1

Appendix B

List of OSHA Data Initiative Collection Quality Reports and Related Studies

The following analyses have been conducted on OSHA’s audit program:

• OSHA Data Collection Validation Study: Pilot Test on the Data Collection

Quality and Verification of Employer Injury and Illness Records. September 12, 1997 (Final Report). Eastern Research Group, Inc. (Contract No. J-9-F-3-0043:

Task Order No. 5, Option Year Two.)

• OSHA Data Initiative Collection Quality Control: Analysis of Audits on 1996

Employer Injury and Illness Recordkeeping. September 17, 1998 (Final Report).

The Lexington Group, Eastern Research Group, Inc., and the National Opinion Research Center. (Contract No. J-9-F-7-0043: Task Order No. 7, Base Year.)

• OSHA Data Initiative Collection Quality Control: Analysis of Audits on 1997

Employer Injury and Illness Recordkeeping. August 23, 1999 (Final Report).

Eastern Research Group, Inc. and the National Opinion Research Center.

(Contract No. J-9-F-7-0053: Task Order No. 7, Option Year One.)

• OSHA Data Initiative Collection Quality Control: Analysis of Audits on 1998

Employer Injury and Illness Recordkeeping. September 29, 2000 (Final Report).

(Contract No. J-9-F-7-0053: Task Order No. 17, Option Year Two.)

• OSHA Data Initiative Collection Quality Control: Analysis of Audits on 1999

Employer Injury and Illness Recordkeeping. September 28, 2001 (Final Report).

(Contract No. J-9-F-7-0053: Task Order No. 24, Option Year Three.)

• OSHA Data Initiative Collection Quality Control: Analysis of Audits on 2000

Employer Injury and Illness Recordkeeping. September 27, 2002 (Final Report).

(Contract No. J-9-F-7-0053: Task Order No. 33, Option Year Four.)

• OSHA Data Initiative Collection Quality Control: Analysis of Audits on 2001

Employer Injury and Illness Recordkeeping. December 5, 2003 (Final Report).

(Contract No. J-9-F-3-0015: Task Order No. 1, Base Year.)

• OSHA Data Initiative Collection Quality Control: Analysis of Audits on 2002

Employer Injury and Illness Recordkeeping—Interim Year Analysis in Multi-Year Reporting Cycle. September 30, 2005 (Final Report). Eastern Research Group, Inc. and the National Opinion Research Center. (Contract No. J-9-F-3-0015: Task

Final Report, September 2008 B-2

Order No. 2, Option Year 1.)

• OSHA Data Initiative Collection Quality Control: Analysis of Audits on 2003

Employer Injury and Illness Recordkeeping—Reporting Year Analysis in Multi- Year Reporting Cycle. September 7, 2006 (Final Report). Eastern Research Group, Inc. and the National Opinion Research Center. (Contract No. J-9-F-3- 0015: Task Order No. 5, Option Year 2.)

• OSHA Data Initiative Collection Quality Control: Analysis of Audits on 2004

Employer Injury and Illness Recordkeeping—Summary Report Year. June 18, 2007 (Final Report). Eastern Research Group, Inc. and the National Opinion Research Center. (Contract No. J-9-F-3-0015: Task Order No. 11, Option Year 3.)

Studies related to ODI collection quality include the following:

• Sample Design for a Statistically Valid Evaluation of Accuracy and Completeness of an Establishment’s OSHA-Mandated Employee Records. 1996. The National Opinion Research Center.

• OSHA Data Collection Validation Study: Initial Assessment of the Accuracy of the OSHA-Collected Data—An Analysis of the Data Edit Reports and a Review of State Agency Impressions. February 1997 (Final Report). Eastern Research Group, Inc. (Contract No. J-9-F-3-0043: Task Order No. 5, Option Year Two.)

• OSHA Data Collection Validation Study: Descriptive Characteristics of the 1995

OSHA-Collected Data and Comparison with the Bureau of Labor Statistics’ Annual Survey on Occupational Injuries and Illnesses. September 12, 1997 (Final Report). Eastern Research Group, Inc. (Contract No. J-9-F-3-0043: Task Order No. 5, Option Year Two.)

• OSHA Data Collection Validation Study: Issues with Creating a Matched File for Comparing the OSHA 200 Log Data Collected by Compliance Officers During Onsite Interventions with the Injury/Illness Data from the OSHA Log Data Collection. September 12, 1997 (Final Report). Eastern Research Group, Inc. (Contract No. J-9-F-3-0043: Task Order No. 5, Option Year Two.)

• A Summary of Findings on the Correlation of Establishment Injury/Illness Rate

Data from the OSHA Data Initiative and the IMIS Log Data. September 25, 2000 (Final Report). The Lexington Group, Eastern Research Group, Inc., and Dr.

Wayne Gray. (Contract No. J-9-F-7-0043: Task Order No. 23, Subtask 1, Option Year Two.)

• A Summary of Findings on the Correlation of Establishment Injury/Illness Rate

Data from the OSHA Data Initiative and the BLS Annual Survey. September 25, 2000 (Final Report). The Lexington Group, Eastern Research Group, Inc., and Dr.

Final Report, September 2008 B-3

Wayne Gray. (Contract No. J-9-F-7-0043: Task Order No. 23, Subtask 2, Option Year Two.)

Final Report, September 2008 C-1

Appendix C

Background on the OSHA Injury and Illness Recordkeeping Audit Program

Program-Related Analyses and Key Findings

As an initial step in assessing the quality of information compiled by OSHA’s Data Initiative (ODI) collection system, the Agency conducted two data validation studies in 1996:

• An analysis of the data collection system’s edit criteria results and commentary on data quality from State agencies assisting in the collection effort.

• Calculation of descriptive statistics on the collected data and comparison of the data with injury and illness data from the BLS Annual Survey.

Findings from the studies indicated that OSHA had implemented a credible system to provide the Agency with useful, establishment-specific data on occupational injuries and acute illnesses.

At the same time, the studies underscored the need for OSHA to continue efforts to ensure the quality of the OSHA-collected data. Under the audit program, OSHA conducts onsite audits of employer injury and illness records to verify the overall accuracy of source records, estimate the extent of employer compliance with the OSHA recordkeeping requirements defined in 29 CFR 1904, and assess the consistency between data on the employer’s Log and data submitted to the Agency under the ODI.

In 1997, OSHA conducted an audit pilot program in nine establishments to test the Agency’s protocol designed for efficient use of resources in performing recordkeeping audits.

The protocol is designed to save auditors time through the review of records for a statistical sampling of employees within an establishment and through use of the OSHA Recordkeeping Audit Assistant (ORAA) software system for streamlining the process of conducting, documenting, tracking, and analyzing the establishment audit.

Overall, OSHA’s analysis of the pilot test, which reviewed calendar year (CY) 1995 records, demonstrated the feasibility of the protocol for use in a larger audit program. In 1998, based on its experience with the pilot test, the Agency modified the protocol slightly for use in the first full-scale program for auditing employer injury and illness records. (That first year involved audits on CY 1996 records.) Similarly, for the next five years of the audit program, OSHA drew upon its earlier experience and made minor adjustments in implementation of the program for audits on establishments’ CY 1997, 1998, 1999, 2000, and 2001 records, respectively.

Final Report, September 2008 C-2

In summary, OSHA’s analyses of the first six years of the audit program found the following:

• The sample of establishments audited was representative of the sampling universe.

• The audit protocol, including sampling of employees within establishments, appears to provide OSHA with a feasible process to monitor the quality of employer injury and illness recordkeeping.

• The estimates of overall accuracy for total recordable and lost workday cases (i.e., establishments at-or-above the 95 percent threshold) suggest that the ODI collection currently provides reasonably accurate data that OSHA can use to help meet its program and performance measurement data needs. Related findings include:

o A small number of establishments account for most of the underrecorded and overrecorded cases. Errors are not widely distributed across the sample establishments.

o Differences found in comparing the audit data with the ODI data submitted to

OSHA result in very few changes of the targeting category status of establishments for either inspections or high-hazard notification letters.

o There is no evidence that small establishments have less accurate injury/illness records than medium or large size establishments.

The sixth year of the audit program (audits on CY 2001 recordkeeping) marked the last analysis of injury/illness recordkeeping under the old version of 29 CFR 1904. Subsequent annual audit program cycles will focus on records maintained by employers under the revised rule, which went into effect on January 1, 2002. The intention of the revisions made to the recordkeeping requirements is to simplify injury/illness recordkeeping for employers and contribute to the quality of establishment injury/illness data.

The seventh and eighth years of the audit program focused on CY 2002 and CY 2003 injury/illness recordkeeping, respectively, and provided preliminary reviews of accuracy in non-construction establishments under the first two years of the revised recordkeeping rule. For CY 2002 recordkeeping, the annual analysis indicated that recordkeeping accuracy was not significantly different than the results found in past years under the old rule. Similarly, the analysis on CY 2004 recordkeeping found that, overall, the universe estimates were consistent with the high level of accuracy found (i.e., above 90 percent) for employer injury and illness recordkeeping over previous years of the audit program.

Final Report, September 2008 C-3

Highlights of Annual Recordkeeping Audits and Analyses

First Year of the Program (Audits on CY 1996 Recordkeeping). The first year of the audit program followed OSHA’s pilot test of the audit protocol in nine establishments, which demonstrated its feasibility for use in a full-scale audit program. In this first year, OSHA implemented the protocol along with the use of an audit software tool as an efficient approach to conducting and documenting recordkeeping audits. Audits were conducted in a sample of establishments randomly selected from a universe of establishments under Federal OSHA jurisdiction that met the criteria for the 1996 ODI collection. In its analysis of the program’s first year, OSHA found the sample to be representative of the universe. Based on this finding, OSHA also estimated the overall accuracy of employer injury and illness recordkeeping in the Federal OSHA jurisdiction. The findings suggested that the ODI was providing reasonably accurate data for use by OSHA in meetings its program needs.

Second Year of Program (Audits on CY 1997 Recordkeeping). Notable differences in implementation of the second-year audit program included expanding the audit universe beyond the Federal OSHA jurisdiction to include establishments in State Plan States. Also, before selecting a sample of audit establishments, OSHA implemented implicit stratification of the universe by first sorting establishments on Standard Industrial Classification (SIC) code, followed by OSHA Region, and last by employment size. This approach is designed to provide sample establishments in similar proportions to their SIC, geographic, and size distribution in the universe. Compared to a simple random sampling approach, implicit stratification distributes the audit workload among the OSHA Regions better and balances the industry (manufacturing vs.

non-manufacturing/non-construction) and establishment size distributions for the analysis.

Third Year of Program (Audits on CY 1998 Recordkeeping). In the third year of the audit program, OSHA began to explore the use of a standard sampling universe to facilitate comparison of year-to-year estimates. OSHA also increased the number of establishments in the audit sample and the number of assigned audits in order to increase the likelihood that the number of audits available for analysis would be closer to the approximate target of 250.

Additionally, the third-year audit program’s coverage was expanded by including establishments with an average employment between 40 and 49 (compared to the previous cut-off at 50 in 1997 and 60 in 1995 and 1996) and by encouraging a greater number of State Plan States to participate.

Fourth Year of Program (Audits on CY 1999 Recordkeeping). For the fourth-year audit program, OSHA modified its approach for selecting audit establishments from a universe of establishments participating in the ODI in a specific year. Instead, OSHA selected a sample from a standard ODI universe that covered all years of the ODI. OSHA’s objective in sampling from a standard ODI universe was to establish a credible basis for generalizing the estimate of overall accuracy for an individual year’s employer injury and illness recordkeeping to ODI establishments nationwide. Additionally, use of a standard universe would anticipate the benefit of conducting year-to-year comparisons to assess recordkeeping under the revised rule.

In the first four years of the program, the analysis found that about 90 percent of establishments in the sampling universe for the specific year were estimated as having accurately

Final Report, September 2008 C-4 recorded the number of total recordable cases; about 88 percent of establishments were found to be accurate in recording lost workday cases.

Fifth Year of Program (Audits on CY 2000 Recordkeeping). For the fifth-year audit program, OSHA selected a sample for a second time from a standard ODI universe that covered all years of the ODI. This enabled OSHA to include a preliminary comparison of recordkeeping accuracy estimates for ODI establishments nationwide. The comparison indicated consistency in recordkeeping accuracy estimates across the fourth and fifth years of the audit program.

Interpretation of the comparison was limited somewhat because OSHA had further refined the definition of the standard ODI universe in the fifth year by excluding two industries that are only selectively included in the ODI. Nonetheless, the preliminary comparison provided potential baseline data for using such comparisons to assess recordkeeping under the revised rule.

Sixth Year of Program (Audits on CY 2001 Recordkeeping). For the sixth-year audit program, OSHA again selected a sample from a standard ODI universe that covered all years of the ODI. The analysis found consistency between CY 2000 and CY 2001 recordkeeping accuracy estimates for ODI establishments nationwide. Further, in applying a statistical test to the comparison of accuracy estimates, OSHA found no significant difference in the means for the two years, suggesting overall recordkeeping improvement. (An additional, minor refinement to the standard ODI universe should be noted regarding this year-to-year comparison; i.e., for the sixth year program, the Agency included SIC 43 (U.S. Postal Service)—now under OSHA jurisdiction—in the universe, which added 297 facilities.) In the fifth and sixth years of the program, the analysis found that about 95 percent of establishments in the sampling universe were estimated as having accurately recorded the number of total recordable cases; about 93 percent of establishments were found to be accurate in recording lost workday cases.

Also for the sixth year’s audit program, OSHA conducted a pilot test of audits at a sample of construction establishments using a protocol that addressed issues unique to the construction industry and its operation of temporary worksites. OSHA selected a sample of construction audit establishments from a universe of about 9,000 establishments that had submitted complete data for the CY 2001 ODI collection and met relevant criteria (e.g., operate under one of the three 2-digit construction SIC codes: SIC 15, SIC 16, or SIC 17).

In analyzing the results of pilot audits on establishments in construction industries, OSHA implemented the same general approach used for audits at non-construction establishments. Overall, the analysis found a slightly lower percent of construction establishments at-or-above the threshold of accuracy for both total recordable and lost workday cases in comparison to the accuracy estimates for non-construction establishments. The construction pilot findings indicate that the audit methodology developed for non-construction establishment can be implemented in construction establishments. At the same time, unique aspects of construction operations related to maintaining injury and illness records resulted in a recommendation that, if OSHA continues collecting data from construction SICs, the ODI construction universe and analysis of audits at construction establishments remain separate from the non-construction analysis.

Final Report, September 2008 C-5

Seventh Year of Program (Audits on CY 2002 Recordkeeping). For the seventh-year audit program, OSHA again selected a sample from a standard ODI universe. This analysis on CY 2002 recordkeeping provided a preliminary review of injury/illness recordkeeping accuracy in non-construction establishments under the first year of employer implementation of OSHA’s revised recordkeeping rule. The study indicated that recordkeeping accuracy in the first year under the revised recordkeeping rule is not significantly different than the results found in past years under the old rule.