Appendix F_Quality Control Manual.pdf

E-0

Appendix F:

Quality Assurance Manual

Cycle 7

E-1

QUALITY CONTROL

MANUAL

Federal Lands Highway

ROAD INVENTORY PROGRAM

CYCLE 7

Developed by the

U.S. Department of Transportation

Federal Highway Administration

Eastern Federal Lands Highway Division

E-2

INTRODUCTION

This document presents the Quality Control plan that will be applied to the EFLHD Road

Inventory Program (RIP) data and report for RIP Cycle 7. The purpose of this QC plan is to ensure high quality, accurate data in the databases and reports delivered. This plan is not intended to replace or supercede the contractor’s QC plan. Only to serve as information to contractor as to the types of tests and processes EFLHD intends to perform on any data submitted to them.

DCV POST-COLLECTION

QUALITY ASSURANCE PROCESS FLOW

Contractor develops and conducts quality checks on park data submission. Contractor delivers park data submission to EFLHD.

EFLHD conducts QC checks on park data submission and generates QC Report. FAIL PASS

EFLHD emails to Contractor:

• QC Report

EFLHD delivers QC Report to

Contractor detailing needed repairs.

E-3

QUALITY CONTROL REPORTS

Quality Control (QC) Reports delivered by email will be used as the means of communication between the contractor and EFLHD regarding data problems or data anomalies identified during data processing QC.

Quality Control can be defined as the process of identifying specific errors in each submittal. Quality Assurance focuses the management process that details how to achieve certain quality targets a submittal. The reports will detail the results of the quality control process. Additionally, it is through this QC process that EFLHD is able to identify errors and if necessary, adjust the process to ensure they do not happen again.

Along with each submittal, the contractor shall deliver a separate exceptions list, QC results database, or similar file that documents any known collection/processing issues, such as any known missing or problematic data or video.

After receiving the submittal from the contractor, EFLHD performs thorough checks on the data tables and video deliverables. EFLHD will document any non-conforming records in the data submittal and communicate these discoveries to the contractor.

It is a goal of EFLHD to be very transparent in their QC processes. The contractor is encouraged to ask questions and inquire as to how we perform a specific or broader check.

The quality checks described throughout this manual will be included in the QC Report, additional issues not included in this appendix may be identified by EFLHD for correction by the contractor.

The QC process is iterative, as each submittal will be subject to thorough and complete testing. The QC process is not complete until a park passes all the QC tests, or an exception is granted.

The QC process will be dynamic, and the tests described in this document may change, be enhanced or eliminated at EFLHD’s discretion. Newly developed quality checks not described in this document may also be added throughout the life of the contract. EFLHD will periodically update this document with these changes and provide the changes to the contractor.

The contractor is encouraged to use our QC protocols as a guideline and employ their own, more extensive QC procedures. The Contractor is expected to update their procedures as the contract progresses, and exceeding QC targets should be the goal

E-4

PARK DATA QUALITY CONTROL TESTS CONDUCTED

BY EFLHD

DATA INTEGRITY TESTS

The data integrity tests that are a part of the RIP QC plan are designed to test:

1. Table Design Consistency

2. Field Data Format Consistency

3. Data Completeness

4. Key Field Uniqueness

5. Data Reasonableness

6. Acceptable Data Range

7. Within-Field Data Consistency

8. Between-Field Data Consistency

9. Between-Table Data Consistency

1. Table Design Consistency checks will be used to confirm that the data delivered by the contractor are in the approved data table format, as specified in Appendix J: Database Data Dictionary. Examples of table design inconsistencies could be misspelled field names, missing fields, or extra fields added to the table.

2. Field Data Format Consistency checks will be used to confirm that data in each field follow the format specified in Appendix F: Database Data

Dictionary. Examples of data format inconsistencies could be numerical data provided in a text format.

3. Data Completeness checks will be used to identify data missing within or between the submission tables. Examples of the missing data within the table could be missing pavement distress data or distress indexes. An example of the missing data between the tables could be an index value that was not rolled up to the highest table of reporting.

4. Key Field Uniqueness checks will be used to assure that no duplicate records are provided for the records identified by a combination of the key fields. For some tables, key fields include road identifier and milepost identifier. An example of records with duplicate key fields could be more than one record with the same ROUTE_IDENT and BEG_MP.

5. Data Reasonableness checks will be used to test if data elements comply with the specified code lists. For example, DOR_FLAG must be either 0 or 1.

6. Acceptable Data Range checks will be used to test if data elements are within the expected range of values. For example, a value of a distress index cannot be more than 100 or less than 0.

E-5

7. Within-Field Data Consistency checks will be used to identify possible equipment malfunction during data collection or errors during data processing. An example of the within-field data consistency check will be a check for 7 or more consecutive records that have the same measured values. These records will be flagged and subjected to manual data review by EFLHD.

8. Between-Field Data Consistency checks will be used to evaluate data consistency within a table. An example of a between-field data consistency check is assuring that distress data reported in the PMS_20 table corresponds to the computed distress index in the same table.

9. Between-Table Data Consistency checks will be used to evaluate data consistency between different submission tables. For example, data in

PMS_MILE and PMS_TENTH tables should be derived from the

PMS_20 table.

The following sections provide detailed descriptions of the QC checks that EFLHD applies to each data submission table.

GEODATABASE TABLES

• CUMULATIVE_STATUS

• ROAD_CONDITION

• ROUTE_ID

• ROUTE_GPS

• ROUTE_GPS_C7

• WIDTH_MEASURE

QC Checks for all tables

Table Design Consistency Check: Table design should conform with Appendix F:

Database Data Dictionary.

Field Data Format Checks: Field values should conform with Appendix F: Database

Data Dictionary.

E-6

QC Checks for CUMULATIVE STATUS Table

Completeness Checks:

1. Record count > 0.

2. No Null (blank/empty) values in any fields.

Visual checks for reasonableness.

QC Checks for ROAD_CONDITION

3. Record count > 0.

4. No Null (blank/empty) values in any fields.

Key Field Uniqueness Checks:

1. No duplicate records defined by the combination of values from the following fields: [ROUTE_IDENT] & [BEG_MP]

Field Name QC Checks for ROAD_CONDITION Table

OBJECTID Data Reasonableness

GLOBAL_ID Data Reasonableness

AGENCY

Data Reasonableness

Between-Tables Data Consistency

UNIT_ID

Data Reasonableness

Between-Tables Data Consistency

ROUTE_IDENT

Data Reasonableness

Between-Tables Data Consistency

ROUTE_NUM

Data Reasonableness

Between-Tables Data Consistency

CYCLE

Data Reasonableness

Between-Tables Data Consistency

INSP_DATE

Data Reasonableness

Between-Tables Data Consistency

INSPECTOR Data Reasonableness

BEG_MP

Between-Field and Data Reasonableness

Between-Field Data Consistency

Data Completeness

END_MP

Between-Field and Data Reasonableness

Between-Field Data Consistency

Data Completeness

INT_LENGTH Between-Field and Data Reasonableness

RTE_LENGTH Acceptable Data Range

NUM_LANES

Acceptable Data Range

Between-Field Data Consistency

E-7

LANE_NO

Between-Field and Acceptable Data Range

Between-Field Data Consistency

LANE_WIDTH

Acceptable Data Range

Between-Field Data Consistency

Between-Tables Data Consistency

ROAD_WIDTH

Acceptable Data Range

Between-Field Data Consistency

Between-Tables Data Consistency

SHOULDER_WIDTH_R

Data Reasonableness

Between-Tables Data Consistency

SHOULDER_WIDTH_L

Data Reasonableness

Acceptable Data Range

Between-Tables Data Consistency

FUNCTIONAL_CLASS Acceptable Data Range

REMARKS

Acceptable Data Range

Between-Field and Data Range

URGENT_REMARKS Data Reasonableness

REMARKS_REPORT Data Reasonableness

TREATMENT Between-Field and Acceptable Data Range

TREATMENT_COST Between-Field and Acceptable Data Range

TREATMENT_DESCRIPTION Between-Field and Acceptable Data Range

UNPAVED_RATING

Data Reasonableness

Between-Field Data Consistency

UNP_CROWN_SCORE

Acceptable Data Range

Between-Field Data Consistency

UNP_CROWN_URG

Acceptable Data Range

Between-Field Data Consistency

UNP_CROWN_HI

Acceptable Data Range

Between-Field Data Consistency

UNP_CROWN_MED

Acceptable Data Range

Between-Field Data Consistency

UNP_CROWN_LOW

Acceptable Data Range

Between-Field Data Consistency

UNP_CROWN_REPORT

Acceptable Data Range

Between-Field Data Consistency

UNP_DRAIN_REPORT

Acceptable Data Range

Between-Field Data Consistency

UNP_RUT_WB_REPORT

Acceptable Data Range

Between-Field Data Consistency

UNP_POT_LA_REPORT

Acceptable Data Range

Between-Field Data Consistency

UNP_DUST_REPORT

Acceptable Data Range

Between-Field Data Consistency

PAVED_RATING

Acceptable Data Range

Between-Field Data Consistency

Between-Tables Data Consistency

DCV_SCR

Acceptable Data Range

Between-Field Data Consistency

DCV_RCI

Acceptable Data Range

Between-Field Data Consistency

AC_INDEX

Acceptable Data Range

Between-Field Data Consistency

LC_INDEX

E-8

MRL_LC_SEV

Acceptable Data Range

Within-Field Data Consistency

Between-Field and Data Range

TC_INDEX

Acceptable Data Range

Between-Field Data Consistency

PATCH_INDEX

Acceptable Data Range

Between-Field Data Consistency

RUT_INDEX

Acceptable Data Range

Between-Field Data Consistency

SC_INDEX

Acceptable Data Range

Between-Field Data Consistency

PATCHING

Acceptable Data Range

Between-Field Data Consistency

MRL_RATING_METHOD

Acceptable Data Range

Within-Field Data Consistency

Between-Field and Data Range

CO_JOINT_FAULT_INDEX

Acceptable Data Range

Between-Field Data Consistency

CO_JOINT_DISTRESS_INDEX

Acceptable Data Range

Between-Field Data Consistency

CO_SLAB_CRACK_INDEX

Acceptable Data Range

Between-Field Data Consistency

CO_DELAM_POP_INDEX

Acceptable Data Range

Between-Field Data Consistency

CO_PATCH_INDEX

Within-Field Data Consistency

Between-Tables Data Consistency

DCV_D_LANE_WIDTH Between-Tables Data Consistency

DCV_FORCED_PCR

Acceptable Data Range

Between-Field Data Consistency

DCV_IRI_AVG

Data Reasonableness

Between-Fields Data Consistency

DCV_IRI_SD

Acceptable Data Range

Between-Field Data Consistency

DCV_IRI_L

Acceptable Data Range

Between-Field Data Consistency

DCV_IRI_R

Acceptable Data Range

Between-Field Data Consistency

DCV_IRI_FLAG

Acceptable Data Range

Between-Field Data Consistency

DCV_RUT_AVG

Acceptable Data Range

Between-Field Data Consistency

DCV_RUT_MAX

Acceptable Data Range

Between-Field Data Consistency

DCV_RUT_SD

Acceptable Data Range

Between-Field Data Consistency

DCV_RUT_LOW

Acceptable Data Range

Between-Field Data Consistency

DCV_RUT_MED

Acceptable Data Range

Between-Field Data Consistency

DCV_RUT_HI

Acceptable Data Range

Between-Field Data Consistency

DCV_AC_LOW

Acceptable Data Range

Between-Field Data Consistency

DCV_AC_MED

E-9

DCV_AC_HI

Acceptable Data Range

Between-Field Data Consistency

DCV_LC_LOW

Acceptable Data Range

Between-Field Data Consistency

DCV_LC_MED

Acceptable Data Range

Between-Field Data Consistency

DCV_LC_HI

Acceptable Data Range

Between-Field Data Consistency

DCV_TC_LOW

Acceptable Data Range

Between-Field Data Consistency

DCV_TC_MED

Acceptable Data Range

Between-Field Data Consistency

DCV_TC_HI

Acceptable Data Range

Between-Field Data Consistency

DCV_VIDEO

Acceptable Data Range

Between-Field Data Consistency

DCV_IMAGE_NAME

Acceptable Data Range

Between-Field Data Consistency

DCV_SPEED

Acceptable Data Range

Between-Field Data Consistency

DCV_CONSTR_FLAG

Acceptable Data Range

Between-Field Data Consistency

DCV_DOR_FLAG

QC Checks for ROUTE_ID Table

2. No Null (blank/empty) values in any fields other than the REMARKS field.

Key Field Uniqueness Checks:

1. [ROUTE_IDENT] field should be unique throughout

Field Name QC Checks for ROUTE_ID Table

1 OBJECTID Key Field Uniqueness

2 GLOBAL_ID Key Field Uniqueness

3 AGENCY

Data Reasonableness

Between-Table Data Consistency

4 REGION

Data Reasonableness

Between-Table Data Consistency

5 ROUTE_IDENT

Key Field Uniqueness

Data Reasonableness

Between-Tables Data Consistency

6 CYCLE

Data Reasonableness

Between-Table Data Consistency

7 UNIT_ID Between-Fields Data Consistency

E-10

Between-Table Data Consistency

8 CYCLE_COLLECTED

Data Reasonableness

Between-Table Data Consistency

9 ROUTE_NUM

Data Reasonableness

Between-Fields Data Consistency

10 ROUTE_NAME Data Reasonableness

11 FROM_DESC Data Reasonableness

12 TO_DESC Data Reasonableness

13 INSP_DATE

Data Reasonableness

Between-Table Data Consistency

14 FUNCT_CLASS

Data Reasonableness

Between-Table Data Consistency

15 USER_ACCESS

Data Reasonableness

Between-Table Data Consistency

16 STATE1

Data Reasonableness

Between-Table Data Consistency

17 STATE2

Data Reasonableness

Between-Table Data Consistency

18 STATE3

Data Reasonableness

Between-Table Data Consistency

19 ASSET_NUM Data Reasonableness

20 SUB_UNIT

Data Reasonableness

Between-Table Data Consistency

21 COLL_METHOD

Data Reasonableness

Between-Table Data Consistency

22 CONCESSION Data Reasonableness

23 MILEAGE_GPS_MATCH_CYCLE

Acceptable Data Range

Between-Tables Data Consistency

25 PAVED_LENGTH

Data Reasonableness

Between-Tables Data Consistency

Between-Fields Data Consistency

26 UNPAVED_LENGTH

Data Reasonableness

Between-Tables Data Consistency

Between-Fields Data Consistency

27 RTE_LENGTH

Data Reasonableness

Between-Tables Data Consistency

Between-Fields Data Consistency

28 SURF_TYPE

Data Reasonableness

Between-Tables Data Consistency

Between-Fields Data Consistency

29 UNPAVED_STATUS

Data Reasonableness

Between-Tables Data Consistency

Between-Fields Data Consistency

30 SQFT

Data Reasonableness

Between-Tables Data Consistency

Between-Fields Data Consistency

31 LANE_NO

Data Reasonableness

Between-Tables Data Consistency

Between-Fields Data Consistency

32 NUM_LANES

Data Reasonableness

Between-Tables Data Consistency

Between-Fields Data Consistency

33 WIDTH_FLAG

Data Reasonableness

Between-Tables Data Consistency

Between-Fields Data Consistency

34 DISTRESS_FLAG

Data Reasonableness

Between-Tables Data Consistency

E-11

Between-Fields Data Consistency

35 ROAD_WIDTH

Acceptable Data Range

Between-Tables Data Consistency

Between-Fields Data Consistency

36 LANE_MILES

Acceptable Data Range

Between-Fields Data Consistency

37 AREA_MAP Acceptable Data Range

38 COMMENTS Data Reasonableness

39 ROUTE_PARENT

Data Reasonableness

Between-Fields Data Consistency

40 SUBCOMPONENT

Data Reasonableness

Between-Fields Data Consistency

41 BEG_LAT

Data Reasonableness

Between-Tables Data Consistency

42 BEG_LON

Data Reasonableness

Between-Tables Data Consistency

43 END_LAT

Data Reasonableness

Between-Tables Data Consistency

44 END_LON

Data Reasonableness

Between-Tables Data Consistency

45 WIP_DATA Acceptable Data Range

46 GIP_DATA Acceptable Data Range

47 ACTION_TIMESTAMP Acceptable Data Range

48 ALIGNMENT_ACTION Acceptable Data Range

49 TEMP_RIP_ID Acceptable Data Range

50 RTE_SERIES Acceptable Data Range

51 FURTHER_ACTION_REQ Acceptable Data Range

52 LANE_WIDTH Acceptable Data Range

53 BEG_MP_DCV

Acceptable Data Range

Between-Tables Data Consistency

Between-Fields Data Consistency

54 END_MP_DCV

Acceptable Data Range

Between-Tables Data Consistency

Between-Fields Data Consistency

55 PARTNER

Acceptable Data Range

Between-Tables Data Consistency

Between-Fields Data Consistency

56 FLTP

Acceptable Data Range

Between-Fields Data Consistency

57 NHS_MI Acceptable Data Range

58 OWNER Data Reasonableness

59 MAINTAINER Data Reasonableness

60 FACILITY_TYPE Acceptable Data Range

61 ASSET_TYPE Acceptable Data Range

62 COUNTY_CODE Data Reasonableness

63 TIER Data Reasonableness

64 REPORT_COMMENTS Data Reasonableness

65 ISVALID Data Reasonableness

66 LENGTH_UNIT Data Reasonableness

67 DISTRICT Data Reasonableness

E-12

QC Checks for ROUTE_GPS Table

2. No Null (blank/empty) values in any fields.

Key Field Uniqueness Check: No multiple records defined by the combination of values from the following fields: [ROUTE_IDENT] & [MP]

Other QC checks are specified in Table: QC Checks for ROUTE_GPS Table.

Field Name QC Checks for ROUTE_GPS Table

1 ROUTE_IDENT

Data Reasonableness

Between-Tables Data Consistency

RIP_CYCLE

RIP_CYCLE_COLLECTED

COLL_ITERATION

COLL_TYPE

PARK_ALPHA

Between-Field

3 STATE Data Reasonableness

10 MP Between-Field and Acceptable Data Range

11 GPS_LAT Acceptable Data Range

12 GPS_LON Acceptable Data Range

13 GPS_ELEV Acceptable Data Range

14 GPS_MODE Acceptable Data Range

15 XFALL Acceptable Data Range

16 GRADE Acceptable Data Range

17 HEADING Acceptable Data Range

INSP_DATE

Data Reasonableness

22 REMARKS Data Reasonableness

E-13

DATA INTEGRITY TEST:

PARK DATABASE VS. CRACK DETECTION OUTPUT

The crack detection output files are matched up to the ROAD_CONDITION table.

TEST

SAMPLE

SIZE

ACCURACY TOLERANCE

ROAD_CONDITION.FILENAME

matches the filename in the crack detection output file.

100% 100% Exactly equal one another.

Not case sensitive.

ROAD_CODITION.D_LANE_WIDTH

matches distress lane width in crack detection output file.

100% 100% +/- 0.0

ROAD_CONDITION.AC_HI,

ROAD_CONDITION.AC_MED,

ROAD_CONDITION.AC_LOW match alligator cracking values in crack detection output file.

100% 100% +/- 0.0

ROAD_CONDITION.LC_HI,

ROAD_CONDITION.LC_MED,

ROAD_CONDITION.LC_LOW match longitudinal cracking values in crack detection output file.

100% 100% +/- 0.0

ROAD_CONDITION.TC_HI,

ROAD_CONDITION.TC_MED,

ROAD_CONDITION.TC_LOW match transverse cracking values in crack detection output file.

100% 100% +/- 0.0

E-14

DATA ACCURACY TEST:

VIDEO QUALITY, PARK DATABASE to VIDEO (WIDTH

MEASUREMENTS & GPS)

WIDTH MEASUREMENTS

ROADWAY VIDEO

Data

Element

Location

Method Precision Accuracy Location

Location

Depicted on

Video

Lane

Width Video Analysis 1 decimal <= 1.0 foot

Milepost at width measurement

Measurement

Location Relative to

Corresponding

Video jpeg image

Pavement

Width Video Analysis 1 decimal <= 1.0 foot

Milepost at width measurement

Measurement

Location Relative to

Corresponding

Video jpeg image

Bike Lane

Width Video Analysis 1 decimal <= 1.0 foot

Milepost at width measurement

Measurement

Location Relative to

Corresponding

Video jpeg image

Refer to Appendix D: Video Measurements for descriptions of Pavement Width and Lane Width.

GPS Coordinate Plotting

TEST

SAMPLE

SIZE

ACCURACY TOLERANCE

Plot all Cycle 7 GPS points from

ROUTE_GPS table. Points should fall along route collected

100% +/- 25ft Pass/Fail

E-15

ROADWAY VIDEO

SAMPLE

SIZE

ACCURACY TOLERANCE

Route initializes in viewing software.

Route information is correct and consistent with the data tables.

100% of full-sized

100% Pass/Fail

All video viewed over the course of other tests is accessible and useable.

variable 100%. Pass/Fail

Routes look correct relative to one another and other routes

100% 100 Pass/Fail

Measured width values based on contractor-provided Width

Measurement Table match EFLHD-measured widths using the same methodology

10% of recorded features

95% <= 1 foot

Width values provided by the contractor in the Width Measurement Table match

EFLHD field-measured widths at the same location

100% of

EFLHD

field-measured width values

95% <= 1 foot

DISTRESS MEASUREMENTS by EFLHD (provided to Contractor)

DCV DATA

Data Element

Detection

Method

Reporting

Unit

Reporting

Precision Location Data Iterations

Rutting DCV Measured

X number of rut depths per 0.02 mile

(X per wheelpath);

tenths of inch 1 decimal DCV raw file

Rut depths converted to severity by contractor and used in rut index formula;

reported back to

EFLHD

Roughness (IRI) DCV Measured

IRI Left wheel path and IRI

Right wheel path per 0.02 mile; inches per mile 1 decimal DCV raw file

IRI used by the contractor in

Roughness

Condition Index formula (RCI);

Reported Back to

EFLHD

Please see Appendix E: Distress Identification Manual for the Road Inventory Program Cycles 7

E-16

DATA ACCURACY TEST:

PARK DATABASE VS. ROW & PAVEMENT VIDEO (PATCHING)

SAMPLE

SIZE ACCURACY TOLERANCE

Percent area of patching recorded in

ROAD_CONDITION.PATCHING

field matches percent area of patching visible in video section

10% 95% Pass/Fail

DISTRESS MEASUREMENTS by Contractor

PAVEMENT VIDEO

Data Element

Detection

Method

Reporting

Unit Accuracy Location

Detection

Elements

Transverse Cracks

Pavement Video

Analysis

Number of cracks per

0.02 mile (1 crack = full distress lane width

As a computed

95% Confidence

Level Video jpeg Image

Type, Severity, Extent

Longitudinal

Cracks

Pavement Video

Analysis

Percent of

Lane Length

(0.02 mile)

As a computed

95% Confidence

Level Video jpeg Image

Type, Severity, Extent

Alligator Cracks

Pavement Video

Analysis

Percent of

Lane Area per 0.02 mile

As a computed

95% Confidence

Level Video jpeg Image

Type, Severity, Extent

Patching/Potholes

Pavement Video

Analysis

Percent of

Lane Area per 0.02 mile

As a computed

95% Confidence

Level Video jpeg Image Type, Extent

Please see Appendix E: Distress Identification Manual for the Road Inventory Program Cycle 7

E-17

PROCEDURES FOR EFLHD QC TESTING OF CRACK

DETECTION SOFTWARE RESULTS

The statistical procedures described in this document are based on a statistical QC acceptance sampling by attributes utilizing Operating Characteristics (OC) Curves (1, 3).

Assumptions Used in Statistical QC Sampling Plan

• Confidence in the results of statistical testing is 95%

• The maximum allowable portion of bad frames per park is 5% (based on estimated 95% reliability of the Crack Detection Software data processing methodology/technology)

Quality Control Criterion

• QC Pass/fail criterion: pass if the difference between Crack Detection

Software reported evaluation results and EFLHD evaluation results is less than 20%; fail if the difference is over 20%.

***NOTE - The sampling process is under review and EFLHD is looking at whether all roads (no matter the length) are being examined adequately.

Theoretical Formulations Used to Determine Sample Sizes

The purpose of this section is to provide theoretical background that is used as a basis for selection of a random sample of n observations to be used in QC testing.

The hypergeometric distribution (3) is used as a basis for the determination of a QC acceptance sampling plan for the data sets consisting of observations that could be classified in two groups, based on established QC criteria:

1. “Good” observations (Crack Detection Software results differ from EFLHD results by less than +/- 20%)

2. “Bad” observations (Crack Detection Software results differ from EFLHD results by more than +/- 20%)

The hypergeometric distribution arises when a random selection (without repetition) is made among objects of two distinct types.

If a total data set of N observations has an actual fraction of defective items equal to p, then the probability g(p) of accepting the total data set N could be determined as:

r x n

N xn pN x

Np pg

)1(

(1)

E-18

If n is small relative to N, the above equation could be approximated using binomial distribution as follows (3):

xnx r x pp x n pg −

)1()(

(2)

Using these two formulations, a maximum acceptable number of “bad” observations r could be determined for any sample of size n.

Selection of Sample Size

For parks with the number of frames, n, the sample is significantly less than the total number of frames for the park (10% or less). A sample size with a maximum acceptable number of bad observations r could be determined based on binomial approximation as follows:

• The size of the random sample of the Crack Detection Software frames, determined by the minimum number of frames n0 that must be tested without a single frame failing the QC Pass/fail criterion, could be found from equation (2) assuming r = 0, as follows:

)ln(

)1ln(

R

C n

(3)

• The size of the random sample of the Crack Detection Software frames, determined by the minimum number of frames n that must be tested with only 1 frame failing the QC Pass/fail criterion, could be found from equation (2) assuming r = 1, as follows:

CRRnR nn −=−+ − 1))(1()( 1

(4)

• C is a confidence level expressed as a fraction. For 95% level of confidence C =

0.95

• R is a reliability of the Crack Detection Software evaluation results expressed as a fraction. The reliability value implies that the acceptable percentage of the frames not passing the QC standard criterion is equal to 100*(1-R).

From the observations presented in the bulleted list above, it could be concluded that for the selected 95% level of confidence the size of the frame sample n will depend on the acceptable level of reliability R of the reported Crack Detection Software results. The following table provides a relation between the selected reliability levels and the corresponding minimum size of frame samples required to pass the QC standard criterion in order for the Crack Detection Software results to be accepted for the park.

E-19

Data set reliability

R, %

Maximum percent of “bad” observations for the population

Sample size n0, frames

(pass with no failures)

Sample size n, frames

(pass with 1 failure)

Minimum data set size, frames

90 10 28 45 450

91 9 32 51 510

92 8 36 57 570

93 7 41 66 660

94 6 48 77 770

95 5 58 93 930

96 4 73 117 1,170

97 3 98 156 1,560

98 2 148 235 2,350

99 1 298 471 4,710

Since the maximum allowable portion of bad frames per park is set at 5%, the required level of reliability could be computed as 1-0.05=0.95 or 95%. Or a confidence reliability level of 95%.

• All Crack Detection Software frames in a random 58-frame sample should pass the QC standard pass/fail criterion for the Crack Detection Software results to be accepted for the park.

• Only 1 Crack Detection Software frame in a random 93-frame sample could fail the QC standard pass/fail criterion for the Crack Detection Software results to be accepted for the park.

QC Sampling Methodology

To achieve 95% confidence that Crack Detection Software data meet the required quality standard, the following QC sampling methodology will be applied:

1. Randomly select n frames. It is advisable to sample as many roads as practically possible. Generally, the number of frames from the same road should not exceed 10% of the total number of frames.

2. Test the first n0 frames in the selected sample against the specified QC pass/fail criterion

3. If none of the n0 frames fails the QC pass/fail criterion, the quality standard is met

– Crack Detection Software results are accepted for the park.

4. If 2 or more of the n0 frames fail the QC pass/fail criterion, the quality standard is not met – Crack Detection Software results are rejected for the park.

5. If 1 frame of the n0 frames fails the QC pass/fail criterion, test the remaining frames in the sample n.

6. If all the remaining frames in the sample of n frames pass the QC pass/fail criterion, the quality standard is met – Crack Detection Software results are accepted for the park.

7. If 1 or more of the remaining frames in the sample of n frames fail the QC pass/fail criterion, the quality standard is not met – Crack Detection Software results are rejected for the park.

E-20

REFERENCES

1. Selezneva, Mladenovic, Speir, Amenta, & Kennedy, Quality Assurance Sampling

Considerations for Automated Collection and Processing of Distress Data, Transportation Research Record No. 1889; Pavement Management, Monitoring, Evaluation, and Data Storage 2004

2. Mahoney, J.P, Statistical Methods for Pavements and Material Applications, Technical Series, Seattle, WA: University of Washington, 1997

3. Freund, R.J., W.J. Wilson, Statistical Methods, Academic Press, Inc., San Diego, CA, 1993

4. Ang, A. and W. Tang, Probability Concepts in Engineering Planning and Design, Volume 1, New York: John Wiley and Sons, 1975.

5. Hald, A., Statistical Theory with Engineering Applications, New York: John

Wiley and Sons, 1952