Appendix_F-Correlation_Test_Plan-Updated_12-Apr-17.pdf

CORRELATION TEST PLAN

for

PRODUCTION TESTING OF LUBE

ACCESSORY COMPONENTS

ON THE LUBE ACCESSORY TEST STAND

BUILDING 3001

06 APRIL 2017

1300626269C Typewritten Text Appendix F

Lube Accessory Components Certification Test Plan For Production Testing on new Lube Accessory Test Stand

2 | P a g e

Introduction

This document describes the certification test plan to qualify the new Lube Accessory test stand (hereinafter referred to as New Stand), to be located in Building 3001 Tank & Cooler shop, to perform and meet the production testing of requirements of all Engine Lube Accessory Components per applicable T.O.(s).

Correlation Process

The New Stand will be correlated to prove that performance meets or exceeds the existing production test stands and applicable T.O. requirements. The New Stand will be approved for production testing of the End Items listed in Table 1 if the Master Stand data and the New Stand data correlate as follows:

1. The final acceptance test will be run five times, per T.O., on the Master Stand to obtain correlation data. Data will be checked for outliers utilizing the Weisberg t-Test. See the section below titled Weisberg t-Test for Outliers for a brief description of testing for outliers. If an outlier is found, results from an additional run will be utilized to ensure that five samples are available for each test point. If an outlier is still indicated, it will be considered normal data scatter and all six runs will be used in the 95% confidence interval analysis.

2. The final acceptance tests will be run five times, on the New Stand to obtain correlation data.

Preset tests will be checked to ensure functionality only. No data will be submitted for Preset tests. Data will be checked for outliers utilizing the Weisberg t-Test. See the section below titled Weisberg t-Test for Outliers for a brief description of testing for outliers. If an outlier is found, results from an additional run will be performed to ensure that five samples are available for each test point. If an outlier is still indicated, it will be considered normal data scatter and all six runs will be used in the 95% confidence interval analysis.

3. Correlation between the two data sets will be analyzed utilizing the MOSI Method with confidence intervals of 95%. See section below titled Mosi Method for brief description of analysis method.

See Attachment 4 for detailed description of analysis method.

4. The results from the correlation methods will be documented in an Excel spreadsheet. See Attachment 1 for an example Certification Test Worksheet.

5. After the correlation data runs are completed a preliminary test report will be submitted to the System Program Office Cognizant Engineering Authority for review. Following approval, a certification test report will be submitted, via AFTO 202, for formal review and approval of the New Stand. See Attachment 2 for an example Certification Test Report.

5 Run Correlation Method

The methods to be used for correlation of test stand data are as follows:

• Weisberg t-Test for Outliers

• Mosi Method for comparing two means with a 95% Confidence interval

Weisberg t-Test for Outliers

All correlation data will be tested for Outliers at an Alpha Error Rate of 5%. All data will be checked for outliers at the time of data generation from the test stand and at the time of data correlation. See Attachment 3 for additional information.

Mosi Method

The Mosi Method (See Attachment 4 for additional information) for comparing two means with a 95% Confidence interval involves the following steps:

• Average the five samples from each stand

• Calculate the Standard Deviation of the five samples from each stand

3 | P a g e

• Calculate the 95% confidence intervals for each mean of stands

The data from each stand is statistically comparable if the intervals overlap such that each interval overlaps with the mean value of the other interval as shown in the diagram below. In our case there is a potential bias that must be taken into account to properly apply the MOSI technique. This bias is the measurement uncertainty of the test stand instrument.

For example:

Test stand A measures an average flow (5 readings) of 960 pph and Test stand B measures an average flow (5 readings) of 1040 pph. The measurement uncertainty for the flow measurement is ± 50 pph. For test stand A we can state the true average flow measurement is between 910 and 1010 pph and for test stand B we can state the true average flow measurement is between 990 and 1090 pph. Due to this bias we must include the measurement uncertainty as a range of acceptable mean values to properly apply the MOSI technique. This is graphically displayed on the following page.

4 | P a g e

Example of two stands that are statistically comparable

Example of two stands that are not statistically comparable

5 | P a g e

Production Testing on new Lube Accessory Test Stand Table 1: End Items Tested

Attachments:

Attachment 1 Example Certification Testing Worksheet Example.xls

Attachment 2 Example Certification Test Report.pdf

Attachment 3 Weisberg-t Test for Outliers.pdf t-Distribution Table.pdf Analysis of two data samples using the MOSI Method.pdf

Noun PN to be Certified T.O. Test Type NSN F101 Fuel/Oil Cooler UA538411-3 7J4-84-3 5 run correlation 2935-01-148-2116

F101 Hydraulic Pump RPV3-104-1I 7J4-2-57-3 5 run correlation 4320-01-147-9071

F101 AFT Tank Assembly 7089M89G07 7J10-19-3 2840-01-146-9375

F101 Lube Flow Sensor Assembly 635E876G07 7J14-4-3 5 run correlation 6620-01-332-9664

F101 Oil Level Temperature Sensor

7081M58P01 7J14-5-3 5 run correlation 6695-01-149-2860

F110 Oil Tank Assembly 1584M29G02 7J10-22-3 5 run correlation 2840-01-349-9023 F110/F118 Fuel/Oil Cooler 7J1-91-3 2935-01-238-8808

F108 Servo Fuel Heater

301-779-401-0 7J1-85-13 2935-01-148-5714

F118 Oil Tank Assembly 7J10-23-3 2840-01-297-1536

5 run correlation

301-779-502-0 6J24-10-13 5 run correlation 2915-01-148-5517

9338M23P04 5 run correlation

F108 Oil/Fuel Heat Exchanger

5 run correlation

5 run correlation

21SN04-222 8S2-5-8-3 5930-01-215-4689

F110 Fuel, Lube And Hydraulic Oil Cooler

70018-000 7J1-96-3 2935-01-319-5010

F110 Oil Differential Pressure Switch

5 run correlation

5 run correlation

F110 Hydraulic Pump 1156M46P16 7J4-2-60-3 4320-01-378-33985 run correlation

7101M43G05

F110 Hydraulic Cooler 1273M82P01 2J-F110-3-10

WP 019

2935-01-192-54385 run correlation

1374774529C

Note: If suggested Part Number to be certified is not available an equivalent Part Number for the NSN can be used.

1459M17G04/GAT1G058

Test Description Paragraph Parameter Name

Parameter Units

Test Spec

MIN

Limit

Test Spec

MAX

Limit

Test Spec Nominal

Value

Req'd Instrument Accuracy

Run #1 08/12/10

Run #2 08/12/10

Run #3 08/13/10

Run #4 08/13/10

Run #5 08/13/10

Run #6 08/13/10 Mean

Mean Less Acc.

Mean Plus Acc.

Std.

Dev.

95% Conf.

MIN

95% Conf.

MAX

95% Conf.

Correlation Master => Modified

Suspect Outlier?

Set-Up Errs or

OOL Reads Run #1

08/20/10 Run #2

07/26/10 Run #3

03/21/11 Run #4

08/09/10 Run #5

03/21/11 Run #6 Mean

Mean Less Acc.

Mean Plus Acc.

Std.

Dev.

95% Conf.

MIN

95% Conf.

MAX

95% Conf.

Correlation Modified => Master

Suspect Outlier?

Setup Errs or

OOL Reads Test Condition 7.10(b) 1.000 PR (062JA | MA) psig 950 1050 1000 ±10.00 1005 1003 1003 1001 1001 998 1001 964 998 967 Test Condition 7.10(b) 1.000 PB1 (062JC | NB) psig 90 110 100 ±0.5 96 101 98 96 97 100 102 100 103 99 Test Condition 7.10(b) 1.000 CDP (060MA|SA|PA) psig 365 375 370 ±0.8 368 370 369 371 368 370 370 370 370 370 Test Condition 7.10(b) 1.000 P5 (062PA | VA) psig 185 205 195 ±5.0 196 196 202 202 198 190 188 191 196 197 Test Condition 7.10(b) 1.000 P6 (062PB | VB) psig 195 245 220 ±5.0 197 244 233 244 215 216 220 219 226 244 Test Condition 7.10(b) 1.000 P7 (062PC) psig 165 215 190 ±5.0 195 207 170 190 182 194 201 196 184 176 Test Condition 7.10(b) 1.000 PC (062JB | NA) psig 275 295 285 ±0.5 287 287 295 287 282 288 290 288 282 280 Test Condition 7.10(b) 1.000 Torque Motor Current mA -5.0 0.0 -2.5 ±0.1 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 Test Result 7.10(b) 1.000 Feedback Volt/Volt Rdg v/v 0.4035 0.4065 0.4050 ±0.001 0.4060 0.4059 0.4060 0.4059 0.4060 0.4060 0.4050 0.4070 0.0001 0.4049 0.4071 Pass No OK 0.4064 0.4063 0.4042 0.4061 0.4041 0.4050 0.4040 0.4060 0.0012 0.4025 0.4075 Pass No OK

Modified Stand (3C3973G03-0C002/0403)Test Point Information

T.O. Number 6J3-2-33-3

C/N

Test Specification/Paragraph/Table Log Sheets: T.O. Section 7.10 (b)

Test Mechanic

Production Engineer

Test Dates

Master Stand (3C3973G03-0C001/0303)

Metering Valve Minimum Stop (Core Cut-In Overlap)

P/N and S/N

Part Name Augmenter Fuel Control

Correlation Worksheet

Equipment Specialist Kenneth Ham/736-3482

Cognizant Engineer Mike Babb/736-3517

1300626269C Typewritten Text Attachment 1

1374774529C

SAMPLE ONLY

CERTIFICATION TEST Report for

Part Types and Numbers on

Test Stand Type

PN: XXXX

LOCATION: BUILDING 3907

REPORT DATE: 1 APRIL XXXX

1300626269C

Attachment 2

(Part Type) Certification Test Report

2 | P a g e

Introduction

This document briefly describes the certification test reports testing results for qualifying the XXX Test Stand, located in Building XXXX.

Background

The new (Test Stand Type) Test Stand (part number) was delivered under contract # XXXXX.

System Overview

General description of equipment and basis of design.

System Calibration and Accuracies

Test Stand calibration will be performed and certified by Tinker PMEL Technicians IAW T.O. 00-20-14.

Calibration accuracies are identical to P/N XXXX.

Part Number For Correlation Testing

The Part Number listed in TABLE 1 was utilized for the correlation study.

TABLE 1 Part Number Used For Correlation Testing Part Number Noun T.O. Test Type

XXX XXX XXX XXX

XXX XXX XXX XXX

Correlation Process

1. A serviceable (part type(s)) listed in TABLE 1 was run five times on the Master Test Stand (test stand part #) to obtain baseline correlation data. The augmenter fuel controller was then run five times on the New Test Stand (test stand part #) and data recorded.

2. Data results were checked for outliers utilizing the Weisberg t-Test (See Attachment M – Statistical Correlation and Validation Method). If an outlier was flagged by the t-Test an additional run was made and replaced the outlier. If an outlier was still indicated after replacing the suspected outlier data point, it was considered normal data scatter and all six runs were used in developing the 95% confidence interval analysis.

3. The results from the correlation methods were documented in Excel spreadsheets. (See Attachments A thru L – (Part Type) Correlation Worksheets)

4. Correlation between the two data sets was accomplished utilizing the MOSI Method with a confidence interval of 95%. (See Attachment M – Statistical Correlation and Validation Method).

Signature Sheet

The undersigned certify XXXX Test Stand, Test Stand Part Number XXXXXXXX meets or exceeds all measurement and test requirements of Technical Order XXXXXXX.

3 | P a g e

Contractor Signature

Cognizant Engineer of Part

4 | P a g e

Attachments:

• Attachments A thru L - (Part Type) Correlation Worksheets

• Attachment M - Statistical Correlation and Validation Method.pdf

Weisberg t-Test for Outliers All correlation data will be tested at an Alpha Error Rate of 5% for Outliers. All data will be checked for Outliers at the time of data generation from the test stand and at the time of data correlation.

Mosi Method The Mosi Method (See Attachment 5 for additional information) for comparing two means with a 95% Confidence interval involves the following steps:

Average the five samples from each stand

Calculate the Standard Deviation of the five samples from each stand

Calculate the 95% confidence intervals for each mean of stands

The data from each stand is statistically comparable if the intervals overlap such that each interval overlaps with the mean value of the other interval as show in the diagram below. In our case there is a potential bias that must be taken into account to properly apply the MOSI technique. This bias is the measurement uncertainty of the test stand instrument. For example:

Test stand A measures an average flow (5 readings) of 960 pph and Test stand B measures an average flow (5 readings) of 1040 pph. The measurement uncertainty for the flow measurement is ± 50 pph. For test stand A we can state the true average flow measurement is between 910 and 1010 pph and for test stand B we can state the true average flow measurement is between 990 and 1090 pph. Due to this bias we must include the measurement uncertainty as a range of acceptable mean values to properly apply the MOSI technique. This is graphically displayed below.

1300626269C

Attachment 3

2 | P a g e

Example of two stands that are statistically comparable

Example of two stands that are not statistically comparable

Confidence intervals

• X at a given confidence level (say 95%) implies that the true value will be found within X of the calculated mean x t s

N p v, x t s

N x t s

N p v p v, ,

• s = standard deviation between individual values

• t = Student t value at a given probability (See Student T Distribution)

• x = Mean of data samples

• N = Number of Samples

Comparing two means

(unpaired data)

• Mosi method

– Calculate the confidence interval for each mean

– Compare the confidence intervals

– The results are statistically comparable if the intervals overlap such that each interval overlaps with the mean value of the other interval as shown in the diagram below.

Process Development

36 BioPharm International MAY 2003 statistical comparisons are relatively straightforward. Those data can be used to set acceptance criteria for validation runs at a new production scale.

Control charts. When about 15 lots have been produced at commercial scale, control charts — which present a picture of a process and its variation over time — are useful for evaluating process stability. Our choice of 15 lots for calculating control limits is a balance between the extreme uncertainty of limits based on few data and the diminishing value of each new data point in further decreasing that uncertainty. An individuals control chart based on 15 lots has 8.9 effective degrees of freedom (df), which is sufficient to reduce the uncertainty in the limits to about �23%. Achieving �10% uncertainty requires about 45 degrees of freedom, which requires more than 70 individual values (1).

Outlier tests and errors. Until there are 15 lots, the most useful method to statistically evaluate a data point that seems to be anomalous is the Weisberg t-test (2,3). The Weisberg t-test can be used for data sets larger than 15 values as well. The other tests we evaluated for application in small data sets were the Dixon (4) and the Grubbs (5).

The latter is also known as the Extreme Studentized Deviate (ESD) (6). The Weisberg t-test can distinguish between normal process variation and a process aberration that yields an outlier. For example, at an alpha (�) error (calling something an outlier when it isn’t one) of 0.05, the � error is only 0.31, and thus the Weisberg t-test is the most powerful test available for small data sets among those considered (Figure 1).

Discordant Observations Until recently, the U.S. Pharmacopeia (USP) did not address the treatment of chemical test data containing discordant observations. Indeed, this “silence” was interpreted to mean a “prohibition” during

Demonstrating the Consistency of Small Data Sets Application of the Weisberg t-test for Outliers

Determining whether a data point is an “outlier” — a result that doesn’t fit, that is too high or too low, that is extreme or discordant — is difficult when using small data sets (such as the data from three, four, or five conformance runs). The authors show that the Weisberg t-test is a powerful tool for detecting deviations in small data sets.

T he attempt to define an “outlier” has a long, diverse history. Despite many published definitions, statisticians in all fields are still interested in objectively determining whether a data point is consistent with the rest of the data set or is, after all, an outlier, signifying a deviation from the norm. For biopharmaceutical companies, the need to evaluate whether a data point is an outlier — inconsistent with the rest of a small set of data — is important in validating process consistency.

The power of a statistical tool increases as sample size (n) increases. So, low power outlier tests used on small data sets — such as the production data derived from three to five conformance runs — have relatively high beta (�) or Type 2 errors. These errors mean there is a high chance of leaving deviant results undetected, making these tests inappropriate for pharmaceutical or biopharmaceutical applications.

The z-test is the most powerful outlier test (the most able to detect a discordant datum) if the data are normally distributed and the standard deviation is known or can be accurately estimated. But z-tests usually require large data sets. Conformance runs from early commercial lots usually produce small data sets, and the standard deviation of the population is not usually known. We show, using representative biopharmaceutical process validation data, that the Weisberg t-test is a powerful outlier test at small values of n. It has a low � error rate in detecting deviations from the mean. Therefore, the Weisberg t-test is suitable for objectively demonstrating consistency of production data.

Process Validation Data During process validation, process consistency is typically demonstrated in three to five conformance runs. When historical data are available, even if those data are from a different production scale, Robert J. Seely, Louis Munyakazi, John Haury, Heather Simmerman, W. Heath Rushing, and Thomas F. Curry

A member of BioPharm International’s editorial advisory board, corresponding author Robert J. Seely is the associate director of corporate validation, corporate QA;

Louis Munyakazi is project biostatistician, six sigma; John Haury is associate director, six sigma;

Heather Simmerman is associate director, quality analytical laboratories; and W. Heath Rushing is a quality engineer III at Amgen Inc., 4000 Nelson Road, Longmont, CO 80503, 303.401.1586, fax 303.401.2820, rseely@amgen.com.

Thomas F. Curry is a technical fellow at Northrop Grumman Information Technology, 5450 Tech Center Drive, Colorado Springs, CO 80919.

38 BioPharm International MAY 2003 the United States v. Barr Laboratories, Inc.

case (7). Judge Wolin’s ruling in that case indicated the need for such guidance (8), and in 1999, a new monograph was previewed in Pharmacopeial Forum (9). That monograph states that when appropriately used, outlier tests are valuable tools for analyzing discordant observations.

The discussions in the Barr case and in the USP monograph suggest the appropriateness of using an outlier test to disregard a data point. In this article, we use such a test — the Weisberg t-test — to objectively identify an outlier as part of a statistical evaluation of small data sets. For process validation purposes, if the Weisberg t-test identifies no outliers, the data can be claimed to be consistent based on an objective statistical method.

This article describes the application of the Weisberg t-test to data from five conformance runs. We examine the ability of the test to demonstrate process consistency.

Subsequent uses of this test would include checking a suspect data point from lot six with the previous five, or lot seven from the previous six, for instance. The Weisberg t-test could also be used during a retrospective review of data sets. For example, an earlier value may stand out as a possible outlier, but only after subsequent data show a pattern that distinguishes it as a possible outlier. As standard practice, we advocate an investigation of the causes of such statistical differences.

As mentioned, at 15 data points, the individuals control chart for each point becomes the preferred tool for detecting discordant observations and for showing process consistency (defined as the absence of discordant observations). If the data are available in subgroups, then an averages control chart is preferred.

Testing for a Single Outlier In this article, we refer to an outlier as a datum that appears not to belong to the same group as the rest of the data. That datum measurement may seem either too large or too small in relation to the general pattern of the rest of the data. The method we propose applies to a single outlier, and is similar to the traditional t-calculated (tcalc) form of the general t-test statistic (Equation 1) (10,11).

[1]

The test hypothesis (Ho or null) can be stated as: The suspected value is not an outlier. Its alternative (Ha or alternative) is stated as: The suspected value is an outlier.

Working with reduced data. The entire set of data should not be used to estimate the standard error (SE). Such estimates would be biased if the suspected outlier were included. The estimate of variation would be inflated, and the estimate of the arithmetic mean would be biased toward the outlier.

The logic of the Weisberg t-test. After computing the estimates without the suspected outlier, the Weisberg t-test statistic for the suspected outlier (denoted by yi) is given in Equation 2, [2] where n is the sample size, y-i

— denotes the computed sample mean, s-i is its standard deviation after the withdrawal of yi, the suspect outlier.

The logic of the Weisberg t-test is that the numerator (yi�y-i

— ) compares the mean value y-i

— to the suspected outlier value yi.

Furthermore, the denominator s-i is the classic sample standard deviation.

In Equation 2, the factor denoted by adjusts the calculated t-value (tcalc) downward and is more conservative for small samples.

Specifically, the above factor is identical to

1 + h n – 1 n tcalc = n–1 n

× y i –y

–i s–i tcalc = Estimate of the difference

Standard error of the difference where h is the leverage matrix (3); that is because the data are reduced by one observation. The estimated SE, of the mean y-i

— is

[3] which makes Equation 2 equivalent to

[4]

The tcalc (as found above) is then compared to percentiles of a t-distribution at the � significant level with (n�2) degrees of freedom.

Probability and degrees of freedom. Table 1 shows the t-critical (tcrit) values at three different α values, in which the df are two less than the sample size. If the absolute value of tcalc is less than tcrit, the point is not an outlier. Table 1 can be generated (in Microsoft Excel or another spreadsheet program) for other values of � (alpha error rate) and degrees of freedom using the inverse of the Student’s t-distribution (TINV) function. The TINV function requires two arguments. They are: “the probability associated with a one-tailed t-distribution” and the “degrees of freedom.”

The critical t-values in the body of Table 1 are derived using the Excel function:

TINV(reference to value at the top of the column as a proportion, times two; and the value for the degrees of freedom from the first column where degrees of freedom is two less than the number of samples). In Excel, the TINV function gives the t-value for two tails, placing one-half of the � value in each tail, whereas the Weisberg t-test is a one-tail test. That is why the � values must be multiplied by two when using the TINV spreadsheet function. When using a published two-sided Student t-table, the results are obtained by shifting one column to the right, that is, by using (n�2) degrees of freedom, as shown in Table 1.

Identifying a Biotech Outlier As a real-world example, we use a data set from monitoring a large chromatography column in a recombinant protein purification process. Table 2 presents a representative set of such data. Step yield (percent recovery), target protein concentration, a purity assay, tcalc = y i

– y

–i

SE y –i

SE y

– i

= s–i 1 + h h = 1 n – 1

Figure 1. The operating characteristic curves for a variety of outlier tests are created by repeatedly drawing four values from a population of known mean and S, with a fifth value taken from a population with a known shift in mean.

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

0.5 1.5 2.5 3.5 4.5 5.5

Grubbs Dixon Weisberg z-test

Standard deviation shift e rr or n � 5, � � 0.05

Two alternative approaches to test for outliers include the regression approach (Alternative 1) and the ANOVA approach (Alternative 2). The results of the two alternatives are compared with the Weisberg t-test in Table 3. The entire data set (n�5) is used for these methods, but the results are identical to those obtained using Equation 2 or Equation 4.

Alternative 1: The Regression Approach An outlier test similar to Equation 2 exploits the full data set by testing the hypothesis that �1�0 using a simple linear model:

[5] in which y is the expected value given Xi.

The term Xi is coded 1, if y is the suspected outlier (y�yi) and 0 otherwise. In this model, �0 estimates the overall mean, and �1 represents the deviation from the mean for the rest of the data. The t statistic for testing �1�0 against a two-sided alternative is the appropriate statistic to use (12). Under the assumption of normal error, the t is a Student t with n�k�1 degrees of freedom, in which k�1 (due to �1).

Therefore, [6] in which b1 and SE(b1) are sample estimates of �1 and its standard error (SE).

Alternative 2: The One-Way ANOVA Model A similar coding of the full data leads to the same test through the use of one-way analysis of variance (ANOVA). The model is:

[7] in which y is the expected value, µ is the overall mean, � represents two classes or categories defined by 0 and 1 depending on whether the observation is a suspected outlier (�1) or not (�0). The degrees of freedom are (n��1), in which n� are the two levels of �, therefore (n��1)�(2�1).

Consequently, the degrees of freedom for the error is (n�1) � (n��1). The ANOVA table is provided below.

The df column in the ANOVA table defines the degrees of freedom; the Fcalc is equivalent to tcalc and have identical probability of discerning an outlier (the p-value); that is, Fcalc equals the t2calc obtained in the Weisberg t-test and in the regression approach. Moreover, estimates of y-i

– and y are obtained by applying estimable functions, that is

[8] where m, a0, and a1 can be obtained from the solution vector of the model in Equation 7. The elements of the solution vector — m, a0, and a1 — represent nonunique estimates of the intercept (m), the effect of observations without the suspected value (a0), and the effect of the suspected value (a1). A test identical to the Weisberg t-test and the regression-based test is obtained by computing the difference between the two estimable functions in Equation 8. The resulting difference is also estimable (12). Thus, [9] provides an equivalent test to the Weisberg t-test and the regression-based test (3,5).

The standard error SE(a0�a1) is

[10]

The Weisberg t-test, the regression-based test, and the ANOVA model are similar because in all three methods, the same quantity (in absolute terms) is represented by the estimated slope �1 of Equation 5, the numerator (a0�a1) in Equation 9, and the numerator of Equation 4. All three methods also have the same SE. The results of the outlier tests of the three methods are compared in Table 3 using the data from Table 2.

Because of their simplicity, the above calculations can be performed in a spreadsheet package that has even limited statistical capability. The SAS code needed to run these methods is listed in the box to the left (13). They can also be obtained from Louis Munyakazi, louism@amgen.com.

yi – y–i

SE a0 – a1 = σ 1 n + 1 tcalc = a0 – a1

SE a0 – a1 y –i

= m +a0 and y i

= m + a1 y = µ + α with var(y) = σ2 tcalc = b1

SE b1 y = β0 + β1X i with var y = σ2

Degrees of Sum of Mean Source Freedom Squares Square Fcalc

Model 1 SSmodel MSmodel a

Error n�2 SSerror MSerror

Corrected total n�1 SStotal aMSE is mean square error.

MSmodel

MSE

ANOVA Table. A one-way analysis of variance model (Alternate 2) delivers the same results as the Weisberg t or regression-based tests: the Fcalc � t2calc ; also the p-values are identical (Table 3).

The Data data o;

input y @@;

x=0;

if _n_=3 then x=1;

datalines;

17.5 17.4 30.2 22.2 27.0 run;

proc print;

run;

Alternative 1: Linear Regression proc reg alpha=.1;

A: model y=x;

*test H0: b=0;

B: model y=t/influence;

*look for R-Studentized Residual;

output out=hat h=hmatrix;

title3 Method Uses ALL the Data;

title4 Simple Linear Regression Model;

run;

Alternative 2: One-Way ANOVA proc glm data=o alpha=.1;

class x;

model y = x/ss3 solution;

output out=g h=hamtrix;

estimate ‘Estimate of mu’ intercept 1 x 1;

estimate ‘Estimate of outlier’ intercept 1 x 0 1;

estimate ‘Weisberg test’ x -1 1;

title3 Method Uses ALL the Data;

title4 Through Linear Model and Contrast;

run;

SAS Codes for the Alternative Methods

40 BioPharm International MAY 2003

Alternative Methods for Determining tcalc

42 BioPharm International MAY 2003 host cell protein (HCP) concentration, and processing time are the primary indicators of step consistency. The data appear to be consistent across the five lots, except in Lot 3, the HCP is apparently high and might be inconsistent with the other four data points.

The Weisberg tcalc for HCP in our example is 1.796. Comparing that number with the tcrit values (Table 1), for n�5, ��0.05, the tcalc is less than the tcrit (2.353), therefore the data point is not an outlier, and the five data points are consistent. So the subjective judgment used to decide that the data point might be discordant is followed by the application of a statistical tool to give an objective assessment.

Choosing an � value of 0.05 means that when the process actually has no outliers, we are willing to accept a 5% chance of a false positive — a 5% chance that a point identified as discordant by the Weisberg t-test is not, actually, an outlier.

Accepting that rate means accepting unnecessary investigations 5% of the time.

If the � value is reduced to avoid those investigations, the � value rises, which means false negatives — the test fails to identify a discordant value. In our application, a � error occurs when the test fails to detect an outlier when one is actually present. We choose to set � at 0.05 and are willing to perform more frequent investigations (as a result of false positives) to keep the � error rate low. At ��0.05, n�5, the � error is a reasonable 0.31 for detecting a shift of three standard deviations (Figure 1). That rate is significantly less than the more commonly used outlier tests (Dixon and Grubbs), which yield � errors of approximately 0.77 as can be seen in Figure 1 (4,5).

The set of operating characteristic (OC) curves in Figure 1 shows a variety of outlier tests constructed by simulating a data set of 5,000 from which four samples were drawn.

A fifth datum was randomly taken from a data set, which was shifted by a given number of standard deviations. To detect a standard deviation change of three, the z-test (the basis for control charts) is clearly the most sensitive, with a � error of 0.08. For the purposes of comparing outlier tests, the z-test is presented here as a one-sided test;

the two-sided z-test is the basis for control charts. The control chart, however, requires a large data set or a good estimate of the variance of the data. When those conditions are available, an individuals control chart (or a control chart of averages) is the recommended method. When a large data set or a good estimate of the variance is not available — for five conformance runs with little relevant data from previous scales, for example — the Weisberg t-test is clearly the next best available tool.

The Bonferroni correction is used for some statistical comparisons. For example, it is used for multiple comparisons (the “family” of comparisons) by dividing the Type 1 error among all comparisons, so that the overall Type 1 error rate of the family does not exceed a desired level. In our example, we use a single hypothesis test for one visually suspected outlier, rather than testing a hypothesis of no outliers by performing multiple tests on every data point versus the remaining set. Because multiple comparisons are not contemplated in our example, we don’t use the Bonferroni correction.

One-sided or two. A final point needs to be made about the one-sided versus the two-sided Weisberg t-tests for outliers. Because an outlier is initially detected as being the farthest from the central tendency (the mean) of the data, the outlier will be either

Degrees of Alpha Error Rate Freedoma 0.01 0.05 0.10

3 4.541 2.353 1.638 4 3.747 2.132 1.533 5 3.365 2.015 1.476 6 3.143 1.943 1.440 7 2.998 1.895 1.415 8 2.896 1.860 1.397 9 2.821 1.833 1.383

10 2.764 1.812 1.372 11 2.718 1.796 1.363 12 2.681 1.782 1.356 13 2.650 1.771 1.350 14 2.624 1.761 1.345 15 2.602 1.753 1.341 16 2.583 1.746 1.337 17 2.567 1.740 1.333 18 2.552 1.734 1.330 aDegrees of freedom are n–2 of sample size.

Table 1. The tcrit values for three different levels of � errors and the degrees of freedom (two less than the sample size)

Parameter Lot 1 Lot 2 Lot 3 Lot 4 Lot 5

Step time (h) 50 49 48 51 51 Step yield (%) 82 83 89 88 89 Concentration (g/L) 13.8 13.6 14.1 13.8 14.0 Purity (%) 97.1 97.2 97.6 97.3 97.5 HCP (ppm) 17.5 17.4 30.2 22.2 27.0

Table 2. A representative set of data from the first five lots of a purification process for a recombinant protein in a large chromatography column; the parameters are the primary indicators of step consistency.

Number of Estimate Method Observations � Standard Error Calculated t p-value

Weisberg t-test (n�1) � 4 21.03 � 4.57 1.796 0.1704 (reduced data)

Alternate 1 n � 5 9.18 � 5.11a 1.796 0.1704 Regression test (full data)

Alternate 2 n � 5 21.03 � 2.28b 1.796 0.1704 ANOVA test 30.20 � 4.57b

(full data) 9.18 � 5.11b aRepresents estimates of �1 (deviation from the mean of the n�1 data) bRepresents estimates of m�a0, m�a1, and a0�a1 (mean of reduced data, suspected outlier, and their difference, � corresponding SEs)

Table 3. A comparison of the Weisberg t-test with two other methods for obtaining identical tcalc values: the regression-based method (Alternate 1) and the ANOVA-based method (Alternate 2); the host cell protein (HCP) observations for testing step consistency are the responses being tested.

Continued on page 58

58 BioPharm International MAY 2003

(2) Brandt, R., “Comparing Classical and Resistant Outlier Rules,” J. Am. Stat. Assoc.

85, 1083–1090 (1990). Note: The error in the formula printed in this reference was corrected in Curry, T.F., “Corrections,” J. Am. Stat.

Assoc. 96(456), 1534 (2001).

(3) Weisberg, S., Probability and Mathematical Statistics: Applied Linear Regression 2nd ed.

(John Wiley & Sons, New York, 1985).

(4) Dixon, W.J., “Processing Data for Outliers,” Biometrics 9, 74–89 (1953).

(5) Grubbs, F.E., “Procedures for Detecting Outlying Observations in Samples,” Technometrics 11, 1–21 (1968).

(6) Bancroft, T.A., “Analysis and Inference for Incompletely Specified Models Involving the Use of Preliminary Test(s) of Significance,” Biometrics, 20, 427–442 (September 1964).

(7) United States v. Barr Laboratories, Inc., 812 F. Supp. 458 (DNJ 1993).

(8) Kuwahara, S.S., “Outlier Testing: Its History and Applications,” BioPharm 10(4) 64–67 (April 1997).

(9) “General Information: <1010> Analytical Data

— Interpretation and Treatment,” Pharmacopeial Forum 25(5), 8900–8909 (September–October 1999).

(10) Draper, N.R. and H. Smith, Applied Regression Analysis (John Wiley & Sons, New York, 1998), p. 4.

(11) Cook, R.D and Weisberg, S., Applied Regression Including Computing and Graphics (John Wiley & Sons, New York, 1999).

(12) Searle, S.R., Linear Models (John Wiley & Sons, New York, 1971).

(13) SAS Institute Inc., SAS/STAT User’s Guide, Version 8.01 (Cary, NC, 1999).

example, the test fits the needs for evaluating biotechnology process data.

The Weisberg t-test can be applied for determining the internal consistency of small data sets and can also be useful in process validation. When validating a process, a protocol with preapproved acceptance criteria is required. For key performance parameters, numerical limits for specific attributes must be defined and met.

Typically, however, many secondary parameters may not have predefined numerical limits, but they are still expected to be internally consistent during the validation runs. For example, during scale-up, the mean of a given output parameter can shift up or down, but if that does not affect product quality, the variation may be perfectly acceptable. To validate that a process is performing consistently, the values of that parameter should be similar for three to five runs. The Weisberg t-test is a useful tool that adds statistical objectivity to the claim that a process is “consistent.” BPI

References

(1) Wheeler, D.J., Advanced Topics in Statistical

Process Control: The Power of Shewhart’s Charts (SPC Press, Knoxville, TN, 1995),

p. 185.

higher or lower than the mean. The Weisberg t-test determines whether the “outlier” is larger if it is to the right of the mean (on a number line) or smaller if it is to the left of the mean (on a number line). The test does not show the differences without reference to the direction of that difference;

therefore, the Weisberg t-test is a one-sided test, and the resulting tcrit values in the table must reflect that.

Alternative methods. Two other methods can be used to obtain identical tcalc values. One uses regression (Alternative 1 in Table 3), and one uses analysis of variance (ANOVA) (Alternative 2 in Table 3). These methods are discussed in the “Alternative Methods for Determining tcalc” sidebar, and the results from those tests are compared with the Weisberg tcalc values in Table 3.

A Superior Tool The Weisberg t-test has a low � error rate (especially when used with a higher � error rate) for small data sets. It is a superior, objective tool for showing consistency within small data sets. As shown in our

ABBOTT BIORESEARCH CENTER 19 22 508.849.2785 508.755.8506 www.abbott.com/abbottbioresearch

ADVANCED RESEARCH TECHNOLOGIES (ART) 7 3 514.832.0777 514.832.0778 www.art.ca

AGILENT TECHNOLOGIES INC. 17 9 800.227.9770 800.519.6047 www.agilent.com

AMERSHAM BIOSCIENCES 39 39 732.457.8000 732.457.0557 www.amershambiosciences.com

ASME INTERNATIONAL 54 43 800.843.2763 212.591.7143 www.asme.org/education

AVECIA 18 17 +44.161.721.1831 +44.161.721.5319 www.avecia.com/biotech

BIACORE 43 25 +49.18.675.700 +49.18.150.110 www.biacore.com

BIOENGINEERING AG 6 5 +41.55.256.8111 +41.55.25.68256 www.bioengineering.ch

BIO-RAD LABORATORIES 68 20 800.424.6723 510.741.5800 www.bio-rad.com/process

BIORELIANCE 37 31 301.738.1000 301.738.4036 www.bioreliance.com

BROADLEY JAMES, INC. 57 7 949.829.5555 949.829.5560 www.broadleyjames.com

CHARLES RIVER LABORATORIES 23 32 978.658.6000 978.658.7132 www.criver.com

CUNO INC. 45 23 203.238.8930 203.238.8977 www.cuno.com

DIOSYNTH RTP, INC. 47 30 919.678.4400 919.678.4499 www.diosynth.com

GIBCO INVITROGEN CORPORATION 11 13 800.955.6288 760.603.7229 www.invitrogen.com/gibco

HAMILTON SUNDSTROM 48 45 909.593.3581 909.392.3207 www.hamiltonsundstrand.com

HYCLONE 13 8 800.492.5663 800.533.9450 www.hyclone.com

IBM LIFE SCIENCES 15 4 914.499.1900 www.ibm.com/solutions/lifesciences

IRVINE SCIENTIFIC 25 16 800.437.5706 949.261.6522 www.irvinesci.com

LAUREATE PHARMA LP 2 1 609.919.3400 609.520.3963 www.laureatepharma.com

LONZA BIOLOGICS, INC. 27 15 603.334.6100 603.334.3300 www.lonzabiologics.com

MALLINCKRODT BAKER 21 12 908.859.2151 908.859.9385 www.mallbaker.com

MILLIPORE CORPORATION 5 2 781.533.2117 781.533.3117 www.millipore.com/bioprocess

NEKTAR 49 28 650.631.3100 650.631.3150 www.nektar.com

NIRO SOAVI 51 24 715.386.9371 715.386.9376 www.niroinc.com

NOVA BIOMEDICAL 35 27 800.458.5813 781.894.5915 www.novabiomedical.com

PALL CORPORATION 33 21 800.717.7255 516.625.3610 www.pall.com/biopharmaceuticals

SEROLOGICALS CORPORATION 41 33 678.728.2000 678.728.2020 www.serologicals.com

SPARTA SYSTEMS INC. 9 6 732.203.0400 732.203.0375 www.sparta-systems.com

SWAGELOK COMPANY 31 29 866.737.6246 866.737.6247 www.swagelok.com

TEFEN OPERATIONS MANAGEMENT CONSULTING 53 14 866.8TEFEN8 ext. 135 212.317.0604 www.tefen.com/biobenchmark

UCB BIOPRODUCTS, INC. 67 19 770.437.5500 770.437.5640 www.ucb-bioproducts.com

YSI LIFE SCIENCES 54 26 937.767.7241 www.ysi.com/lifesciences

INDEXADINDEX

Company Page Info # Phone Fax Web Site

Weisberg t-test continued from page 42

1.3.6.7.2. Upper Critical Values of the Student's-

t Distribution

How to

Use This

Table

This table contains the upper critical values of the Student's t-distribution. The upper critical values are computed using the percent point function. Due to the symmetry of the t-distribution, this table can be used for both 1-sided (lower and upper) and 2-sided tests using the appropriate value of .

The significance level, , is demonstrated with the graph below which plots a t distribution with 10 degrees of freedom.

The most commonly used significance level is = 0.05. For a two-sided test, we compute the percent point function at /2

(0.025). If the absolute value of the test statistic is greater than the upper critical value (0.025), then we reject the null hypothesis. Due to the symmetry of the t-distribution, we only tabulate the upper critical values in the table below.

Given a specified value for :

1. For a two-sided test, find the column corresponding to

/2 and reject the null hypothesis if the absolute value of the test statistic is greater than the value of in the table below.

2. For an upper one-sided test, find the column corresponding to and reject the null hypothesis if the test statistic is greater than the tabled value.

http://www.itl.nist.gov/div898/handbook/eda/section3/eda3664.htm http://www.itl.nist.gov/div898/handbook/eda/section3/eda3664.htm http://www.itl.nist.gov/div898/handbook/eda/section3/eda362.htm#PPF http://www.itl.nist.gov/div898/handbook/eda/section3/eda362.htm#PPF

3. For an lower one-sided test, find the column corresponding to and reject the null hypothesis if the test statistic is less than the negative of the tabled value.

Upper critical values of Student's t distribution with degrees of freedom Probability of exceeding the critical value

0.10 0.05 0.025 0.01 0.005 0.001

1. 3.078 6.314 12.706 31.821 63.657 318.313

2. 1.886 2.920 4.303 6.965 9.925 22.327

3. 1.638 2.353 3.182 4.541 5.841 10.215

4. 1.533 2.132 2.776 3.747 4.604 7.173

5. 1.476 2.015 2.571 3.365 4.032 5.893

6. 1.440 1.943 2.447 3.143 3.707 5.208

7. 1.415 1.895 2.365 2.998 3.499 4.782

8. 1.397 1.860 2.306 2.896 3.355 4.499

9. 1.383 1.833 2.262 2.821 3.250 4.296

10. 1.372 1.812 2.228 2.764 3.169 4.143

11. 1.363 1.796 2.201 2.718 3.106 4.024

12. 1.356 1.782 2.179 2.681 3.055 3.929

13. 1.350 1.771 2.160 2.650 3.012 3.852

14. 1.345 1.761 2.145 2.624 2.977 3.787

15. 1.341 1.753 2.131 2.602 2.947 3.733

16. 1.337 1.746 2.120 2.583 2.921 3.686

17. 1.333 1.740 2.110 2.567 2.898 3.646

18. 1.330 1.734 2.101 2.552 2.878 3.610

19. 1.328 1.729 2.093 2.539 2.861 3.579

20. 1.325 1.725 2.086 2.528 2.845 3.552

21. 1.323 1.721 2.080 2.518 2.831 3.527

22. 1.321 1.717 2.074 2.508 2.819 3.505

23. 1.319 1.714 2.069 2.500 2.807 3.485

24. 1.318 1.711 2.064 2.492 2.797 3.467

25. 1.316 1.708 2.060 2.485 2.787 3.450

26. 1.315 1.706 2.056 2.479 2.779 3.435

27. 1.314 1.703 2.052 2.473 2.771 3.421

28. 1.313 1.701 2.048 2.467 2.763 3.408

29. 1.311 1.699 2.045 2.462 2.756 3.396

30. 1.310 1.697 2.042 2.457 2.750 3.385

31. 1.309 1.696 2.040 2.453 2.744 3.375

32. 1.309 1.694 2.037 2.449 2.738 3.365

33. 1.308 1.692 2.035 2.445 2.733 3.356

34. 1.307 1.691 2.032 2.441 2.728 3.348

35. 1.306 1.690 2.030 2.438 2.724 3.340

36. 1.306 1.688 2.028 2.434 2.719 3.333

37. 1.305 1.687 2.026 2.431 2.715 3.326

38. 1.304 1.686 2.024 2.429 2.712 3.319

39. 1.304 1.685 2.023 2.426 2.708 3.313

40. 1.303 1.684 2.021 2.423 2.704 3.307

41. 1.303 1.683 2.020 2.421 2.701 3.301

42. 1.302 1.682 2.018 2.418 2.698 3.296

43. 1.302 1.681 2.017 2.416 2.695 3.291

44. 1.301 1.680 2.015 2.414 2.692 3.286

45. 1.301 1.679 2.014 2.412 2.690 3.281

46. 1.300 1.679 2.013 2.410 2.687 3.277

47. 1.300 1.678 2.012 2.408 2.685 3.273

48. 1.299 1.677 2.011 2.407 2.682 3.269

49. 1.299 1.677 2.010 2.405 2.680 3.265

50. 1.299 1.676 2.009 2.403 2.678 3.261

51. 1.298 1.675 2.008 2.402 2.676 3.258

52. 1.298 1.675 2.007 2.400 2.674 3.255

53. 1.298 1.674 2.006 2.399 2.672 3.251

54. 1.297 1.674 2.005 2.397 2.670 3.248

55. 1.297 1.673 2.004 2.396 2.668 3.245

56. 1.297 1.673 2.003 2.395 2.667 3.242

57. 1.297 1.672 2.002 2.394 2.665 3.239

58. 1.296 1.672 2.002 2.392 2.663 3.237

59. 1.296 1.671 2.001 2.391 2.662 3.234

60. 1.296 1.671 2.000 2.390 2.660 3.232

61. 1.296 1.670 2.000 2.389 2.659 3.229

62. 1.295 1.670 1.999 2.388 2.657 3.227

63. 1.295 1.669 1.998 2.387 2.656 3.225

64. 1.295 1.669 1.998 2.386 2.655 3.223

65. 1.295 1.669 1.997 2.385 2.654 3.220

66. 1.295 1.668 1.997 2.384 2.652 3.218

67. 1.294 1.668 1.996 2.383 2.651 3.216

68. 1.294 1.668 1.995 2.382 2.650 3.214

69. 1.294 1.667 1.995 2.382 2.649 3.213

70. 1.294 1.667 1.994 2.381 2.648 3.211

71. 1.294 1.667 1.994 2.380 2.647 3.209

72. 1.293 1.666 1.993 2.379 2.646 3.207

73. 1.293 1.666 1.993 2.379 2.645 3.206

74. 1.293 1.666 1.993 2.378 2.644 3.204

75. 1.293 1.665 1.992 2.377 2.643 3.202

76. 1.293 1.665 1.992 2.376 2.642 3.201

77. 1.293 1.665 1.991 2.376 2.641 3.199

78. 1.292 1.665 1.991 2.375 2.640 3.198

79. 1.292 1.664 1.990 2.374 2.640 3.197

80. 1.292 1.664 1.990 2.374 2.639 3.195

81. 1.292 1.664 1.990 2.373 2.638 3.194

82. 1.292 1.664 1.989 2.373 2.637 3.193

83. 1.292 1.663 1.989 2.372 2.636 3.191

84. 1.292 1.663 1.989 2.372 2.636 3.190

85. 1.292 1.663 1.988 2.371 2.635 3.189

86. 1.291 1.663 1.988 2.370 2.634 3.188

87. 1.291 1.663 1.988 2.370 2.634 3.187

88. 1.291 1.662 1.987 2.369 2.633 3.185

89. 1.291 1.662 1.987 2.369 2.632 3.184

90. 1.291 1.662 1.987 2.368 2.632 3.183

91. 1.291 1.662 1.986 2.368 2.631 3.182

92. 1.291 1.662 1.986 2.368 2.630 3.181

93. 1.291 1.661 1.986 2.367 2.630 3.180

94. 1.291 1.661 1.986 2.367 2.629 3.179

95. 1.291 1.661 1.985 2.366 2.629 3.178

96. 1.290 1.661 1.985 2.366 2.628 3.177

97. 1.290 1.661 1.985 2.365 2.627 3.176

98. 1.290 1.661 1.984 2.365 2.627 3.175

99. 1.290 1.660 1.984 2.365 2.626 3.175

100. 1.290 1.660 1.984 2.364 2.626 3.174

1.282 1.645 1.960 2.326 2.576 3.090

Weisberg T-test-Attachment 3