CAP_Laboratory_Standards_for_Next-Generation_Sequencing_Clinical_Tests_(2015).pdf

CAP Laboratory Improvement Programs

College of American Pathologists’ Laboratory Standards for Next-Generation Sequencing Clinical Tests

Nazneen Aziz, PhD; Qin Zhao, PhD; Lynn Bry, MD, PhD; Denise K. Driscoll, MS, MT(ASCP)SBB; Birgit Funke, PhD;

Jane S. Gibson, PhD; Wayne W. Grody, MD; Madhuri R. Hegde, PhD; Gerald A. Hoeltge, MD; Debra G. B. Leonard, MD, PhD;

Jason D. Merker, MD, PhD; Rakesh Nagarajan, MD, PhD; Linda A. Palicki, MT(ASCP); Ryan S. Robetorye, MD; Iris Schrijver, MD;

Karen E. Weck, MD; Karl V. Voelkerding, MD

� Context.—The higher throughput and lower per-base cost of next-generation sequencing (NGS) as compared to Sanger sequencing has led to its rapid adoption in clinical testing. The number of laboratories offering NGS-based tests has also grown considerably in the past few years, despite the fact that specific Clinical Laboratory Improve-ment Amendments of 1988/College of American Patholo-gists (CAP) laboratory standards had not yet been developed to regulate this technology.

Objective.—To develop a checklist for clinical testing using NGS technology that sets standards for the analytic wet bench process and for bioinformatics or ‘‘dry bench’’ analyses. As NGS-based clinical tests are new to diagnostic testing and are of much greater complexity than traditional Sanger sequencing–based tests, there is an urgent need to develop new regulatory standards for laboratories offering these tests.

Design.—To develop the necessary regulatory frame-work for NGS and to facilitate appropriate adoption of this technology for clinical testing, CAP formed a committee in 2011, the NGS Work Group, to deliberate upon the contents to be included in the checklist.

Results.—A total of 18 laboratory accreditation checklist requirements for the analytic wet bench process and bioinformatics analysis processes have been included within CAP’s molecular pathology checklist (MOL).

Conclusions.—This report describes the important issues considered by the CAP committee during the development of the new checklist requirements, which address documen-tation, validation, quality assurance, confirmatory testing, exception logs, monitoring of upgrades, variant interpreta-tion and reporting, incidental findings, data storage, version traceability, and data transfer confidentiality.

(Arch Pathol Lab Med. 2015;139:481–493; doi: 10.5858/ arpa.2014-0250-CP)

DNA sequencing has evolved from Maxam-Gilbert1 and Sanger2,3 methods in the 1970s to a set of technologies that are collectively referred to as next-generation sequencing (NGS).4–12 The primary difference between NGS and first-generation technologies is that sequencing of millions of short fragments of DNA occurs in parallel instead of one DNA fragment at a time. Sequencing of DNA as a clinical test became routinely possible only after the automation of Sanger sequencing methods introduced in the mid-1990s, which used capillary gel electrophoresis with fluorescence-based detection.13,14 The throughput of NGS far surpasses that of automated Sanger sequencing. The higher through-put and lower per-base cost of NGS have contributed to its rapid adoption in clinical testing,15 despite the fact that several aspects of NGS analysis have much higher complexity. Examples include the acquisition and storage of data sets that far exceed those commonly generated in a Clinical Laboratory Improvement Amendments of 1988 (CLIA) laboratory and downstream challenges in computa-tion and interpretation. Areas in which NGS testing is being

Accepted for publication June 19, 2014.

Published as an Early Online Release August 25, 2014.

From Molecular Medicine (Dr Aziz), Laboratory Improvement

Programs (Dr Zhao and Ms Palicki), and Laboratory Accreditation and Regulatory Affairs (Ms Driscoll), College of American Patholo-gists, Northfield, Illinois; the Department of Pathology, Brigham & Women’s Hospital, Harvard Medical School, Boston, Massachusetts (Dr Bry); the Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts (Dr Funke); the Department of Clinical Sciences, University of Central Florida College of Medicine, Orlando (Dr Gibson); the Divisions of Medical Genetics and Molecular Diagnostics, Department of Pathology & Laboratory Medicine, Pediatrics, and Human Genetics, UCLA School of Medicine, UCLA Institute for Society and Genetics, Molecular Diagnostic Laboratories and Clinical Genomics Center, UCLA Medical Center, Los Angeles, California (Dr Grody); the Department of Human Genetics, Emory University School of Medicine, Decatur, Georgia (Dr Hegde); Robert J. Tomsich Pathology & Laboratory Medicine Institute, Cleveland Clinic, Cleveland, Ohio (Dr Hoeltge); the Department of Pathology, University of Vermont College of Medicine, Burlington (Dr Leonard); the Departments of Pathology (Drs Merker and Schrijver) and Pediatrics (Dr Schrijver), Stanford University School of Medicine, Stanford, California; the Department of Pathology & Immunology, Washington University School of Medicine, St Louis, Missouri (Dr Nagarajan); the Department of Laboratory Medicine & Pathology, Mayo Clinic in Arizona, Phoenix (Dr Robetorye); the Departments of Pathology & Laboratory Medicine and Genetics, University of North Carolina at Chapel Hill, Chapel Hill (Dr Weck); and ARUP Laboratories Institute for Clinical and Experimental Pathology, and Department of Pathology, University of Utah School of Medicine, Salt Lake City (Dr Voelkerding). Dr Aziz is now with Phoenix Children’s Hospital, Phoenix, Arizona.

The authors have no relevant financial interest in the products or companies described in this article.

Reprints: Nazneen Aziz, PhD, Phoenix Children’s Hospital, 1919 E Thomas Rd, Phoenix, AZ 85016 (e-mail: naziz@phoenixchildrens.

com).

Arch Pathol Lab Med—Vol 139, April 2015 CAP Laboratory Standards for NGS Clinical Tests—Aziz et al 481 applied currently include inherited diseases, solid tumors, hematologic malignancies, infectious diseases, human leukocyte antigen analysis, and noninvasive prenatal screening to detect fetal chromosome defects.

The number of laboratories offering NGS testing has grown considerably in the past few years, despite the fact that specific CLIA/College of American Pathologists (CAP) laboratory standards had not yet been developed to regulate this technology. To address this need, the CAP formed an ad hoc committee, the NGS Work Group, to develop the first set of clinical laboratory standards for this nascent technology. Given that NGS-based testing represents an evolving technology with continued improvements in instrumentation, sequencing chemistries, and bioinformatic and computational analyses, the work group aimed to develop standards that provide a necessary regulatory framework for clinical NGS tests (which to date are laboratory-developed tests) without inhibiting further adop-tion of NGS-based testing technology.

Next-generation sequencing incorporates 2 processes: (1) the analytic wet bench process and (2) bioinformatics analysis of sequence data. The wet bench component generally includes any or all of the following processes:

handling of patient samples, extraction of nucleic acids, fragmentation, barcoding (molecular indexing) of patient samples, enrichment of targets for exome or gene panels, adapter ligation, amplification, library preparation, flow cell loading, and generation of sequence reads. Sequence generation is almost entirely automated and the output consists of millions to billions of short sequence reads. The wet bench workflow is followed by intensive computational and bioinformatics analyses that use a variety of algorithms to map and align the short sequence reads to a linear reference human genome sequence. After mapping and alignment, variant calls are made at locations where nucleotides differ from the reference sequence. Separate processes develop content needed to analyze the clinical relevance of variants, either singly or in combination, relative to their contribution to a given clinical phenotype.

For individual patient cases, identified variants are evaluated against annotated content to infer the potential for impairments to normal gene function (eg, premature transcript or protein truncation, impact of nonsynonymous amino acid changes to protein function, or alternative splicing). Interpretation requires integrating genomic find-ings with the patient’s clinical phenotype in order to make an informed decision regarding causality and correlation of the deleterious mutation(s) with the patient’s disease. The mapping, alignment, variant calling, and variant annotation steps, and, to some degree, clinical interpretation (if decision support tools are used), comprise the overall bioinformatics analysis workflow.

The CAP NGS Work Group approached the analytic wet bench process and the bioinformatics or ‘‘dry bench’’ analyses as 2 discrete processes requiring separate consid-erations for standards. This division was leveraged to support the fact that some laboratories use external facilities to conduct either portion of NGS-based testing.

In a laboratory offering the entire process from wet bench through bioinformatics analysis, clinical validation of their test will incorporate the validation of both parts. A total of 18 laboratory accreditation checklist requirements for the analytic wet bench process and bioinformatics analysis processes have been included within CAP’s molecular pathology checklist (MOL). The NGS checklist items include new standards for documentation, validation, quality assurance, confirmatory testing, exception logs, monitoring of upgrades, variant interpretation and report-ing, incidental findings, data storage, version traceability, and data transfer confidentiality. As described in this report, the work group’s goal was to initially develop foundational accreditation requirements for NGS that could be applied across multiple testing areas including inherited disorders, molecular oncology, and infectious diseases. It was anticipated that once foundational requirements were in place, there would be the need to subsequently develop additional, discipline-specific (eg, molecular oncology) NGS checklist requirements, and this is further addressed in the ‘‘Comment’’ section. This report describes important issues considered by the NGS Work Group during the development of each of the new checklist requirements. In addition, this report serves as a supple-ment to the CAP NGS checklist requirements and therefore the contents are closely aligned to each require-ment for the 2014 checklist.

WET BENCH ANALYTIC PROCESS

NGS Wet Bench Process Documentation

The Laboratory Uses a Standard Operating Procedure to Document the Analytic Wet Bench Process Used to Generate NGS Data.—The detailed documentation of the wet bench processes is a critical part of quality assessment in the clinical laboratory. All standard operating protocols of DNA/RNA sample preparation, fragmentation, library preparation, barcoding (molecular indexing), sample pool-ing, and sequence generation must be documented so that each step and subsequent manipulations can be traced. This includes documentation of all methods and reagents as well as instruments, instrument software, and versions used throughout the wet bench process. In addition, controls used need to be described. A few examples will be highlighted below. Targeted NGS assays (such as multigene panels or exome sequencing) allow selective capture of genomic regions of interest before sequencing, and detailed information regarding the captured region(s) (using geno-mic coordinates of capture probes and lists of genes) and target-enrichment protocols should be documented. Clinical laboratories that process different types of samples (eg, blood, formalin-fixed paraffin-embedded specimens) should develop standard operating procedures (SOPs) for each validated sample type. The reagents and protocols used for pooled analysis of patient specimens must be detailed and should include the sequence information of the barcodes used for each patient sample. Metrics and quality control parameters used to assess run performance must also be documented. Commonly used metrics include the percent-age of reads mapping to the target region, the fraction of bases meeting specified quality and coverage thresholds, and average coverage/base and target region. The laboratory must define and document acceptance and rejection criteria for the wet bench process inclusive of sample preparation and sequencing. It is critical to determine and summarize regions that failed analysis (eg, due to inadequate coverage) if they are not covered by orthogonal technologies (such as Sanger sequencing).

Evidence of compliance for this requirement includes a written SOP that describes the analytic wet bench process and the ability to demonstrate that the laboratory follows its policies and procedures.

482 Arch Pathol Lab Med—Vol 139, April 2015 CAP Laboratory Standards for NGS Clinical Tests—Aziz et al

NGS Wet Bench Process Validation

The Laboratory Validates the Analytic Wet Bench Process and Revalidates the Entire Process and/or Confirms the Performance of the Components of the Process as Satisfactory When Modifications Are Made.

The Extent of Revalidation and/or Confirmation Is Modification Dependent.—Like all laboratory-developed tests in molecular diagnostics and other areas of the clinical laboratory, analytic performance of NGS procedures must be internally validated before clinical implementation. Next-generation sequencing analysis is a complex procedure with many steps within the wet bench workflow. Each step needs to be individually optimized to empirically determine optimal assay conditions and analysis settings. Once those are in place, an analytic validation must be performed for the whole test in a ‘‘beginning-to-end’’ fashion, including the entire wet bench process as well as the bioinformatic analyses. Essential performance characteristics that need to be determined during the validation are the analytic sensitivity and specificity, accuracy (the degree of closeness of measurements to the actual [true] value), precision (reproducibility and reliability), and limit of detection (if applicable). As for any molecular assay, validation should also be conducted independently for each accepted speci-men type (blood, saliva, tissue, etc). Next-generation sequencing tests are typically designed to interrogate large and multiple regions of the genome, and its use can range from mutational hotspots for oncology applications to gene panels to exomes or genomes. As a consequence, NGS permits the detection of novel as well as known sequence variants, which necessitates a comprehensive approach to be able to determine test performance with adequate confidence. Because it is not possible to validate all theoretically possible variants that can occur, it is necessary to use a combination of a ‘‘methods-based’’ 16 and ‘‘analyte-specific’’ validation approach for determining a test’s analytic performance. Consulting the published literature for studies regarding the accuracy of the relevant NGS platform can be useful to inform the laboratory’s own validation work. In most cases, variants will have been identified via Sanger sequencing, considered (at least for now) the gold standard comparative technique. However, variant validation information may also be obtained from oligonucleotide microarray genotyping data in some cases.

Several professional organizations have issued guidance regarding validation of molecular tests and, more recently, NGS tests in specific to which the reader is referred.17–21

As the NGS Work Group debated NGS validation requirements, the concept of requiring a minimum number of samples for inclusion in a validation was extensively discussed. It was concluded that adding a minimum sample number requirement was premature given the ongoing evolution of NGS technology and the diversity of applica-tions being implemented in diagnostic laboratories. Further, the concern existed that establishing a minimum sample number requirement may result in laboratories conducting an insufficient validation for a given NGS diagnostic application. The work group noted that NGS validations reported in the literature have varied considerably in sample number size (eg, ~20–80 plus samples),22–32 reflecting that individual laboratories are on a validation ‘‘learning curve.’’ The total number of samples that needs to be run to appropriately validate an NGS test is driven partly by the size of the test (larger assayed regions will have more variants available for deriving their technical performance), by the number of specific analytes (variants) that need to be assessed, by the possible requirement to determine limit of detection across a range of allele frequencies, and by the number of runs and samples needed to determine precision.

At this juncture in time, the NGS Work Group concluded that statistical considerations with regard to the number of samples cannot be universally or comprehensively applied across the numerous assays that are possible when using NGS (eg, amplicon versus targeted capture; small numbers of genes versus exome or genome; inherited disease versus oncology versus infectious disease) as the sequencing methodology. Therefore, we have described different scenarios (eg, samples needed for methods-based approach, samples needed to assess reproducibility and reliability, and clinical samples used to assess diagnostic specificity and sensitivity), each of which will necessitate samples whose numbers will vary with the context of each assay. We emphasized the principles of validation in the requirements and several analytic performance parameters as highlighted below.

Analytic sensitivity can be assessed by using a methods-based approach that aims at maximizing the number of sequence variants that are compared to a gold standard method to increase confidence of analytic performance.

These values may then be extrapolated to all bases. For this methods-based approach, pathogenicity of analyzed vari-ants does not matter as this has no bearing on their technical detectability. However, it is important to deter-mine this ‘‘baseline’’ performance by using as many different genomic regions as possible, as sequence context can be an important influence. In addition, laboratories should determine analytic performance separately for all variant types that are relevant for the test (eg, single nucleotide variants, indels, copy number variants, structural variants, homopolymers). Approaches to maximize the number of appropriately identified variants may include cumulative analysis of different in-house–developed tests (eg, different gene panels), provided that they rely on identical protocols. In addition, several publicly available databases provide exome/genome-wide variant calls that can be used in the clinical validation efforts (eg, HapMap or 1000 Genomes). In addition, the Centers for Disease Control and Prevention and National Center for Biotech-nology Information have collaborated to establish a Web browser to facilitate access to 2 well-sequenced genomes (NA12878 and NA19240)33 and to provide access to clinical-grade targeted data sets (gene panels) and exome/genome-wide data sets created by various laboratories.34 These databases provide access to large sets of variants that can aid in deriving technical performance specifications. However, an analyte-specific validation may be necessary in addition to the more global methods-based approach when the NGS test includes genes that are known to harbor well-known, disease-causing variants. In such cases, it is important to include traditional positive controls with patient samples, including relevant variants (eg, p.F508del in CFTR) to demonstrate adequate detection by that NGS test. Analytic specificity is often calculated by using ‘‘negative’’ samples (ie, samples that have no pathogenic variant) to determine the fraction that is correctly identified as negative. However, this concept does not work well for NGS-based tests. Once again, a methods-based approach can be leveraged to calculate analytic specificity across the assayed region, for example, by determining the false-positive rate (fraction of

Arch Pathol Lab Med—Vol 139, April 2015 CAP Laboratory Standards for NGS Clinical Tests—Aziz et al 483 variants detected that are incorrect calls). It is also useful to determine the average number of false-positive calls for the regions tested in a clinical sample. Note that the analytic specificity accounts for numerous sources of type I error, including base-calling error, errors due to misalignment, and variant-calling errors. Determining the limit of detec-tion is important for assays that interrogate samples with heterogeneous genotypes (eg, tumor specimens, maternal blood used for noninvasive prenatal testing for fetal aneuploidies, and mosaic specimens). This can be challeng-ing given that Sanger sequencing, which is often used as an alternate gold standard technology during validation, is not as sensitive as NGS. Sample ‘‘mixing’’ experiments (eg, dilution of samples with known allele frequencies) may be used to assess the limit of detection of each variant type.

Precision (interrun and intrarun variability) should be determined by using at least 3 samples. For tests that are performed with single-lane sequencers, intrarun variability may be determined by using bar-coded replicates of the same sample.

Homologous sequences such as pseudogenes can inter-fere with accurate variant calling and therefore pose significant challenges for correctly analyzing affected genes.

An upfront bioinformatics homology analysis is useful to determine possible interference by homologous sequences.

In addition, read-mapping quality can be used to identify problematic regions. If such genes are included in the NGS test, the laboratory must devise a method to ensure that identified variants are not due to pseudogene sequence and must document the accuracy of the method. When pooled sequencing of bar-coded samples is performed, the labora-tory must document that individual sample identity is maintained throughout the wet bench process.

The extent of revalidation and confirmation is dependent on the magnitude of the introduced changes and their potential consequences. For example, minor changes, such as the introduction of a new lot of capture reagent that has already undergone comprehensive validation, can be addressed by confirming adequate performance. In this example, it would be deemed acceptable if the laboratory sequences a previously tested sample and documents that the main run metrics (eg, coverage, read quality) are unchanged and that the same results are obtained.

Conversely, a major change, such as the introduction of a new sequencing platform or different target enrichment method, would require a more extensive revalidation.

NGS Wet Bench Process—Quality Management Program

The Laboratory Follows a Documented Quality Management Program for the NGS Analytic Wet Bench Process.—CAP-accredited laboratories must develop and follow a quality management plan. The CAP All Common Checklist (COM) applies to every part of a multispecialty laboratory and includes entire sections on Quality Manage-ment and Test Method Performance. However, NGS Wet Bench Process—Quality Management Program was added to the NGS portion of the checklist to highlight the particular needs of laboratories performing NGS. No two quality management programs are alike. Each is shaped by the laboratory’s scope, clinical market, and expertise, and the laboratory director is given wide latitude in the design of the quality assurance program. The design of the program must be written, and compliance with that design docu-mented. A good quality assurance program for laboratories performing NGS will include the following attributes35,36:

1. The quality assurance program follows the path of workflow. The programs should assess preanalytic steps occurring before NGS, analytic testing, and postanalytic processes used in sequence analysis through reporting.

2. The NGS quality program should be integrated within the institution’s overall quality assurance program. If it is part of a larger institution, such as a hospital or medical center, the NGS quality program should fit well within its overall context.

3. The program should address common problems that arise in the course of testing. ‘‘Problems’’ include events that can affect the test result or its clinical use as well as nonconformance with the laboratory’s own policies and procedures. Documentation includes both review of the effectiveness of corrective actions taken and the revision of policies and procedures intended to prevent recur-rence.

4. The overall goal of the quality program aims to ensure that testing is clinically relevant. This is particularly important for tests such as NGS, for which no comparative analytic result of greater sensitivity may exist. The appropriateness of test orders and analytic decisions must be grounded in medical science and evidence.

5. The program should also encourage laboratory employ-ees to communicate concerns about the quality of laboratory testing. The investigation of employee com-plaints and suggestions must be a part of the quality assurance program.

NGS Confirmatory Testing

The Laboratory Has a Policy That Documents Indica-tions for Confirmatory Testing of Reported Variants.— While the accuracy of NGS technologies is continuing to improve, it is widely accepted that most NGS-based sequencing assays will yield false-positive and false-negative results. CAP preferred to give laboratories per-forming NGS-based assays flexibility in determining when confirmatory testing should be performed, how this testing is performed, and whether to recommend confirmatory studies for follow-up testing for additional family members, which may or may not be NGS based. For example, some laboratories might determine during validation studies that confirmatory testing of identified variants was not necessary owing to the very high coverage achieved by their assay (ie, 10003 coverage of a single-gene NGS-based assay) and/or very high confidence in the identified variants.37,38 However, others may find that they need confirmatory testing by an alternative method to achieve the desired confidence in the variants that are reported. Some laboratories might decide that they will perform confirmatory testing on variants for a predetermined trial period and then reevaluate this decision at a later date. Each laboratory performing NGS must have a policy in place that clearly documents indications for confirmatory testing and/or documents how their assay validation determined that such testing was not required.

Laboratories must be able to document compliance with their confirmatory testing policy and show evidence of ongoing monitoring of their NGS assay(s) to ensure that the benchmarks achieved during the validation process are maintained during the routine performance of NGS-based clinical testing and variant reporting. CAP also desired to give laboratories flexibility in deciding the methods used to perform any needed confirmatory testing. Although Sanger sequencing is likely to be the method most commonly

484 Arch Pathol Lab Med—Vol 139, April 2015 CAP Laboratory Standards for NGS Clinical Tests—Aziz et al chosen for confirmatory testing of NGS-identified variants, CAP did not want to mandate such testing in order to provide clinical laboratories with the flexibility to use other appropriate confirmatory testing methods consistent with the existing expertise of the laboratory and the type and frequency of variants requiring confirmation (ie, allele-specific polymerase chain reaction, melting curve analysis, other NGS-based method).

Laboratory Records

Methods, Instrument(s), and Reagents Used for Processing and Analyzing a Sample (or Batch of Samples) Can Be Identified and Traced in the Labo-ratory’s Records.—Comprehensive records of laboratory assay ‘‘runs’’ are essential to document the conditions and events associated with the complex processes and algo-rithms involved in the performance and interpretation of clinical NGS–based analyses. Accordingly, such archived information must be maintained within an overarching framework where all reagents, primers, sequencing chem-istries, and platforms used for the analysis of each patient sample are traceable. Such records must contain a descrip-tion of the test performed including the nature of the targeted sequence (eg, genome, exome, specific genes for targeted panels, transcriptome, or methylome) and depth of coverage (eg, range and average). It is also necessary to cite details of the analysis, including any publications or Web sites (with dates accessed) describing the pertinent param-eters or other information and/or notations relative to the testing and reporting processes. While all details of the analysis need not be included in the patient report, it is critical that the laboratory maintain a documentation system from which detailed information regarding the analysis of individual patient specimens can be obtained.

Exception Log

The Laboratory Maintains an Exception Log for Patient Samples Where Steps Used in the NGS Analytic Wet Bench Process Deviate From Standard Operating Procedures.—The laboratory must document any deviation from the SOP along with an explanation for the deviation, and the resulting outcome. Examples of anticipated devia-tions may include altered processing upon receipt of a suboptimal specimen, changes to the library preparation, and sequencing of libraries with suboptimal concentrations.

Exceptions may pertain to specimen quality and to the analytic process. At the time of specimen accessioning, an assessment is made as to whether or not a sample is in optimal condition for testing. If there is a concern, this can be documented on the worksheet or on a pending log and communicated to a supervisor or laboratory director. The director may decide to proceed with the testing, but should communicate the issue to the ordering physician and document this communication electronically or on the worksheet. One example of such a scenario is a sample that was not transported under optimal conditions. A decision may be made to process the sample and to proceed with subsequent testing only if the DNA specimen is found to be adequate.

Issues related to specific steps of the wet bench procedure should be reported to the laboratory supervisor or the director of the laboratory. It can then be assessed whether or not the testing was compromised and if the testing can be completed. If, after troubleshooting, the testing is assessed as satisfactory, the results can be interpreted by the laboratory director, provided that the quality controls of the run and the sample results are deemed adequate. All aspects of the testing issue(s) should be thoroughly documented in an ‘‘exception log,’’ including the trouble-shooting, the resolution, and the pertinent communications (especially regarding who was involved and who was informed by whom and on what date), and may also be incorporated into the monthly quality assurance report.

On occasion, the laboratory SOP itself may have to be revised to improve phrasing, to make process steps more clear, or to remove small inaccuracies in order to optimize the protocol. In such cases, the proposed correction should ideally be supported by at least 2 additional individuals, including the laboratory supervisor and either the technol-ogist who developed the assay or a reference technologist.

Any such corrections must be approved, signed, and dated by the director of the laboratory. This is not an exception log issue per se but rather a correction in the manner the assay is described.

Monitoring of Upgrades

The Laboratory Has a Policy for Monitoring, Imple-menting, and Documenting Upgrades to Instruments, Sequencing Chemistries, and Reagents or Kits Used to Generate NGS Data.—Laboratories must be aware of upgrades to ensure that they are not using obsolete methods. The laboratory must implement a policy to monitor and implement upgrades to instruments, sequenc-ing chemistries, and reagents or kits used to generate NGS data. The policy should address how laboratories perform-ing NGS-based testing can ensure that they are using the most up-to-date sample library preparation as appropriate for that assay, clonal fragment amplification, and sequenc-ing methods in this rapidly evolving environment provided that these newer methods have been validated by the laboratory to improve the quality, reproducibility, and accuracy of the assay. The policy should also address the methods used to monitor upgrades and when a relevant upgrade(s) will be implemented and further validated before productive clinical use. For example, the laboratory’s policy may be to monitor and implement upgrades at specified intervals (such as quarterly, biannually, or annually), depending on the relevance of the new upgrade for enhancing assay performance. Additionally, since the implementation of upgrades may require revalidation of the entire wet bench process, or at least the relevant steps, it may be convenient to set time intervals accordingly.

BIOINFORMATICS PROCESS

A variety of open-source and commercial bioinformatics algorithms and software is available for analyzing NGS data.39 While these tools continue to improve, they each have strengths and weaknesses with respect to their performance in diagnostic applications. Operationally, the bioinformatics processes applied to NGS data can be conceptualized into 3 major steps. First, is the generation of a sequence read file consisting of a linear nucleotide sequence (eg, ACTGGCA), with each nucleotide assigned a numerical value (termed its base quality score) that correlates to its predicted accuracy. The generation of sequence read files uses instrument-specific software that analyzes several physical parameters, such as signal to noise ratios, during the sequencing run. Sequence read files are usually configured in the FASTQ file format, which contains the compilation of individual sequence reads, each with its own

Arch Pathol Lab Med—Vol 139, April 2015 CAP Laboratory Standards for NGS Clinical Tests—Aziz et al 485 identifier, and an associated base quality score for each nucleotide. FASTQ files have become a dominant form of information exchange in the field of NGS. The next step consists of aligning the sequence reads to a reference sequence, typically a human genome reference sequence, to identify differences between the patient sequence reads and the reference. Identified variants may include single nucleotide variants, insertions and deletions, copy number variants, and other structural variations (translocations, inversions, etc). Identified variants are then annotated to provide information regarding their impact on gene and protein function. Separate processes within the laboratory implement, or otherwise develop, curated content for assessing the clinical relevance of particular variants to a given disease or condition. Lastly, annotated variants are interpreted within the context of the patient’s phenotype to render a clinical report. For gene panels and exome or genome sequences, the large list of annotated variants is typically reduced by excluding variants with a higher population frequency and by focusing on rare variants that are of greatest predicted deleterious impact that correlate with patient phenotype.40,41 When analyzing exome or genome sequences within a family unit, variant prioritiza-tion typically takes into account variant cosegregation within the family, based on affected versus unaffected family members. Variant prioritization during the tertiary step uses previous knowledge of association of variants and disease within public or private databases of human mutation, such as the Human Gene Mutation Database (HGMD),42 Online Mendelian Inheritance in Man (OMIM),43 and/or other disease/locus-specific databases.44

Developing a cohesive diagnostic pipeline that incorpo-rates bioinformatics steps, and content development for variant annotation, usually requires the integration of multiple algorithms and software applications. As such, laboratories must empirically determine which algorithms and associated bioinformatics tools to apply to each diagnostic application. An iterative pilot process commonly uses known patient samples and training data sets, which may be synthetic or from prior cases, to test algorithms and software parameters. Having established a working set of bioinformatics tools and parameters, the laboratory per-forms a bioinformatics validation with a larger set of samples to determine analytic sensitivity and specificity for the types of variants assayed (eg, single nucleotide variants, insertions and deletions, homopolymer or repetitive se-quences, or copy number variants) and reproducibility (ie, concordance within and across runs, instruments, and technical personnel). The samples used for validation will contain previously confirmed variants, or the identified variants may be confirmed post bioinformatics analysis. The validation may confirm that the bioinformatics tools and parameters are performing satisfactorily (eg, high specificity and sensitivity if the assay is a stand-alone assay for variant detection and reporting versus high sensitivity if it is a screening assay followed by a second assay that is used for confirmation) per laboratory requirements and clinical criteria for reporting, or adjustments or alternative tools may need to be further evaluated.

Once a satisfactory bioinformatics validation has been achieved, translation of the NGS assay into the clinical laboratory requires that laboratories document all aspects of the bioinformatics processes used for clinical diagnostics and implement a quality management program for these steps. Further highlights of the bioinformatics requirements for NGS are discussed below.

NGS Bioinformatics Pipeline Documentation

The Laboratory Uses an SOP to Document the Bioinformatics Pipeline Used to Analyze, Interpret, and Report NGS Results.—Laboratories must document all algorithms, software, and databases (referred to as components) used in the analysis, interpretation, and reporting of NGS results.45 The versions of each of these components in the overall bioinformatics pipeline must be recorded and traceable for each patient result (Version Control). For each component, the laboratory may use a baseline, default installation, or may customize the pipeline by using alternate configuration parameters in deploying individual bioinformatics tools or in running specific algorithms. In either case, laboratories must document any customizations that vary from default configuration or should indicate which parameters, cutoffs, and values are used. Most NGS bioinformatics analyses are conducted by aligning sequence reads to a reference sequence. The reference sequence version number and assembly details need to be identified. When describing the bioinformatics pipeline, laboratories should document the overall workflow of data analysis and include the input and output files for each process step. For each step, laboratories should also develop and document quality control parameters for optimal performance. For example, in the primary step, a laboratory would determine acceptable criteria such as the number of reads passing instrument-specified quality filters.

Criteria for variant calling are essential and parameters that are invoked include thresholds for read coverage depth, variant quality scores, and allelic read percentages. Each of these requirements applies to multigene panel applications as well as to exome and genome sequencing. Laboratories should also document the bioinformatics processes that are used for reducing a large variant data set to a list of causal and/or candidate genes and/or variants. For example, in inherited disease assays, laboratories should document approaches used to identify recessive, dominant, and de novo variants. Evidence of compliance for this requirement would be demonstration of appropriate documentation and that the laboratory follows its outlined procedures.

NGS Bioinformatics Pipeline Validation

The Laboratory Validates the Bioinformatics Pipeline and Revalidates the Entire Pipeline and/or Confirms the Performance of the Components of the Pipeline as Satisfactory When Modifications Are Made. The Extent of Revalidation and/or Confirmation Is Modification Dependent.—As with wet bench processes, laboratories use an iterative process during the establishment of a bioinformatics pipeline that involves analyzing sequence read files containing known variants and demonstration that the pipeline can identify the variants.17 For laboratories offering the entire process from wet bench through bioinformatics analysis, the validation of the bioinformatics pipeline should be included in the overall test validation.

Once the laboratory has developed and empirically deter-mined optimal performance, and performed adequate testing of its pipeline, the next step is to perform and document a comprehensive validation, again using se-quence reads generated from samples with variants that cover the spectrum of the diagnostic testing that the laboratory intends to perform. These steps are essential for

486 Arch Pathol Lab Med—Vol 139, April 2015 CAP Laboratory Standards for NGS Clinical Tests—Aziz et al both in-house–developed tools and in those cases where a vendor-provided tool or pipeline is used in a manner where it is locked down, for example, the laboratory does not modify or alter any components or parameters of the underlying tools. As with wet bench processes, a sufficient number of samples need to be analyzed to assess the pipeline’s analytic and diagnostic sensitivity and specificity as well as the assay’s reproducibility. The number of samples assessed should be determined from the assay.

Parameters such as the number of genes assessed, which regions of a gene are assessed, and types of variants that need to be detected should ultimately be used to determine the number of control, well-characterized samples (eg, HapMap samples or cell lines with known inherent or engineered variants) and previously analyzed diagnostic samples. The presence of pseudogene sequences and other sequences highly homologous to the target are known to interfere with accurate sequence mapping, alignment, and, by extension, variant calling. The degree of interference, if applicable in a given diagnostic assay, needs to be determined. While it may be possible to address the challenge of coalignment of highly homologous sequences bioinformatically, laboratories may need to set up indepen-dent alternative method assays for these problematic regions. The NGS Work Group acknowledged that it was not feasible to comprehensively and exhaustively define the error rates for false positives and false negatives in variant calls. However, the laboratory should assess the error rates for several representative examples by variant type. These rates may be assessed analytically by using well-character-ized, control samples. False-positive error rates can be ascertained by sequencing using an alternative method.

False-negative error rates are more difficult to ascertain because they may originate from several sources, including insufficient read coverage in a given target region, a lack of variant calling due to parameters such as lower variant quality scores, or a distribution of sequence read directions (ie, forward and reverse reads) that does not meet quality control criteria. In the process of validating variant calls by an alternative method, it may be possible to analyze flanking regions to determine if the variant calling pipeline is identifying all possible variants. For example, when using Sanger sequencing to confirm a variant, primer pairs can be designed to sequence the region of the variant as well as generous portions of flanking regions, which can then be inspected for the presence of variants and correlated with those identified by the bioinformatics pipeline.

A now common practice in NGS is the use of molecular barcodes or indexes during the preparation of libraries.

Indexed sequences need to be validated with respect to their uniqueness in a pool and the pipeline must be able to accurately bin (segregate) such indexed sequences. In the analysis of indexed and pooled samples, it is essential to establish criteria for retention or exclusion of sequence reads. For example, some laboratories will only accept sequence reads with indexes in which the index sequence is identical to the index that was used during library preparation. Other indexed reads that do not align in a completely identical fashion are not assigned to the respective sample. The monitoring of the percentage of indexed reads that maintained full identity can be a measure of the presence of contamination from other index sequences. For those assays in which limit of detection is relevant, such as identification of somatic mutations in tumor samples, the bioinformatics pipeline needs to be assessed for that parameter. One approach that can be used to validate limit of detection is to sequence samples with decreasing concentrations of target variants that have been created from a cell line or DNA dilution series.

Validation of the bioinformatics pipeline for identification of variants is application specific and the above discussion is broadly pertinent, with the exception of the limit of detection analyses being specific to samples with heteroge-neous genotypes. When using exome and genome sequenc-ing for causal and candidate gene identification, the laboratory must additionally validate its bioinformatics pipeline for this purpose. For example, in the case of inherited diseases, laboratories may approach this by analyzing sequence read sets with known pathogenic variants that are present in several deleterious variant configurations, such as recessive, dominant, and de novo.

Once a bioinformatics pipeline has been validated to meet laboratory requirements and has been implemented, reval-idation is required when any changes are made in the pipeline. A practical approach that can be used to revalidate a sequencing pipeline is to use sequence read files from the original validation and simply reanalyze them with the new parameters. This approach may result in identical, smaller, or larger numbers of identified variants and these findings would need to be confirmed. For exome and genome sequencing, changes in bioinformatics pipelines can also result in a new list of presumptive causal or candidate genes.

Evidence of compliance for a bioinformatics pipeline validation/revalidation would include the records of valida-tion and any subsequent revalidation and their documented approval for clinical use.

NGS Bioinformatics Pipeline—Quality Management Program

The Laboratory Has a Documented Quality Management Program for the NGS Bioinformatics Pipeline.—The routine application of a validated bioinfor-matics pipeline must be accompanied by monitoring of laboratory-determined quality control metrics.46 Divergence from expected quality metrics during the analysis of clinical samples requires investigation and resolution. Some exam-ples include the following situations: the bioinformatics output of NGS data analysis may demonstrate that an insufficient number of sequence reads passed the expected or required base quality score threshold. Alternatively, the number of variants identified in a data set may deviate substantively from an expected value, based on prior information regarding known frequencies of variation in the human genome. Another example may be an inappro-priately high number of indexed sequencing reads that cannot be specifically segregated. Such deviations may indicate a technical aberration or process failure occurring during technical wet bench procedures or during a step in the bioinformatics pipeline. An appropriate quality man-agement program provides the structure and process for investigating these divergences to pinpoint possible causes, and institute appropriate corrective measures. Laboratories must maintain a record of deviations from expected results and document the investigative measures that were used to determine the cause as well as the corrective measures that were implemented. Evidence of compliance would include documentation of monitoring quality control metrics as well as records describing any divergences, including appropriate investigative measures and subsequent corrective actions.

Arch Pathol Lab Med—Vol 139, April 2015 CAP Laboratory Standards for NGS Clinical Tests—Aziz et al 487

Bioinformatics Pipeline—Updates

The Laboratory Has a Policy for Monitoring, Documenting, and Implementing Patch-Releases, Up-grades, and Other Updates to the Bioinformatics Pipeline.—This checklist item addresses the requirement for laboratories to establish and follow a procedure for identifying and implementing updates to components of the bioinformatics pipeline. Next-generation sequencing bio-informatics pipelines often use multiple packages of open-source software with additional scripts and databases for managing content and aspects of analysis and reporting.

Owing to the ongoing evolution of the field, laboratories must have a policy for monitoring updates, patch-releases, and other upgrades to the bioinformatics pipeline. This policy should also address when such updates will be implemented. For example, the laboratory may decide to do this at the time of the update release or at specified intervals (such as quarterly, biannually, or annually), depending on the nature and relevant urgency of the update.