Attachment 7 Accuracy Standards.doc

Attachment No. 7, Accuracy Standards

Accuracy Standards

Per C.4.3.8 the Government intends to award a separate Contract for the performance of quality assurance review of the work performed under this contract for Automated Litigation Support services. The Government expects the Contractor to provide a ‘complete’ and ‘accurate’ work product for all of its pipeline processes. ‘Completeness’ shall refer primarily to the output of electronic file processing (EFP) as well as the migration of data from one workspace / platform to another, but may include other processes as defined by the COR. ‘Accuracy Standards’ shall apply primarily to processes such as coding, keying, OCR and digital image scanning, but may include other processes as defined by the COR. These standards are defined below.

1. Completeness standards for EFP

Unless otherwise specified by or agreed to by the COR, the Contractor shall demonstrate the completeness of its EFP work product. Where the Contractor is required to submit a notification of this work to the Quality Assurance (QA) vendor, the following conditions shall be met. Where any of these conditions are NOT met and where this is duly noted by the QA vendor, it will be in the COR’s discretion whether to accept or reject that work.

· The Contractor shall ensure that the information contained on its QA notification is correct and, where appropriate, include appropriate explanatory language to assist the QA Vendor in assessing completeness.

· The Contractor shall ensure that the original source media (e.g., hard drive, thumb drive, DVD) being submitted is correctly labeled, contains the correct data and that the data is accessible to the QA Vendor.

· The Contractor shall ensure that the database workspace is accessible to the QA Vendor.

· The Contractor shall ensure that the file count on the e-media matches that in the respective database.

· The Contractor shall ensure that the QA Numbers assigned in the database match those recorded on the notification.

· The Contractor shall ensure that the Document Numbers assigned match those recorded on the notification.

· The Contractor shall ensure that the File Names on the notification and in the database match those contained on the original source media

· The Contractor shall ensure that the metadata is loaded to the correct fields in the database.

· The Contractor shall ensure that there are no unaccounted for Gaps, Overlaps or Duplicates in the database. Such accounted for conditions shall be included on the QA notification.

· The Contractor shall ensure that archive files such as .ZIP, .PST etc. were appropriately and properly processed.

· The Contractor shall ensure that none of the excluded files, as listed in the accompanying Exception Report, were loaded to the database.

2. Completeness Standards for Data Migration

Unless otherwise specified by or agreed to by the COR, the Contractor shall ensure specific conditions of its data migration from one workspace / platform to another are met, in demonstrating completeness of that process. Where the Contractor is required to notify this work to the Quality Assurance (QA) Vendor, the following conditions shall be met. Where any of these conditions are NOT met and where this is duly noted by the QA Vendor, it will be in the COR’s discretion whether to accept or reject that work.

· The Contractor shall ensure that the QA Vendor has both access and export rights to the appropriate database workspaces.

· The Contractor shall ensure that the information contained on the QA notification is correct and, where appropriate, includes appropriate explanatory language to assist the QA Vendor in assessing completeness.

· The Contractor shall ensure that both databases contain the same number and types of fields.

· The Contractor shall ensure that both databases contain the same number of records.

· The Contractor shall ensure that the metadata has been populated to the correct fields.

· The Contractor shall ensure that the correct number of native files and images are reflected in the new database.

· The Contractor shall ensure that all of the naming conventions for folders, tags, searches, etc are reflected in the new database.

3. Accuracy Standards for Digital Imaging Unless otherwise specified by or agreed to by the COR, the Contractor shall ensure the following conditions are met with respect to generation of digital image files.

· The Contractor shall ensure that all images, whether scanned to images from paper or rendered through the EFP process, are generated at 300 dpi single-page TIFF files, CCITT Group IV (2D Compression).

· The Contractor shall ensure that all images are of a standard length, uniquely and sequentially numbered and endorsed, and stored with the “.tif” extension.

· The Contractor shall ensure that all images are accompanied by a Concordance® Image Cross Reference file that associates each Bates number with its corresponding single-page TIFF image file. The Cross Reference file should also contain the image file path for each Bates numbered page.

Image Cross Reference File Sample Format:

ABC00000001,OLS,D:\DatabaseName\Images\001\ ABC00000001.TIF,Y,,, ABC00000002,OLS,D:\DatabaseName\Images\001\ ABC00000002.TIF,,,, ABC00000003,OLS,D:\DatabaseName\Images\001\ ABC00000003.TIF,,,,

· [E-media considerations? E.g – all pages of a document or all pages of a collection of documents that comprise a folder or other logical grouping, including a box, shall be delivered on a single piece of media; no image folder shall contain more than 2000 images]. From Civil Production Specs Sept 2017.

4. Accuracy Standards for OCR The Government recognizes that OCR is a function now predominately performed in an automated environment as part of the EFP process. Given the recognized industry tools that can perform text extraction, an accuracy standard based on a defined accuracy algorithm and an assigned (%) value is not (no longer?) a feasible measure. The Government…accuracy…as defined by the COR.

5. Accuracy Standards for coding/keying The Government recognizes that ‘coding’ and ‘keying’ functions per se are now largely obsolete and that metadata related tasks are predominately addressed in an automated environment as part of the EFP process. Given the recognized industry tools that can perform metadata extraction, an accuracy standard based on a defined accuracy algorithm and an assigned (%) value is not (no longer?) a feasible measure. The Government… accuracy… as defined by the COR.

Do we completely exclude references to the below standards, then?

Accuracy: Definition and Formula

Subject to the conditions stated below, the Contractor shall ensure that the document coding and data reduction tasks are performed at a level of quality such that:

1.

all recorded information is accurate;

2.

all relevant information is recorded; and, 3.

no irrelevant information is recorded.

"Accuracy" shall be determined on the basis of a character count. The formula for determining the accuracy of each batch of document coding or its associated data tape shall be:

A - (B + C + D) X 100

A where:

A = all characters which should be recorded

B = all recorded characters which are inaccurate

C = all relevant characters which are not recorded

D = all irrelevant characters which are recorded.

As the Statement of Work provides, the COR shall determine for each Task Order the size of the coding/keying "batch." For example, assume the document coding forms associated with the documents from microfilm reels ABC 001 through ABC 003 should contain 10,000 characters (in the fields other than Document Number and Document Date). If the coding batch in this example had 10 relevant characters which were inaccurately coded ("B" in the formula above), 5 relevant characters which were omitted ("C") and 6 irrelevant characters recorded ("D"), the accuracy of the coding batch would be:

10,000 - (10 + 5 + 6)

10,000

X 100 = 99.79%

This coding batch would be rejected on the basis of the accuracy requirements and definition of "characters" given in Section II and III below:

II.

Document Coding: Definition of "Characters"

For the purposes of bill and determination of coding accuracy, characters shall be counted as illustrated below:

EXAMPLE 1:

(Document Date field)

*DT

760101

6 characters

EXAMPLE 2:

(Document Title, illustrating a variable length text field)

REPORT ON EXPENDITURES

20 characters

EXAMPLE 3:

(Document Type field, illustrating "check-offs" for valid field types)

*TY

(LET) ; MEM ; REP ; (CHR)

2 characters

Only the coded characters, as defined and illustrated above, are counted.

For litigation document databases, the Document Number and Document Date fields must be 100% accurate. A single error in either of these fields shall constitute grounds for rejection of the entire batch. The minimum acceptance standard for the remaining fields will be 99.95% based on a character count, as defined and illustrated above.

For all databases other than litigation document database, the COR will define the field which constitutes the "unique identifying record number" and any other primary sort fields (for instance "Deposition Date" in the Case Management System). For each of these databases, the "unique identifying record number" and the other primary sort fields (no more than two other sort fields) must be 100% accurate. A single error in such a field constitutes grounds for rejection of the entire batch. The minimum acceptance standard for the remaining fields will be 99.95% based on a count of characters.

All determinations of accuracy will be made by comparing the document coding forms with the associated source documents for a statistically valid random sampling of documents from the batch. Coding deliverables will not be considered accepted until 30 days after the successful loading of all DCFs in a given batch, including correction of errors detected during the loading process.

[Note: The contractual standards for accuracy are provided above. However, in practice, the current acceptance standards are 98.5% accuracy for Key Fields and 95% accuracy for Other Fields.]

III. Keying/Key Verification: Definition of "Characters"

For the purposes of billing and determination of data reduction accuracy, characters shall be counted as illustrated below:

(Document Date Field)

*DT

760101

1 + 1(tab) + 6 + 1 (enter)

9 characters

EXAMPLE 2:

(Author field, illustrating all sub fields completed)

*AU

JONES,LP

ABC MINING

l + l(tab) + 8 + 1(DELIM.) + 10 + l (enter) = 22 characters

EXAMPLE 3:

(Title field)

*TI

REPORT ON EXPENDITURES

1 + 1(tab)

+ 1 (enter)

20 characters

EXAMPLE 4:

(Document Type field, illustrating "check-offs" for valid field types)

*TY

LET ; CHR

1 + 1(tab) 3 + 1(delim.) 3 + 1(enter)

10 characters

Only the keyed characters, as defined and illustrated above, are counted; key verification characters are not counted.

For litigation document databases, the Document Number and Document Date fields must be 100% accurate. A single error in either of these fields shall constitute grounds for rejection of the entire batch. The minimum acceptance standard for the remaining fields will be 99.95% based on a character count, as defined and illustrated above.

For all databases other than litigation document database, the COR will define the field which constitutes the "unique identifying record number" and any other primary sort fields (for instance "Deposition Date" in the Case Management System). For each of these databases, the "unique identifying record number" and the other primary sort fields (no more than two other sort fields) must be 100% accurate. A single error in such a field constitutes grounds for rejection of the entire batch. The minimum acceptance standard for the remaining fields will be 99.95% based on a count of characters.

All determinations of accuracy will be made by comparing the keyed data with the associated document coding forms for a statistically valid random sampling of document coding forms from the batch. Keying deliverables will not be considered accepted until 30 days after the successful loading of all DCFs in a given batch, including correction of errors detected during the loading process.

[Note: The contractual standards for accuracy are provided above. However, in practice, the current acceptance standards are 98.5% accuracy for Key Fields and 95% accuracy for Other Fields.]

IV.

Overall Database Accuracy Standards

For purposes of determining overall database accuracy with respect to the documents, characters shall be counted as illustrated below:

EXAMPLE 1: (Document Type Field)

MEM

3 Characters

CHR

3 Characters

6 Characters

EXAMPLE 2: (Author Field)

ANDREWS,J

SAS

l2 Character

Only characters of data present in the database are counted; tags, "repeat" numbers, trailing blanks, etc. are not counted.

For litigation document databases, the Document Number and Document Date fields must be 100% accurate. A single error in either of these fields shall constitute grounds for rejection of the entire batch. The minimum acceptance standard for the remaining fields, when compared with the information in the documents themselves, shall be 99.90% based on a character count, as illustrated above.

For databases other than litigation document databases, the COR will define the fields which constitute the "unique identifying record number" and the other primary sort fields (no more than two other sort fields). For each of these databases, the "unique identifying record number" and other primary sort fields must be 100% accurate. The minimum acceptance standard for the remaining fields will be 99.90% based on a character count, as illustrated above.

All determinations of accuracy will be made by comparing printouts of the database records with a statistically valid sampling of the hard copies of the documents which they represent. Coding, keying, tape loading, and error correction deliverables will not be considered accepted until 30 days after the successful loading of all DCFs in a given batch, including correction of errors detected during the loading process.

[Note: The contractual standards for accuracy are provided above. However, in practice, the current acceptance standards are 98.5% accuracy for Key Fields and 95% accuracy for Other Fields.]

V.

Accuracy of OCRed Text

For purposes of calculating the accuracy of OCRed text, characters and characters in error will be counted as per the following examples:

[Original Transcript Text] more if they look like it would be complicated. In the

Total character count = 55.

[OCRed Text For Same Line] more hey look like it would be cohplicated. In.the

Total missing characters = 3

Total incorrect characters = 1

Total unnecessary/irrelevant characters = 1.

Accuracy will be measured by visually comparing OCR output against the original page. Accuracy for this line would be calculated as follows:

55 – (3 + 1 + 1)

55 x 100 = 90.9%

Notes:

Extra spaces between words in the OCRed text are not counted as errors. However, an extra space in the middle of a word is counted as an error.

Special characters inserted by the OCRing device to flag potential OCRing errors are counted as erroneous characters.