Attachment_(8)_Case_Classification_and_Documentation_Procedures_Manual.pdf

Case Management System Quality Assurance Review

Procedures Prepared For:

Department of Justice Civil Division

Office of Management Information

Prepared By:

DRC

Metro Place One

2650 Park Tower Drive Suite 300

Vienna, VA 22180

DRAFT- December 15, 2006

DYNAMICS

RESEARCH

CORPORATION

Table of Contents

1.0 Introduction 4

2.0 Classification QA Reviews 5

¯ 2.1 TrialiPrefiling Classification, Appellate Classification and Contractor QC Reviews

2.2 Contractor Processing ....~ ............................................ ...................’ ....~ ..............................ii

2.3 Sample Selection ......’ ...................................................................................~

2.4 QA Review Procedures

2.5 Completing the QA Review Form

2.6 Accuracy Rate Calculation

2,6.1 Caption Keying Accuracy ..................................................................................................15.

2.6.2 Content Accuracy

2.6.3 Combined Accuracy

2.7 Contractor QC Reviews

3.0 DBI QA Reviews 17

3.1 The DBI QA Review

3.2 Reviewing CASES for Updates After the Interview ....................................................,

3.3 Completing the QA Forms

3.4 Accuracy Rate Calculation

3.4.1 DBI Field Accuracy

3.4.2 Other Field Accuracy

3.4.3 Combined Accuracy

3.5 Pre-Filing/Trial Matters

3.5.1 Case Information ............................................................................~

ID~bYNAMIC$

CORPORATION 12/15/2006

3.5.2

3.5.3

3.5.4

3.5.5

3.5.6

Case Party Display .............’

Relief Synopsis ...................................................................................-

Case Time Inquiry

Government Exposure and Special Code .............i

CaseDisposition and Closure

3.5.7 Notes Screen ......: .............................................. .....................’

3.6 Appellate Stage Cases ......................i ...............................~

3.6.1 Case Synopsis

3.6.2 Case Party Display

3.6.3 Relief Synopsis ..................................................................................................................35 -

3.6.4 Case Time Inquiry

3.6.5 Notes Screen ........................."

3.7 Monitored Cases

4.0 Mail Routing/Identification QA Review 38

4.1 Sample. Selection. .....................................................................:

4.2 The Mail Routing Review

4.3 Accuracy Rate Calculation

4.3.1 Mail Routing/Time AcCuracy ................................................:

4.3.2 Mail Identification Accuracy .......................2 .....................................................................44 "

4.3.3 Combined Accuracy ................2

5.0 QA Review Result Submissions 44

5.1 Classification / DBI / Mail Alerts and Flashes

5.2 Correction Verification ....................................~

6.0 Reporting Procedures 47

6.1 Daily Reports ..............., t~q~ ~YNAMICS

1DRC] RESEARCH

~,~ CORPORATION 12/15/2006

6.2 Monthly Reports ..................................................................i ............,

6.3 Quarterly Reports

6.4 Fiscal Year Reports ........................................................................................, Appendix A - Sample Summons and Complaint

Appendix B - Trial/Pre-Filing Classification Field Evaluation Instructions

Appendix C - Appellate Field Evaluation Instructions

Appendix D - Sample Comprehensive Attorney Case Listing (CACL)

Appendix E - DBI CACL QA Checklist

Appendix F - Sample Completed QA Reviews

Appendix G - Sample Monthly Status Report

Figure 2-1, Figure 2-2, Figure 2-3, Figure 2-4, Figure 2-5, Figure 3-1, Figure 3-2, Figure 3-3, Figure 3-4, Figure 3-5, Figure 4-1, Figure 5-1, Sample Case New Case Data Card For a Trial Stage Case .....~

Sample Case New Case Data Card For an Appellate Stage Case

Trial/Pre-filing Case Classification Quality Assurance Review Form

Appellate Case Classification Quality Assurance Review Form

Contractor QC Quality Assurance Review Form ......................................................’.. 12

DBI Quality Assurance Review Form ....~

Trial Case Detail Screen - Stages Tab

Trial Case Detail Screen - Attorneys Tab .............................................:

Appellate Case Detail Screen -Stages Tab .................................................:

Appellate Case Detail Screen - Notes Tab ...................i

Mail Routing/Identification QA Review Form ............................................................,. 41

Correction Verification Letter

~1~ ~)YNAMICS

DRC: RESEARCH

I~-.~i CORPORATION 12/15/2006

1.0 INTRODUCTION

DRC provides quality assurance (QA) services to the Civil Division of the Department of Justice

(DO J) under the terms and conditions of Contract 01-C-0433. These procedures are in response to delivery orders issued by the Office of Management Information (OMI) for QA of the Case

Management System services provided to the OMI by another government contractor.

The Civil Division represents Federal agencies in litigation. All cases handled by the Civil Division are tracked on an automated database called CASES. The Case Management System database

(CASES) contains information on personally and jointly handled cases and monitored active Civil

Division cases. It tracks detailed information on case status from receipt through the trial and appellate stages. Maintained on a Dell NT server operated by OMI, the database assists trial attorneys and paralegals in the Civil Division. The Civil Division will make numerous entries over the life of a typical case and will use this information, not only in tracking Civil litigation information, but also in its planning and budgeting process, in determining attorney case load, among other purposes.

The Case Management Services contractor is responsible for:

Identifying and entering new civil cases involving the government

Entering the mandatory and discretionary data elements for the new cases;

Updating the case records as litigating activities occur; and

Entering case disposition/closure information. ¯

To ensure that the contractor services for OMI meet contractual quality requirements, the Civil

Division instituted a QA program that requires an independent, third party to review contract deliverables and processes on a systematic basis. The QA program assesses the accuracy of the contractor’s Classification, Data Base Integrity (DBI), Mail Routing and Attorney Time Reporting activities in following procedures and performing tasks regarding the gathering and entering of case

IDRC] RESEARCH

B~ CORPORAIION 12/15/2006

information into CASES. This manual documents DRC’s procedures for the quality assurance of the

Classification, DBI and Mail Routing functions.

2.0 CLASSIFICATION QA REVIEWS

When a civil lawsuit involving the federal government is filed in court, the Attorney General and/or appropriate U.S. Attorney are served with documents called a "Summons"’ and "Complaint" (Appendix

A). These documents are forwarded tO the Civil Division for handling. Within one working day of receipt of these documents, the civil division mail unit will send them, along with any other documents that may create a new case, to the classification department. The Classification Unit will then extract pertinent information from the court documents to create a record of the new case in CASES. This new record will become the foundation for tracking the case throughout the entire litigative process.

The information contained in CASES is used by all of the civil division attorneys and managers for tracking case progress, resource management and budgeting. Therefore, it is critically important that the information in CASES be complete and accurate.

When a new case is received, the classifier assigns a unique identification number known as a

Department of Justice (D J) number. Numerous data elements are also determined for original entry into the case record. Certain components of the DJ number and the relevant data elements are determined by extracting information from the summons and complaint. The summons notifies the defendant that he has been sued. The complaint gives notice to the adversary of the nature and basis of the claim asserted. DRC’ QA of classification consists of reviewing the court documents against the

CASES database for accuracy.

2.1 Trial/Prefiling Classification, Appellate Classification and Contractor QC Reviews

There are three types of classification reviews performed by DRC. The first is the Trial/Prefiling Case

Classification review. A DJ number can be created prior to a case being filed in court. This is known as a prefiling stage case. When a case is filed in a trial stage court, it is referred to as a trial stage case.

IDR~C~~)YNAMICS

.RESEARCH

5 12/15/2006

The TrialiPrefiling Case Classification QA Review is a review of a sample of pre-filing and trial stage cases.

The second review type is the Appellate Case Classification Review. This review contains all appellate stage cases. Cases that originate in an appellate co,urt are called appellate stage cases. Since new cases that originate at the appellate stage have numerous unique fields that are not applicable to trial and prefiling stage cases, the two types of cases are reviewed in separate reviews by DRC.

The third review type isthe Contractor QC Review. Several members of the contractor staff provide quality control (QC) on the classifier’s work product. At the direction.of OMI, DRC recently began performing QA reviews of the contractor’s QC work.

All three of these reviews have separate excel workbooks. The excel workbooks are found on the "Q:"

drive and are accessed through Microsoft Excel. Each workbook is mad~ up of worksheets used to complete the individual reviews (see figures 2-3, 2-4 and 2-5). A new Workbook is created each month. At the beginning of the month, the QA reviewer will need to open the Classification, Appellate or Contractor QC workbook template, found on the Q: drive, and use the "save as" function to create a new workbook using the month and year as the file name. The QA reviewer will then complete all of the classification reviews for the month in these workbooks. Each workbook has an "Instructions" tab which provides a detailed guide for using the excel spreadsheets.

2.2 Contractor Processing

Mail for the Civil Division is received by the Mail Identification Unit located in the OMI office at the

~ L Street building. The Mail Identification Unit has one working day to identify and process this mail. Any documents not associated with a case already present in the Case Management System

(CASES) database are forwarded to the Classification Unit.

The Classification Unit will then confirm that the unmatched material sent by Mail Identification is, in fact, not associated with a new case. If not, this unit is responsible for assigning the DJ number and

~I1~ ~)YNAMICS

DRC~ RESEARCH

I~ CORPORAllON 6 12/15/2006 creating the original record in the database, which then becomes the foundation on which the entire case record is built. The Classification Unit must accurately and quickly classify court documents~ including summons and complaints. Delays or inaccurate classifications can impede the litigation process.

After the Classi.fication Unit enters the initial information on a new case, two printed cards are produced which, display the classification entries.

¯ . DOJ branch card

¯ Classification file card (new case data card)

The Classification Unit performs quality control after cases are entered as new. Cases are then returned to the Mail Unit for branch dissemination. Documents received and processed are picked up for delivery three times during the day: 8:30 a.m., 10:50 a.m. and 3:45 p.m.

2.3 Sample Selection

DRC selects a sample of the new classified cases before the Contractor has completed its quality control review (except for Contractor QC Reviews, discussed in section 2.7). The documents are located in an outgoing box in the Contractor Classification room. The QA reviewer retrieves cases from the box prior to the three designated mail pickup times. A sample is taken that is encompassing of all the different classifiers, trial/prefiling and appellate stage cases, as well as, DJ number prefixes.

The sample documents are then copied. Care should be taken to not damage the documents since these are often the original documents for the case. The originals are then returned to the Contractor outgoing cases box so that the mail processing procedure is not interrupted.

2.4 QA Review Procedures

The DRC QA Review consists of four basic, consecutive steps:

IDR~~)YNAMICS

CORPORAllON 7 12/15/2006

Determinin~ and collecting the sample items;

Searching the CASES database to determine that the document is a new case;

Reviewing the sample summons and complaint (Appendix A) against the new case data card

(figure 2-1 and 2-2) and CASES for accuracy;

Recording errors and observations in the Classification excel workbooks

The TrialiPrefiling Case Classification review and Appellate Case Classification review are completed on individual classifiers. Each TrialiPrefiling Case Classification review contains a sample of ten cases from one classifier encompassing a variety of different DJ numbers. Each Appellate Case

Classification review contains a sample of five to ten cases from one classifier.

As each case is reviewed, the QA Reviewer enters pertinent information and errors into an excel spreadsheet that calculates the accuracy rates for the entire sample and the review results i~or each case inspected.

Defensive Trial CIVIL DIVISION NEW CASE DATA C~RD Branch C[assificalion Date: 01tl 5/2005 02t1512005 10:41158

101-17M.4599 ~ v .lYlBI~: US; HUD; ~ Homeowners Assoc ~

First Party; ~ Party Rote: Plaintiff Other Parties: ~ Party Role: Defendant

HUD Party Role: Defendant

Primery Agency: HUD Rcvd Date: 02t’I5!2005 Other Agencies:

Other ID Number:

Branch: CommercJal Litigation Branch District: Florida-Middle

Ctvil Action No: ~ Court: State Court

AG Served Date:

Attorney: ~

Reviewer; ~ Handling: Monitored

Casetype: Suits Under 28 use 2410

Other ID Type:

Govt Role: Defensive

Filing Date:

US Arty Served Date:

No. Plaintiffs:

No, Defendants:

Country:

RELIEF SOUGHT:

Sought by Oppn: No Relief Sought Sought by Govt: Yes

Amount: $ Amount: $1,957 ’

Figure 2-1, Sample case new case data card for a trial stage case

~,-~ CORPORATION 8 12/15/2006

Defensive Appellat~ CIVIL DIVISION NE’W CASE DATA CARD Classification Date: 02J1512005 Immigration

39-37-1569 ~ v Gonzalez, Alberto Atty General

Branch Copy 02115t2005 11:19:25

First Party:~l~l~ Other Parties: Gonzalez, Alberto

Party Role: Ptaint~ Party Role: Defendant

Primary Agency:

Other Agencies:

Oft of Arty Gen

Other ID Number: ~ Branch: Office of Immigration Litigation Reason’. Issues

Asgn Type: Defense of Petition for Review Court: 6th Circuit- Court of Appeals

Docket No: ~ Attorney:

Reviewer:.

Handling:

Casetype:

Immigration Cr[m: No

Rcvd Date’. 0211512005

Other ID Type: Alien Number App Rcvd Date: 021i 5f2005

Govt Part: Respondent Asgn Date: 02/15/2005 Filing Date: 02/11/2005

No. Plaintiffs: 1 No, Defendants: 1

Country: Unknown Countries

Figure 2-2, Sample case new case data card for an appellate stage case

ID~~)YNAMICS

CORPORATION 9 12/15/2006

CASE CLASSIFICATION

Date of Review:

QA Reviewer:.

Number of Cases Reviewed:

Number of Errors Identified:

Caetion Accuracy Total Word: 0 Error Words: 0 Correct Words: 0 Accuracy Rate: #DIVI01

Review # C

Classifier Initials: i

(~or)teF~t ,~ccuracy Total Fields: 0 Error Fields: 0 Correct Fields: 0 Accuracy Rate: #DIVI01

~ombir~ed Accuracy Accuracy Rate: #DIVI01

~)YNAMICS

RESEARCH

CORPORATION

DJ Number

’’ ’:!" I i 0 o o o o o o o o o o o oio o o o o o o o o o o oo o

0 "

0 ’

0 .

Totals

Correclion EX

Analysis of Results and Recommendations:

Figure 2-3, Trial/Pre-filing Case Classification Quality Assurance Review Form

IDRC~ RESEARCH

B~ CORPORATION 10 12/15/2006

APPELLATE CASE CLASSIFICATION

Date of Review:

QA Reviewer:

Number of Cases Reviewed:

Number of Errors Identified:

Classifier Initials:

Review # ~)YNAMICS

RESEARCH

CORPORATION

Caption Accuracy Total Word: 0 Error Words: 0 Correct Words: 0 Accuracy Rate: #DIM/01

DJ Number

"orals

Branch

Content Accuracy Combined Accuracy Total Fields: 0 Accuracy Rate:

Error Fields: 0 Correct Fields: 0 Accuracy Rate: #DIV/01

#DtV/01 i 0 i , o o o o o o 0 o o OlO o o 0

0 . I

0 0 0 0 0 0 00

Analysis of Results and Recommendations:

Figure 2-4, Appellate Case Classification Quality Assurance Review Form

DYNAMICS

CORPORATION 11 12/15/2006

Contractor QC Review

Date of Review:

QA Reviewer:

Number of Cases Reviewed:

Number of Errors Identified:

Qcer Initials:

Caotlon Accuracy Content Accuracy Total Word: 0 Total Fields: 0 Error Words: 0 Error Fieids: 0 Correct Words: 0 Correct Fields: 0 Accuracy Rate: #DIV/01 Accuracy Rate: #DIVI01

Review #’ Q

Combined Accuracy Accuracy Rate: #DIVI01

Totals 0 0 0 0 0 0 0 0 0 0 0 0

I0

= 0

I. I ’ ’ ’ ’ , , 0

0 0 0 0 0 0 0 0 00iO 0 0 00 0 0 0

RESEARCH

Analysis of Results and Recommendations:

Figure 2-5, Contractor QC Quality Assurance Review Form

2.5 Completing the QA Review Forms

The QA Reviewer reviews the fields on the new case data card against the copied court documents following the Classification Field Evaluation Instructions (Appendix B) or the Appellate Field

Evaluation Instructions (Appendix C) depending on the type Of review. The QA Reviewer should also refer to the tools used by the classifiers: the classification chart, classification procedures manual, classification alerts, classification.flashes and the abbreviation]acronym list when completing a review.

Both the Trial/Prefiling Classification review and the Appellate Classification review excel spreadsheets are completed as detailed below:

P!~ ~)YNAMIC5

1DRC] R ES EARCH

~,~ CORPORATION 12 12/15/2006

At the top of the form, enter the Re.view Number. This number is in the format COO 1 for

Trial~re-filing reviews or A001 for Appellate reviews and continues sequentially. It is used to assist in tracking reviews and correction verification.

2. Enter the Date of the QA review, the QA Reviewer’s name and the initials of the Classifier being reviewed.

3. Next, enter the DJ Number listed on the new case data card.

4. Enter the classifier initials listed on the new case data card.

o Count the number of words in the Case Caption and enter it in the column under "Cap Wrds".

The caption, which appears at the top of the new case data card, indicates the litigants of the case.

Review the spelling and abbreviations in the case caption. If there are errors, the Reviewer lists each error as it appears and the recommended correction in the "Error" and "Correction" columns. The QA Reviewer enters the number of words that are incorrect, missing, and extraneous in the column under "Caption" on the same line as the error.

Count the number of fields reviewed and enter it in the column under "No. of Fields".. This number varies from Case to case depending on the numb.er of plaintiffs, defendants, agencies and/or other ID numbers involved in the case. Even when information is not included in a field, it is still counted as part of the total "No. of Fields" figure since the field is still.

reviewed for possible missed information. Since the AG served date has three additional related fields that are not present on the new case data card but are entered in CASES by the classifiers, the AG served date field should always be counted as four fields.. The four fields are: the AG served date, the What was AG Served field, the How was AG Served field and the Mail Registered/Certified Number field. The US Attorney’s office Served date field t~1~ ~YNAMICS

DRCI RESEARCH

1~,~ CORPORA’lION 13 12/15/2006 should always be counted as two fields since the US Attorney’s office number field is also quality assured but is not present on the new case data card.

Enter the DJ number prefix (the numbers before the first hyphen) in the "D J# Pre-fix" column, ffthe DJ number prefix is not one of the numbers listed in the first column of DJ number prefixes on the "Statistics Page", included at the beginning 0fthe excel workbook, a

"0" should be entered in this column.

All of the relevant fields on the new case data card are then inspected for accuracy. To complete the review, the QA reviewer utilizes all of the tools used by the classifiers. These tools include the Contractor’s Case Classification Procedures Manual, the Classification Chart, the Authorized Abbreviations and AcronYm. lists and classification alerts and flashes.

If an error is identified, a "1" is entered in the column pertaining to the error type and a description of the error and correction are provided in the corresponding columns. If multiple errors are found for a single DJ number, it is important to include the Classifier initials on each line containing an error. It is not necessary to include the DJ number on each subsequent error line (see Appendix F for examples). Certain fields can have multiple entries. For example, the other party field can have up to twenty-six parties listed for one case. Each one of these entries is considered a single field. So if there are, for example, two parties listed in

CASES that are incorrect, each would be entered separately on individual lines on the excel spreadsheet. When there are multiple parties, agencies or other ID numbers involved in a case, they may not all be listed on the new case data card. The QA reviewer will. need inspect

CASES to verify that these data elements were in fact, entered in the system and if so, for accuracy.

10. The QA Reviewer searches CASES to ensure that none of the cases in the review are already in the system. This is done by performing a "contains" search on the caption field using the plaintiff’s name. If only one DJ number is returned as the search result, it is considered a new case. If more than one case is returned from the caption search, the other cases are reviewed to determine if the case was already entered in CASES. If so, this is a DJ number error that

~’~1~_ ~YNAMICS

|DRC] RESEARCH

I~ CORPORAllON 14 12/15/2006 should be reported in the QA review. If it is determined that a duplicate case exists in CASES but was entered on ~he system after the case that is being reviewed, this is not an error for the classifier who originally entered the case. When this situation occurs, the QA reviewer should bring this to the attention of OMI so that the duplicate case can be removed from

CASES.

11. When an error is identified, the documentation for the case should be designated as an exhibit on the new case data card and the excel spreadsheet. The error should be noted on the new case data card and the section of the attached documentation for the case that confirmed the error should be highlighted.

2.6 Accuracy Rate-Calculation

OMI has established three different accuracy categories: Caption Keying, Content, and Combined.

All of these accuracy rates are calculated automatically by the excel, spreadsheet. A description of how the accuracy rates are calculated is provided below.

2.6.1 Caption Keying Accuracy

The case caption keying accuracy is calculated using the following formula:

(Total Caption Words - Total Caption Word Errors) / Total Caption Words x 100 --

Caption Keying Accuracy rate percentage

2.6.2 Content Accuracy

The content accuracy is calculated using the following formula (note: the "Total no. of Fields" and"Total Field Errors" figures do not include the caption field):

rl~ ~YNAMICS

DRC~- RESEARCH

I~.~i CORPORAllON 15 12/15/2006

(Total No. of Fields - Total Field Errors) / Total No. of Fields x 100 = Content Accuracy rate percentage

2.6.3 Combined Accuracy

The combined accuracy is calculated using the following formula:

(Total Caption Words - Total Caption Word Errors) + (Total No. of Fields - Total Field

Errors) / (Total Caption Words + Total No. of Fields) x 100 = Combined Accuracy rate percentage

2.7 Contractor QC Reviews

The Contractor QC Review is completed on individual QC Reviewers. Each review contains a sample often cases. Documents that have been QC’d have a yellow form entitled "JSC Case Classification

Quality Control Review" form attached to them. This form will provide the name of the classifier as well as the name of the QC reviewer. Documents that have been QC’d are found in two places. Some are found in the outgoing box in the Contractor Classification room mixed in with other non QC’d cases. Others called "Specials" are found in office #~. "Specials’.’ are documents that require special attention and are separated from the other new cases. Both types of QC’d cases are pulled for the sample.

The Contractor QC review has its own excel workbook and is completed the same as the

Trial~refiling Case Classification review and Appellate Case Classification Review. The only exception is that there is a column to enter the classifiers initials found at the far right of the Contractor

QC Review form.

DYNAMICS

CORPORA[ION 16 12/15/2006

3.0 DBI QA REVIEW

The DBI program, was designed to supplement the Division’s efforts to ensure that all data.elements are correctly entered, that all records in the database are unique, and that the database is current, consistent, and accurate. As the lawsuits progress, the DBI unit makes sure that CASES is updated with the current information for the case. They do this by reviewing the attorney’s Comprehensive

Attorney Case Listing (CACL), visiting with the attorneys (called "attorney interviews") and asking them specific questions about the case and its progress. The CACL, Appendix D, is the primary tool used by the DBI analysts to complete DBI reviews. The CACL is a printout, derived from CASES, containing information on all open cases assigned to an attorney.

Each DBI analyst is assigned a group of DOJ attorneys and meets with each individual attorney approximately once every three months (bi-annually for immigration attorneys). The QA reviewer attends or reviews various interviews to quality assure that the right questions were asked and that the information obtained at the interview was updated accurately in CASES.

3.1 The DBI QA Review

The Quality Assurance contractor performs two types of DBI QA reviews: the DBI file.review and the

DBI Interview review.

In the DBI file review, the CACL (pulled from the DBI files and copied) is analyzed to determine that all updates to CASES, as a result of information gathered at interviews by the DBI analyst, have been entered into CASES and that all disposition forms completed during the interview are found in the files. To assist in the analysis of the CACL, the QA reviewer should use all of the tools available such as: The DBI Procedures Manual, the DBI QA Checklist (Appendix E), DBI alerts and flashes, PACER, etc. The QA Reviewer does not attend the attorney interview in a DBI file review. The determination of errors is mostly based on notes written on the CACL by the DBI analyst.

IDR~~)YNAMICS

CORPORArION 17 12/15/2006

In the DBI Interview review, the QA Reviewer accompanies the DBI analysts on their attorney interviews to audit the information gathered and the pertinent questions asked. When accompanying the DBI analyst on attorney interviews, the QA Reviewer brings the CACL which has already been analyzed for apparent anomalies. To print out a CACL, the QA reviewer will click on the "Special

Reports" icon in the navigation bar in CASES, click on the folder entitled "CACL Reports" and then select "Trial CACL" or "Appellate CACL," in the file tree on the left side of the screen, depending on the attorney being interviewed. Next, the QA reviewer will select the attorney being interviewed by double clicking on the expression column and searching for the attorney!s name. To print out the

CACL, click on the print icon to the right of the report. During the interview, the QA Reviewer takes detailed notes of any changes to the. status of the attorney’s cases that have occurred since the previous interview. Any issues or data elements that the QA Reviewer thinks were overlooked and should have been addressed in the interview are noted on the CACL and later identified on the QA review forms.

The QA Reviewer does not take part in the discussion at any time and is merely present to observe.

3.2 Reviewing CASES for Updates After the Interview

The DBI Quality Assurance review is performed no sooner than five working days after the interview to allow for the entry of all updates gathered during the interview. The QA reviewer should then confirm that all of the updates were entered on CASES. To check for updates, the QA Reviewer logs onto CASES and searches each DJ Number printed on the CACL. Each case is reviewed to ensure that the updates have been entered by the DBI analyst. The QA Reviewer also verifies that any anomalies overlooked by the DBI analyst during the interview are still present on the system. If an error or anomaly is found, the QA reviewer prints all the screens containing errors for inclusion with the QA review forms and highlights the data elements that are in error on the printouts. Additionally, the QA

Reviewer writes directly on the printout explanations of the errors and/or anomalies and the necessary corrections. All the printouts for one review are grouped in a logical, sequential order and marked alphabetically as exhibi.ts.

3.3 Completing the QA Review Forms

~"I1~ ~)YNAMICS

IDRC] RESEARCH

B~ CORPORATION 18 12/15/2006

The QA review forms used to complete the DBI reviews are found in excel workbooks on the "Q:"

drive and are accessed through Microsoft Excel. Each workbook is made up of worksheets used to complete the individual reviews. A new workbook is created each month. At the beginning of the month, the QA reviewer will need to open the DBI workbook t~mplate, found on the Q: drive, and use the "save. as" function to create a new workbook using the month and year as the file name. The QA reviewer will then complete all of the DBI reviews (the DBI file review and the DBI interview review) for the month in this workbook. Each workbook has an "Instructions" tab which provides a detailed guide for using the excel spreadsheets. A sample DBI Quality Assurance Review form is found in figure 3-1.

To complete the DBI QA review form, the QA Reviewer fills out all the necessary identifying information that is requested at the top of the DBI QA Review forms, The QA Reviewer enters the following identifying information: Review Number (in the format D00! and continuing sequentially), Type of Review (interview or file), Interview Date, Attorney interviewed, Civil Division Branch

(Comm, FP, Torts, Imm or App), DBI Analyst initials, Date of QA review and QA Reviewer. The QA

Reviewer counts and enters the number of cases in the CACL and the number of cases updated.

Before the individual cases are listed on the QA review form, the QA Reviewer should enter a "1" in the first row under the DBI rids column and either, "CACL=OK’’ or "CACL" in the field column, depending on whether a CACL error was made by the DBI analyst. A CACL error can occur one of two ways. First, the DBI analyst is required to ask the attorneys at the interviews if there are any new cases not listed on the CACL. If the DBI analyst fails to ask the attorney this question, it is considered a CACL error. Second, the CACL should not be printed out by the DBI analyst prior tofive business days of the interview. If so, it is considered a CACL error.

Next, the QA reviewer enters the DJ number of the first case in the CACL in the DJ number column.

The "DBI Flds" and "Other Flds" columns are determined by the number of fields the DBI analyst was responsible for reviewing. For Trial/Prefiling cases, the number of DBI fields is always seventeen and the number of other fields is always nine. For Appellate stage cases, the number of DBI fields is always fourteen and the number of other fields is always eight. Since the DBI analyst is only t~.~ ~)YNAMiCSIDRC~ RESEARCH

B~ CORPORATION 19 12/15/2006

DATABASE INTEGRITY

Date of Review:

QA Reviewer:

Number of Cases in CACL:

Number of Cases Updated:

DB! Field Errors;

[AT] Attomsy/Rev.

[BR] Branch

[CACL] CACL

[CI] Case Ignored [CS] Case Type [CA] Civil Action / Dockst No, [Cl.] Closure [DP] Disposition [FD] Filing Date

DBI Fields Accuracy Rate:

Total Fleldm Errors Fields;

Correct Fields:

Accuracy Rate:

DBI Other DJ Number Flds Flds o o o

#DIV/0i

Review # D

Type of Review:

Date of Interview:

Attorney Interviewed:

Branch:

DBt Analyst Initials:

[GE] Govt. Exposure [HNDL] Handling [NS] Note Screen ’ [$] Relief [SE] Section [SC] Special Codes [ST] Stage [TR] Time Reported [AC] Action Code [RE] Reason

Other Field Errors:

[CAP] Caption [CC] Court Code [DS] Distdct [DJ] DJ Number

Other Fields Accuracy Rate:

Total Fields; 0 Errors Fields: 0 Correct Fields; 0 Accuracy Rate; #DIV/01

0 [GR] Govt, Role/Part.

0 [OA] Other Agency 0 [O1] Other ID No.

0 [OP] Other Party

[PAl Primary Agency

Combined Accuracy Rate:

Accuracy Rate: #DIVIOI leacJ RESEARCH

Field Description of ErrodDlscrepancy Recommended Correction EX

Totals

Analysis of Results and Recommendations:

Figure 3-1, DBI Quality Assurance Review Form responsible for asking if monitored cases are still pending, monitored cases only require that a 1 be entered in the DBI fields column.

The DBI review separates the fields to be quality assured into two types: DBI Fields and Other Fields.

DBI fields are fields that are often directly updated or need to initially be verified by the DBI analysts.

t~1~_ ~YNAMICS l~i CORPORATION 20 12/15/2006

Other fields are fields that are most often entered by the classifiers. The DBI analyst should still check the other fields for accuracy. Above the accuracy rate calculations on the QA review form is a list of all the DBI and other field error types. The abbreviations, found in parenthesis, for these fields are to be used in the "Field" column. The "Field" column indicates the field that was questioned or updated by the DBI analyst and if there was an error in the field. For example, if the DBI analyst questioned the attorney about the filing date, the attorney provided the filing date and the DBI analyst correctly entered it in CASES, "FD=OK’’ would be entered in the field column. If the DBI analyst failed to

¯ enter the filing date, entered it incorrectly or failed to ask the attorney for a missing filing date, the field column would be entered as "FD" and the Description of Error/Discrepancy and Recommended correction columns would be completed. The QA reviewer prints all the screens from .CASES containing errors for inclusion with the QA review forms, highlights the data elements that are in error on the printouts and writes the exhibit letter (A, B, C ....) on the printout. The letter of the exhibit used to document the error is written in the "EX" column of the review form in the row corresponding to the error. This process is completed for each field updatedor questioned by the DBI analyst for a case.

The DJ number and number of fields should not be reentered for each field listed for a single case. If there are no errors for a DJ number and the DBI analyst did not question the attorney about any particular field, the QA Reviewer marks an "OK" in the "Field" column. The QA Reviewer repeats the above process for all the DJ numbers reviewed in the CACL.

3.4 Accuracy Rate Calculation

OMI has established three different accuracy categories: DBI Fields, Other Fields and Combined. The excel spreadsheet automatically calculat.es the DBI field accuracy rate, the other field accuracy rate and the combined accuracy rate. A description of how the accuracy rates are calculated is provided below.

3.4.1 DBI fields Accuracy

The DBI fields accuracy is calculated using the following formula:

DYNAMICS

CORPORAtiON 21 12/15/2006

(Total DBI Fields - Total DBI Field Errors) / Total DBI Fields x 100 -- DBI Fields

Accuracy Rate percentage

3.4.2 Other Fields Accuracy

The other fields accuracy is calculated using the following:

(Total Other Fields - Total Other Fields Errors) / Total Other Fields x 100 = Other Fields

Accuracy Rate percentage

3.4.3 Combined Accuracy

The combined accuracy is calculated using the following formula:

(Total fields - Total field Errors) / Total fields x 100 = Combined Accuracy rate

. percentage

3.5 Pre-Filing/Trial Matters

3.5.1 Case Information

Figure 3-2 shows an example of a case printout of DJ number 145-16-5861. After logging onto

CASES clicking onthe "C" icon in the navigation bar at the top of the CASES screen to display the

CASES filter and entering the DJ number, the case detail screen appears. This first screen gives general information about the case such as the DJ number, the caption, the amount of money sought by the government and/or the opposition, the amount of money awarded to date, the government exposure, the current primary attorney and reviewer, and the disposition status of the case (whether it is open or closed). The QA reviewer analyses all of this information during the review process

~’~1~ ~YNAMICS

1DRC] R ES EARCH

~ CORPORATION 22 12/15/2006

Figure 3-2, Case Detail Screen - Stages Tab using all of the tools available such as: the DBI Procedures Manual, the Classification Procedures

Manual, the Appellate Case Processing Procedures Manual, the DBI QA Checklist (Appendix E), DBI alerts and flashes, PACER, Abbreviation and Acronym lists, disposition forms, etc.. The QA

Reviewer first verifies that none of the words in the caption have been misspelled and that all of the appropriate DOJ classification abbreviations and acronyms have been used. The caption is originally entered in the system by the classification unit and should reflect the original court caption for the case as closely as possible. The QA Reviewer then makes sure that the current attorney listed in CASES matches the attorney name on the CACL being reviewed. If the name does not match, the DBI analyst should ask the attorney at the interview whether he or she is the

~’!1~ ~YNAMICS

RESEARCH

CORPORATION 23 12/15/2006

lead attorney on the case.

Because the first screen in CASES is a general screen, the QA Reviewer needs to gain access to more in depth information about the case. The bottom section of the case screen is divided into tabs. These tabs can be clicked on with the mouse to gain access to different case information. The first tab on the left side of the case screen is the "Stages" tab. In Figure 3-2, the case stage shown is

Pre-filing/Trial stage. The QA Reviewer double clicks on the highlighted section, and the screen in

Figure 3-3 appears..

If the case has been verified as filed, there will be a "Yes" in the "Has this case been filed?" field.

The QA Reviewer verifies that the "Filing Date" field has a date input into this field. If the case has not been verified as filed, the "Filing Date" field will be blank.

Next, the QA Reviewer verifies that the "Government Role" field corresponds to the case caption.

For instance, if the government is taking a defensive stance in the case, the "Government Role" will be marked as "D" and the government agency or agent for the government will appear on the right of the "v" (the abbreviation for versus) in the case caption. Exception: The government’s role in bankruptcy cases may be affirmative even though they are listed in the caption as a defendant.

The QA Reviewer then verifies that the "’DJ number" and "Branch" correctly correspond to one another and to the numbering system established by the DOJ Classification chalk. For instance, in

Federal Programs Branch cases, the DJ number often begins with the number "145" or "35."

3.5.2 Case Party Display

When the QA Reviewer clicks on the "Parties" tab of the synopsis, the screen displays up to 26 parties involved in a case. The first field is the "Name." The CASES screen displays the name of the party involved in the suit. The second field is the ’,Role" field which shows the role of each party in a case. The five roles that may appear in this field are: Plaintiff"P," Defendant "D,"

Amicus "A," Intervenor "I," and Unknown Defendants "U." The third field, entitled "Role Desc,"

DYNAMICS

CORPORAJ~ON 24 12/15/2006

displays the description of the role of the party. For example, the description of "D" is Defendant.

The fourth field, entitled "Start Date" is the date the party became active in the case. And the fifth

Figure 3-3, Case Detail Screen - Attorneys Tab field, "End Date," is the date that the case or party became inactive within the scope of the Civil

Division. The last field is "Secured Party." All cases in the pre-filing stages will have a "Yes" in this field, and most viewers of CASES will not have access to view the parties involved in this litigation. If the case has proceeded to trial, then a filing date is requested and; if the filing date is available, it is entered into CASES. The "Secured Party" field is changed to "No," meaning that the

~’!1~_ ~)YNAMICS ’

I~-~li CO RPORAT,O N 25 12/15/2006 parties wiI1 be visible to all users on CASES. The QA Reviewer checks the case party display for typos, the use of proper abbreviations for party name, and correctness of the role.

3.5.3 Relief

The relief involved in a case can be seen on the synopsis screen in CASES. The relief section displays the kind of relief that has been sought by and awarded to the government and the opposing parties in the case. There are four segments to the relief synopsis: the "Govt - $ Sought," the "Oppn

- $ Sought," "Govt - Awarded to Date," and the "Oppn - Awarded to Date."

Monetary Relief Sought by the Opposition is displayed in CASES to the right of the words "Oppn -

$ Sought" (see figure 3-2). There may be one of four letters entered: "U," "Y," "E," or "N". In

Figure 3-2, there is an "E" entered for the monetary relief sought by the opposi.tion. The "E," for equitable only, indicates that only equitable relief is being sought in the case. A "U" is used when no one has yet ascertained the amount 0fmoney that is being sought in a case. A "Y" signifies that there is a specific amount being sought, and a dollar amount should be entered next to "Amt." An

"N". signifies that there is no monetary relief being sought.

Tlie "Govt - $ Sought" field is set up in a similar fashion to the "Oppn - $ Sought" field, except that all the information pertains to the relief sought by the government. The QA Reviewer will verify the relief information in both segments to see that it corresponds with the DJ number and the government role for the particular case.

If no monetary relief has yet been awarded, the space next to the phrase "$ Awarded to Date:"

should be left blank. If monetary relief has been awarded, an amount should be entered next to the

"$ Awarded to Date:"

If the amount of monetary relief sought by the opposition has been left unspecified, the QA

Reviewer notes on the CACL that the amount being sought by the opposition has been left

1~1~ CORPORATION 26 12/15/2006

unspecified and that the DBI analyst should ask the attorney in the interview whether or not the opposition is seeking a specific amount.

Cases with the following prefixes in their DJ numbers should often have a record of the specific amount of money being sought:

46 - Fraud Against the Government

77 - Claims in Favor of the U.S.

105 - Reconstruction Finance Corporation

106 - Agricultural Marketing Act

120 - Commodity Credit Corporation

130 - Federal Housing Act

136 -Farmer’s Home Administration

151 - World War II Veteran’s Matters

In addition, most Torts cases (DJ # 157...) have an exact amount for monetary relief sought.

Attorneys working on affirmative Commercial cases who have recorded attorney time on discovery or settlement related activities should be asked in interviews if the government is seeking a specific amount. Specific amounts should also be requested from attorneys .working on defensive Torts cases who have recorded attorney time on defensive discovery and settlement activities. If relief has been awarded, the QA reviewer examines the relief detail information inCASES and confirms that all of the information entered is correct and appropriate for the case.

3.5.4 Case Time Inquiry

CASES maintains a record of all the time reported by the government attorneys, reviewers and paralegals working on each case. The time reporting function can be accessed by clicking on the clock icon in the navigation bar at the top of the CASES screen. All of the time on the system is entered according to DOJ assigned activity codes. Each activity code is to be used in a specific

IDR~~YNAM ICS

CORPORATION 27 12/15/2006

situation in the progression of a case. The activity codes specifically used for pre-filing/trial stage case~ are as follows:

01 ’- Grand Jury

02- Other Pre-Filing Acti;cities

03 - Claim/Settlement Approvals

05- Pleadings & Motions

07 - Government’s Discovery

10 -Opposition’s Discovery

11 - Trial/Evidentiary-H-earirtgs-

12 - Post-Trial Activities

The QA Reviewer examines the time activities entered for each case on the CACL being reviewed.

If there are time activities entered on the system that do not correspond to the stage of the case or conflict with other information on the system, tile QA Reviewer looks for a note entered in CASES explaining the use of the conflicting activity code. For example, .if an attorney is reporting "Court of Appeals" time to a case that does not have an appellate stage, there should either be a note explaining the anomaly or the DBI analyst should ask the attorney at the interview if this time entry is correct.

If a case opened at the pre-filing stage is an administrative claim being heardby a U.S. Government

Board, it may have trial stage codes recorded to it. However, if a case is to have activity recorded to it that is unusual for the stage of litigation at which it has arrived, then a note should be found in

CASES explaining the anomaly.

The QA Reviewer goes through all the screens of time entered in CASES to make sure that the correct time codes are recorded to the case. The "last time reported" field in CASES and on the

CACL shows the last time that was reported to the case. The QA Reviewer compares this last time reported date to the received date for the case to assess the amount of activity occurring in the case.

COR~ORAnON 28 12/15/2006

If a case has had no time reported for an extended period of time, the DBI analyst should specifically ask the attorney why the case has been inactive. Any information provided by the attorney should then be included in a note in CASES.

Reviewing the time reported and activity codes for each case is an important step in identifying changes in the status of a case and to identify incorrect information in CASES. For example, if the attorneys have begun reporting pleadings and motions time for a pre-filing stage case, this indicates that either the case has been. filed and hence the analyst should ask the attorney for the filing date, civil action number and relief sought amount for the case or that there is a potential attorney time reporting error.

3.5.5 Government Exposure and Special Codes

The "Government Exposure" field in CASES records the amount of risk that the government believes the case represents. If the opposition is seeking monetary relief, the DBI analyst should ask the attorney the amount of government risk. This amount should be entered into CASES and is reviewed by the QA reviewer.

The DBI analysts are responsible for updating the special codes field in the CASES database.

Special code information for each case can be seen by clicking on the "special code" tab on the case synopsis screen in CASES. All special codes are verified by the QA Reviewer during the’quality assurance review. The Special codes relating to monetary relief include 504, 505, 515, and 516.

Special code 504 (affirmative) and special code 505 (defensive) are the codes used to indicate relief sought greater than or equal to $5 million is accurate and has been verified by the attorney. Special code 515 is entered into CASES if the amount sought is exaggerated, but the attorney is not willing to provide a government risk amount. A note must be entered stating the name of the attorney and the date the attorney did not provide the risk information. Special code 516 is entered into CASES if the amount sought is exaggerated, and the attorney is willing to provide an estimate of the government risk. The estimated risk amount is entered into the "Government Exposure" field in

CASES. A note is also entered stating the name of the attorney providing the risk information, the l~ CORPORAll~]N 29 12/15/2006 date the information was obtained, and the risk amount provided.

necessary for monitored cases.

Note: this procedure is not

Some sPecial codes have expiration dates. When the expiration date expires and the situation warranting the special code is still relevant, the DBI analyst should ask the attorney about the information pertaining to the special code. If the special code is still applicable, it should be reentered into CASES to reset the expiration date. For example, special code 517 is used when all of the stages for a case are disposed and the attorney wishes to keep the case open. If special code

517 was entered for a case and the expiration date expired prior to the interview, the DBI analyst should ask the attorney if the case should still remain open.. If so, the code should be reentered into

CASES to reset the expiration date. For a list of all of the relevant special ’codes see the contractor’s

DBI Procedures Manual.

3.5.6 Case Disposition and Closure

One of the primary duties of the DBI analyst is to make sure that the CASES database reflects the true disposition of the cases. At the interview, as each case is discussed, the attorney notifies the

DBI analyst if a case has been disposed and if it is still pending or if it can be closed. If the case has been disposed, the DBI analyst will complete a disposition form and have the attorney sign it. If the case is a fraud case, is in a national court or if the disposition involves monetary relief the DBI analyst, must also obtain the reviewer’s signature on the disposition form. The typeof disposition and the date disposed along with closure and relief information are completed on the disposition form. The QA Reviewer notes these dates and disposition types and writes them on his.or her Copy of the CACL during the interview.