Attachment_9_Media_Processing_Pipeline_and_Recommind.pdf

SEC Procedures for Data Staging, Loading and Migration into

Recommind

SEC Headquarters 100 F Street, NE

Washington, DC 20549 Office of Enforcement

Modified: 3/17/2015

SEC Procedures for Staging, Loading and Migration

DRAFT ii

1 REVISION HISTORY

Date Revised Version Revised by Reason For Revision

3/17/2015 3.0 Kathy Stewart Sanitized for attaching to the SOW.

NOTE – All sensitive graphics or contents have been removed from this SOP.

SEC Procedures for Staging, Loading and Migration Table of Contents

DRAFT iii

Table of Contents

1 Introduction

1.1 File Folder Structure

1.1.1 NAS-Hosted File Share and Server

1.1.2 Partition/Share

1.1.3 Database

1.1.4 Case Number

1.1.5 Database Name

1.1.6 Media Type

1.1.7 Volume ID

1.2 Applications Used

1.3 Media Processing Pipeline

1.3.1 Receive Submissions of Media

1.3.2 Copy and Log Submission

1.3.3 Load Imaged Data

1.3.4 Quality Assurance

1.3.5 Disposition of Media

2 Receiving Media

2.1 Dating Submissions

2.2 Reconciling the Item

3 Copying and Logging Submissions

3.1 Setting up the File Folder Structure

3.2 Copying the New Data

3.2.1 Copying Audio Files

3.3 Verifying the Copy Process

3.4 Logging the New Data

4 Loading Data

4.1 Loading Imaged Data

4.1.1 Before Loading

4.1.2 Loading the Data

4.1.2.1 Analyze the Data

4.1.2.2 Create the Database

4.1.2.3 Prepare for Export

4.1.2.3.1 Check Dates

4.1.2.3.2 Add Fields

SEC Procedures for Staging, Loading and Migration Table of Contents

DRAFT iv

4.1.2.3.3 Complete Paths to Native Files

4.1.2.3.4 Create Hyperlinks

4.1.2.3.5 Perform a Hyperlink Check

4.1.2.3.6 Import OCR Text

4.1.2.3.7 Perform a Field Comparison

4.1.3 Export the Database to the Live Location

4.1.4 Run a Duplicate Detection Check

4.1.5 Load the Images

4.1.6 Isolate the Load

4.1.7 Perform Quality Control Checks on the Live Database

4.1.7.1 Check Dates

4.1.7.2 Perform a Hyperlink Check

4.1.7.3 Perform an OCR Check

4.1.7.3.1 Perform a CPL OCR Check

4.1.7.3.2 Perform a Search Check

4.1.7.3.3 Check the First and Last Images

4.1.7.4 Perform an Image Link and Path Check

4.1.7.5 Perform a Concatenation Check

4.1.7.6 Check Menu Lists

4.1.7.7 Check Security Settings

4.1.8 Close Out

4.1.8.1 Close Out

4.1.8.2 Close out

5 Data Staging, Loading and Migration to Recommind

5.1 Completing Recommind Case Questionnaire

5.2 Staging Recommind Requests

5.2.1 Processing ESI Formatted Data

5.2.2 Processing CSV Formatted Data

5.2.2.1 Receiving and Copying the CSV Data

5.2.2.2 Modifying the CSV Data

5.2.2.2.1 NATIVELINK

5.2.2.2.2 TEXTLINK

5.2.2.2.3 Images

5.2.2.3 Verifying the Data

5.2.2.4 Exporting OCR

5.2.2.5 Exporting Concordance Database to .DAT File

5.2.2.6 Tracking the Information

5.2.2.7 Notifying the Loading Team

5.3 Loading CSV Data to Recommind

5.3.1 Validating the Data

5.3.2 Confirming the Document Data

5.3.3 Normalizing Headers

5.3.4 Ingesting and Loading the Data

5.3.5 Validating the Data Load

Media Processing Pipeline Table of Contents

DRAFT v

5.3.6 Adding Documents to Matter

5.3.7 Publishing Documents

5.3.8 Validating the Publish Process

5.3.9 Exporting CSV Data

5.3.10 Merging CSV Data into Ingestion or Review

5.3.11 Migrating Legacy Data

Appendix A: Pre-processing Quality Assurance

CSV

Migration

ESI

Exceptions Form

Appendix B: Processing QA

Processing Tracking Form

ESI Ingestion

Converting Concordance Data

CSV Data Load

Appendix C: Post-Publishing QA

Appendix D: Creating a New Security Group in Recommind

Create a New Security Group

Starting the Engine

Appendix E: Performing a Bulk Conversion

Logging In

Selecting Documents Selecting Documents Individually Selecting All Documents in the Document List Selecting the Documents in a Tag Category Selecting All the Documents in the Database Selecting Unconverted Documents Only Selecting Only Documents with No Annotations

Performing a Bulk Conversion of Selected Documents

Index

SEC Procedures for Staging, Loading and Migration List of Figures

DRAFT vi

List of Figures

Figure 1 Sample Copy To File Folder Structure Figure 2 Media Processing Pipeline Figure 3 Emailed Request Figure 4 Copying and Logging Process Figure 5 Copy_Check Folder Figure 6 Copy Form Figure 7 Sample Email Figure 8 Loading Queue Figure 9 Load Search Form Figure 10 Load Form Figure 11 Special Instructions Figure 12 Data Properties Dialog Box Figure 13 Unedited Opt File Figure 14 Unedited Opt with \IMAGES\ selected Figure 15 Edited Opt File Figure 16 Count the Y Breaks Figure 17 File Format and Encoding Figure 18 Date Format Figure 19 Header Row (Bates and Imagekey) Figure 20 Extra Blank Line Figure 21 SAL Preload Analysis: Step 1 User Input Screen Figure 22 SAL Preload Analysis Report: Top of Screen Figure 23 SAL Preload Analysis Report: Middle of Screen Figure 24 SAL Preload Analysis Report: Bottom of Screen Figure 25 Date Check CPL Figure 26 Date Check Results Figure 27 Modify Screen Figure 28 Natives 1 Figure 29 Natives 2 Figure 30 Global Replace Screen Figure 31 Full Path Figure 32 Create Hyperlinks Figure 33 Create Hyperlinks: Step 2 Figure 34 Hyperlink Created Figure 35 Hyperlink Check Figure 36 Hyperlink Check Results Figure 37 Readocr CPL Figure 38 Read OCR Utility Figure 39 OCR Import Menu Figure 40 Field Check: Step 1 Figure 41 Database Field Comparison Figure 42 Sort by Panel

Media Processing Pipeline List of Figures

DRAFT vii

Figure 43 Export Wizard – Database Figure 44 Export Wizard – Fields Figure 45 Append/Replace Figure 46 Export Wizard – Export Figure 47 Reindex Figure 48 Duplicate Check Figure 49 Imagebase Management Dialog Box Figure 50 Register Load Tab Figure 51 CPL Date Check Figure 52 Date Check Results Figure 53 Hyperlink Check Figure 54 Hyperlink Check Results Figure 55 OCR Check Figure 56 OCR Check Screen Figure 57 OCR Check Results Figure 58 Image vs. OCR Text Comparison Figure 59 Mpg - Image Link and Path Check CPL Figure 60 CPL Image Link And Path Check Instructions Figure 61 Image Check Complete Figure 62 Security Settings: Live on Left; Copy To on Right Figure 63 Close Out BDMT Figure 64 Load Form Figure 65 Workflow Overview Figure 66 Copy_Check Folder Figure 67 Recommind Axcelerate Workflow Overview Figure 68 Concordance File Figure 69 CSV Header Normalizer Figure 70 Enter Delimiter Message Figure 71 CSV Header Normalizer Browse Figure 72 CSV Header Normalizer Browse 2 Figure 73 CSV Header Normalizer Mappings Figure 74 Normalization Results Figure 75 Issue Example Field Headers Not Mapped Correctly Figure 76 CSV Header Normalizer: Settings Figure 77 Field Mappings/Headers From File Figure 78 List of Available Fields Figure 79 New Field Mappings Figure 80 Positive Results for Headers Figure 81 Image Error Figure 82 Field Mappings Screen Figure 83 Add New Field Figure 84 Match Result Screen for Headers Figure 85 Recommind Application Manager Login Screen Figure 86 Application Manager Screen Figure 87 CORE Administration Screen Figure 88 CORE Administration Screen Workplace Tab

SEC Procedures for Staging, Loading and Migration List of Figures

DRAFT viii

Figure 89 New Data Source Wizard (Screen 1) Figure 90 Define Data Source Template Figure 91 Data Source Details Screen Figure 92 Summary Screen Figure 93 ENF Workspace (2) Figure 94 Options List for Data Source Figure 95 Configuration Screen Figure 96 URI for DAT file Figure 97 General Settings Figure 98 Field Mappings Figure 99 Start Data Source Screen Figure 100 Data Sources Tab to Check Status Figure 101 Matters Tab for Validating Investigation Figure 102 Explore Tab for Validating Document Counts Figure 103 Explore Tab for Validating Received Production Figure 104 Explore Tab Validating Data Figure 105 New Matter Wizard: Define Name and Description Figure 106 Check the Result and Save Screen Figure 107 Add Saved Search Wizard: Select Matter Figure 108 Add Saved Search Wizard: Define Search Figure 109 Add Saved Search Wizard: Define options Figure 110 Add Saved Search Wizard: Check the result and save Figure 111 Publish to Review Wizard: Publish mode selection Figure 112 Publish to Review Wizard: Publish Options Figure 113 Review & Analysis: Jobs tab Figure 114 Export Wizard: Define Output Format Figure 115 Export Wizard: Define Name and Description Figure 116 Export Wizard: Select fields to include Figure 117 Export Wizard: Field Export Order Figure 118 Export Wizard: Define Generic Output Options Figure 119 Export Wizard: Define Advanced Output Options Figure 120 Export Wizard: Summary Figure 121 Ingestion Jobs Tab Figure 122 CSV Merge Wizard: Load File Definition Figure 123 CSV Merge Wizard: Parse Preview Figure 124 CSV Merge Wizard Figure 125 Merge Wizard: Define Merge Type Figure 126 CSV Merge Wizard: Text Mapping Mode Figure 127 CSV Merge Wizard: Field Definition Figure 128 CSV Merge Wizard: Additional Options Figure 129 CSV Merge Wizard: Persistency Options Figure 130 CSV Merge Wizard: Summary Page Figure 131 File Menu with Modify Selected Figure 132 Modify Screen Figure 133 Modify Screen with TAGS and Paragraph Added Figure 134 Global Replace Screen

Media Processing Pipeline List of Figures

DRAFT ix

Figure 135 Concordance Image: Imagebase Management Figure 136 Imagebase Management: Directory Tab Figure 137 Begin Program Figure 138 Recommind Scripts\Export CPL Figure 139 Pre-Data Load Validation Checklist for CSV Figure 140 Pre-Data Load Validation Checklist for Migration Figure 141Security Group Figure 142 Create Group Screen Figure 143 ENF Applications Workspace Figure 144 Create Workspace or Select Workspace to Open Screen Figure 145 Default Workspace Figure 146 Assign Application Screen Figure 147 Change Display Name Figure 148 Jobs Tab Figure 149 Monitor Tab/Data Sources Figure 150 Prior to "Use HTTPS" Figure 151 Use HTTPS Figure 152 Kill the Selected Process Figure 153 R&A Application: Selection List Figure 154 Smart Filters Figure 155 Select or Input Conversion Screen Figure 156 Select or Input Redaction Figure 157 Actions Dropdown List Figure 158 Bulk Conversion Wizard: Production Check Figure 159 Bulk Conversion Wizard: Conversion Check Figure 160 Bulk Conversion Wizard: Redaction Check Figure 161 Bulk Conversion Wizard: Define Name and Description of Conversion Job Figure 162 Bulk Conversion Wizard: Check the Result and Save

SEC Procedures for Staging, Loading and Migration List of Figures

DRAFT x

List of Tables

Table 1: Exceptions Form Table 2 Processing Tracking Form Table 3 Field List Table 4 Post-Processing Validation Form

SEC Procedures for Staging, Loading and Migration Introduction

DRAFT 11

2 Introduction The Securities and Exchange Commission (SEC) Media Processing Pipeline processes electronic data for ongoing Enforcement Division investigations and litigation, in which electronic evidentiary materials (imaged documents) are loaded into a database. Staff working on a specific case have access to the native and imaged documents associated with that case through the use of an application called Concordance (which will be replaced over time by Recommind Axcelerate). This application provides search and retrieval capabilities.

The objectives behind the implementation of this system are:

• To remove all boxes containing evidentiary materials from Enforcement hallways, offices, and storage rooms nationwide.

• To provide Enforcement staff with an efficient and effective electronic search and review tool.

• To provide a backup of data and a prompt disaster recovery solution.

The specific objective of the Media Processing Pipeline system is to load electronic files carrying evidentiary information for all cases currently in the Discovery or Litigation stage handled by the SEC Headquarters or any of the Regional Offices, into a comprehensible database where any attorney or qualified staff member can access the necessary information to perform their duties.

This document describes the different stages of the Media Processing Pipeline. The process has a five-day turnaround time for a submission of data.

2.1 File Folder Structure

It is important to understand the organization of the folder structure before attempting to perform any of the tasks in the Media Processing Pipeline. The folders use a special naming and structure convention, which makes it easy to locate, copy, and load data. All evidentiary data resides on the on a NAS (Network Attached Storage), and security is controlled by Microsoft Active Directory.

Note: In practice, various SEC offices handle their file folder structure differently. This document addresses the file folder structure in the Home Office.

Figure 1 provides an example of a Copy To file folder structure for images in the Home Office.

Figure 1 Sample Copy To File Folder Structure

SEC Procedures for Staging, Loading and Migration Introduction

DRAFT 12

Note: If the images folder contains Optical Character Recognition (OCR) text, you must create an OCR folder, as well.

The following sections describe each level of the file folder structure.

2.1.1 NAS-Hosted File Share and Server

The first level of the folder structure for the Copy To as well as the Live location is, in the case of the Copy To location a Network Attached System (NAS)-hosted file share and in the case of the Live location, a server. The NAS houses the Copy To (also referred to as Temporary) databases and image shares. The \\Xand the \\X servers are on the AD domain. The servers are the Live (or production) locations:

• X (Copy To location)

• X (Live location)

• X (Live location)

2.1.2 Partition/Share

The second level of the folder structure is partition/share level. The X NAS-hosted file share is divided further into shares (folders) and the X and X servers are divided into partitions.

The partitions for the Live locations are named X, X and X.

The Copy To location is where data is copied before it is loaded to the Live or Production location. The shares starting with the prefix ENF have been given the naming feature of ENFimages. The ENF shares (or folders) provide the Copy To location for cases handled by the Home Office. These shares are in numeric order beginning with the number one (1). As a share becomes full, a new one is created. (Ask your supervisor which partition is currently being used.)

The shares starting with the prefix REG have been given the naming feature of REGimages.

The REG files provide the Copy To location for cases handled by the Regional Offices.

These shares are in numeric order beginning with the number one (1). All Copy To databases and their corresponding image files are copied to ENFimagesX or REGimagesX.

2.1.3 Database

The third level of the file folder structure is the Database folder. This folder does not change names; it remains constant in all paths.

2.1.4 Case Number

The fourth level of the file folder structure is the Case Number folder. All cases use a special case number made up of location acronym followed by a 0 and four numbers (e.g., FWRO-

07536 where FWRO stands for Fort Worth Regional Office). The acronym is for the location handling the case, the 0 is constant, and the four numbers vary randomly by case. The attorneys create the case numbers.

Note: In practice, case numbers often depart from the official naming convention. For example, for regional offices, the RO is usually dropped from the case number. Therefore, FWRO-07536 would actually be FW-07536 and ARO-06666 would be A-06666.

The abbreviation used for the home office in Washington DC is HO – Home Office (District of Columbia).

The abbreviations used for the SEC Regional Offices are:

• ARO – Atlanta Regional Office

• BRO – Boston Regional Office

• CHRO – Chicago Regional Office

• DRO – Denver Regional Office

• FWRO – Fort Worth Regional Office

• LARO – Los Angeles Regional Office

• MIRO – Miami Regional Office

• NYRO – New York Regional Office

• PLRO – Philadelphia Regional Office

• SFRO – San Francisco Regional Office

• SLRO – Salt Lake Regional Office

2.1.5 Database Name

The fifth level of the file folder structure is the Database Name (may also be referred to as Source Name) folder. Sources are different entities or groups that send information for a particular case; therefore, a case can have one or more databases. The Data Transmittal Letter or the Electronic Data Transmittal Form indicates the source and database for each submission.

2.1.6 Media Type

The sixth level of the file folder structure is the Media Type. Inside the Database Name folder, a number of folders are created by the Copier staff. The following folders are typical, but the folders in the Database Name folder (other than Images or Edocs) have no set rules for names (could be PDF).

• The Images folder is created by a Copier during the copying process. It contains uniquely named folders which are named after the media Volume Name (if specified on the media

DRAFT 14

label) or a generic volume name (e.g., Vol 00, Vol 01) for the respective submission number.

• The Edocs folder contains files that are not TXT or TIFF files, for example: PDFs or .docs, etc.

2.1.7 Volume ID

The seventh level of the file folder structure is the Volume ID. The Volume ID is used to identify and verify each piece of media. This folder is created under the Images or Edocs folder. The Volume ID (for example, V-HO-01234-053) is composed of three parts.

The first part of the Volume ID tells you something about the media collection. If the number starts with a “V”, the collection came from a producing party and stands for “as is”. If it is listed as “L”, the collection was scanned by X. Following is list of designations for the first part of the number:

• V – producing party loaded “as is”

• L – X

• LI – X

• LE – X

• NX - X

• CE – X

• CO – X

• IGE – X

• IGO – X

The second part of the number is the case number; for example, HO-01234.

The last part of the number is a number that is incremented every time a volume is created;

for example 053.

2.2 Applications Used

Following are the applications used at different stages of the Media Processing Pipeline process as well as the applications used by the attorneys to view the information from the Live location. All of these programs are accessible by double-clicking the corresponding icon on the desktop.

• ETS – Agency Tracking System

• Robocopy Syntax – An application used to file and copy data onto the Copy To location.

• Richcopy – A Microsoft application used to file and copy data onto the Copy To location.

• Multi-Edit/Ultra-Edit/TextPad – Text file viewers and editors.

• Concordance – The litigation support database software that organizes, houses, and provides through searches the imaged evidentiary materials for authorized staff and attorneys.

• Concordance Image (formerly Opticon) – The image viewer that comes with Concordance used to view TIFF images.

• SAL – The SEC Assistant Loader (SAL) is a versatile tool that facilitates the loading of data into Concordance databases by assisting with routine tasks and validation checks.

• Nexidia – An application that performs fast and efficient searches on audio files.

2.3 Media Processing Pipeline

The Media Processing Pipeline takes the electronic data through the following processes in order to properly, accurately, and rapidly load data into Concordance databases (Figure 2).

Figure 2 Media Processing Pipeline

2.3.1 Receive Submissions of Media

Media is prepared and sent by attorneys and vendors to one of the following groups:

• Litigation Support Staff (LSS)

• Centralized Production Unit (CPU) A submission can be delivered as hardcopy or on electronic media, such as one or more CDs, DVDs, thumb drives, hard drives (internal or external), via email, or by File Transfer Protocol (FTP). LSS or CPU staff creates the BDMT request.

Note: All media is tracked using the BDMT Request-SEC E-Media Request Form that is attached to the media. Headquarters and Regional media is sent to the LSS or CPU for review and preparation of the BDMT Request-SEC E-Media Request Form which is then sent to the Copy (Media In/Media Out (MIMO) group.

2.3.2 Copy and Log Submission

DRAFT 16

The Copy staff receives the media from the LSS or CPU staff. Verification of the file folder structure is completed and verification that the media is a new submission and not a resubmission is done as well. The new data is copied to a Copy To location; the copied data is verified against the data in the media; and entered into agency tracking system so that the Loading staff can locate it for loading.

2.3.3 Load Imaged Data

The Loading staff receives the media requests electronically. The Loading staff is notified electronically from the system to begin loading and processing the data. The Loader creates a Copy To database in Concordance and imports the Copy To database to the Live Concordance database.

2.3.4 Quality Assurance

The Quality Assurance staff receives an electronic notification of the loaded data. The QA staff then verifies that the data on the Copy To location matches the data that was loaded into the Live location. BDMT then generates an email to notify the point of contact (POC) that their evidentiary material is ready for use electronically.

2.3.5 Disposition of Media

The original media is packaged and returned to the sender or POC.

SEC Procedures for Staging, Loading and Migration Receiving Media

DRAFT 17

3 Receiving Media Media is prepared and sent, along with a cover letter describing the media, by attorneys and vendors to either the Litigation Support Staff (LSS) or the Centralized Production Unit

(CPU).

LSS reviews the media and creates a request using BDMT based on the information in the cover letter.

CPU is a pilot program. It is tasked with tracking incoming productions (hardcopy or electronic documents) by centralizing the receipt of productions, logging those productions, and generating a production log for investigations. Automatic email notifications are sent to the attorney when a production has been received. After the production has been logged, CPU uses the information in the cover letter to create a request. The media is sent, along with the BDMT request, to a vendor for further processing or directly to the Copy group. For detailed instructions on CPU procedures, refer to the CPU training materials at the following location: X (you will need access permissions to download the training materials).

Note: If media arrives in hardcopy form, it is sent to the Scanning group to be scanned.

Note: If media arrives in the form of audio files, the files are loaded both to Concordance and to Nexidia.

Requests can also arrive via FTP or email. If a request arrives via FTP, you will be directed to one of the following FTP sites to collect the media in the form of one or more ZIP files:

• X

• X If a request arrives via email, it will come in the form of an attached ZIP file and will look something like Figure 3 (most requests that arrive via email come from X).

Figure 3 Emailed Request

A Data Transmittal Letter is included with the media in the ZIP file.

The Transmittal Letter includes the following information:

• The contents of the media (e.g., Concordance database, images, Concordance Image load files)

• Total number of Documents and Pages (or images)

• List of fields in the order in which they are listed in the data file

• The serial or volume number of the media

Media Processing Pipeline Receiving Media

DRAFT 18

You must complete the following major steps to receive new media into the Media Processing Pipeline.

1. Date submissions.

2. Reconcile the item.

3.1 Dating Submissions

Date stamp each request.

3.2 Reconciling the Item

Go to the main page of agency tracking system and reconcile the item by selecting the Processing tab and then Item Shipping Reconciliation. This notifies the requester that you have received the request. Then place the item in a X folder for copying.

SEC Procedures for Staging, Loading and Migration Copying and Logging Submissions

4 Copying and Logging Submissions The copying and logging of submissions is done at the SEC Office, Room SP3 5978. During this step in the process, the Copier verifies that the file folder structure of the new data is correct and that the data has not been received before. The new data is copied to a Copy To location; the copied data is verified against the data in the media; and the location of the new data in the Copy To database is logged into the tracking system so that a Loader can locate it for loading. Use the agency tracking system.

The following steps must be completed to successfully copy and log a submission.

1. Set up the folder structure in the Copy To location.

2. Copy the new data to the Copy To location.

3. Verify the Case Number and Database (or Source Name) of the request.

4. Enter information into the agency tracking system.

The following flowchart provides a visual breakdown of the major steps required to copy and log new data (Figure 4).

Figure 4 Copying and Logging Process

4.1 Setting up the File Folder Structure

It is important to set up the folder structure by creating the necessary folders for the new data.

If folders are missing or if the designated folder structure is not applied, the new data could get lost or could overwrite existing data. (See the File Folder Structure section of this document to learn how the folder structure is organized.)

1. Make note of the volume of the media (CD, DVD, etc.).

2. Insert the CD or DVD into the CD-ROM drive or hook up the hard drive or fire wire.

Note: Some hard drives are not shipped with their own power source; power sources are located in the Copy office.

3. Open a window to view the contents of the new media.

a. Right-click My Computer and select Open. The new media appears in the folder structure.

DRAFT 20

b. Click the plus sign next to the icon with the name of the new media and the contents of the media expand under the media name. Click the icon with the name of the media and the contents appear in the right panel.

4. Open a window to the Copy To location where the new data will be copied.

a. Right-click My Computer and select Explore to open another window.

b. Go to the Copy To location where the new data will be copied.

5. If the request has a case number that does not appear in the Copy To location, create a new folder and give it the name of the new case number.

6. Inside the Case Number folder, create a new folder and use the database name of the submission. The database name is included on the request form.

Note: The Database Name folder may already exist if data had previously been sent for the current case number.

7. Inside the Database Name folder, create a new folder named Images.

Note: If the Database Name folder already exists, create another Images folder labeled 01 and so on.

8. Inside the Images folder, create a new folder using the name of the CD or DVD volume.

The name is located on the label of the media, the Data Transmittal Letter, or the Electronic Transmittal Form. The contents of the new media will be copied into this folder.

4.2 Copying the New Data

The new data is copied into the media volume folder with the help of a program called Robocopy. The program opens up as an MS TextPad application.

1. From the desktop, double-click the robocopy syntax.txt icon.

2. Scroll down and locate the first available drive letter and path that represents the new media.

For example:

robocopy L:\Images "T:\Database\H0-09439" /S /E /Z /R:5 /W:30 /V /xc /xo /x0

The different sections of the path are defined as follows:

• SECTION 1: robocopy L – The location of the new media is the drive letter.

SECTION 1 SECTION 2 SECTION 3

• SECTION 2: "T:\Database\H0-09449" – The Copy To location where the new media will be copied is inside the quotes.

• SECTION 3: /S /E /Z /R:5 /W:30 /V /xc /xo /x0 – Other commands specific to the copy process and live status report.

3. After locating the path in Robocopy, edit the location of the new media and the location where the new media will be copied to ensure that the path points to the locations needed for this submission.

a. SECTION 1 – Delete any additional file folder information. Only the drive letter should appear.

From the Windows Explorer window that shows where the new data will be copied, open the volume number folder and copy the entire path to this location. Be sure it is a mapped drive path meaning that it begins with a drive letter rather than a server name.

b. SECTION 2 – Paste the path of the location. Be sure the path is pasted inside the quotation marks.

c. SECTION 3 – No changes should be made to this section.

The following example of a complete path shows the drive letter of the media, the drive and path where the new data will be copied (in quotes) and the additional copying commands.

robocopy L: "T:\Database\HO-09449\X\Images\X" /S /E /Z /R:5 /W:30 /V /xc /xo /x

4. Highlight the entire new path in Robocopy, right-click, and select Copy.

5. Go to Start > Run.

6. Enter cmd and click OK. A command prompt appears.

7. Right-click at the command prompt. The new path automatically appears. (If the copying does not start, press Enter.) Copying automatically begins.

8. Close all Windows Explorer windows. The copying process speeds up.

9. When the copying is complete, a summary of the progress and a message stating that the copying was successful appears. Close the Command Prompt window.

4.2.1 Copying Audio Files

Enforcement receives audio recordings in many investigations. Nexidia software allows the recordings to be loaded into a database and indexed. Nexidia then performs searches that dramatically improve the efficiency of the review process, allowing staff to zero in on important conversations quickly.

To work with audio files, follow these steps:

1. Receive the audio files and the request from the LSS or the CPU.

DRAFT 22

2. Copy the files to the Copy To location.

3. Push the request to the loading queue (for Concordance loading).

4. Email Nexidia (ENF-Nexidia-Mailbox) providing them with the # and the path to the audio files (for Nexidia loading).

4.3 Verifying the Copy Process

After the new data has been copied to the Copy To location, it must be randomly checked to verify that it matches the data in the media, as follows:

1. Check that the file folder structure of the media volume folder in the Copy To location matches the structure in the media.

2. Check that the size of individual files in the Copy To location matches the size of the files in the media.

3. Create a Copy_Check folder in the Copy To location and take a screenshot of each case copied (property/source and temp) to ensure and record that the data was copied correctly and to the correct location (Figure 5).

Figure 5 Copy_Check Folder

4. Close all windows.

The remainder of the process cannot be done at the Copying Station. It must be performed at another computer that has access to the tracker application.

4.4 Logging the New Data

After copying, use the Request form to enter the following information into the Copy Form (Figure 6):

• Date Received

• Copy Staff

• Request ID

• Case Number

• Live Database

• CopytoLocation

• Manifest ID

• Request Size (if more than one database)

• LoadID Size (if only one database)

• Media Type – From the pull-down menu, select the type of media: CD-ROM, DVD, IDE Hard Drive, Firewire/USB, Floppy Drive, etc.

• Data type

• Container (.exe, .zip, .rar, other)

• Container Size

• Priority (Normal or High)

• Copy Complete Date

• Save / Add New Load for same Request (if you want to create a new load for the same request)

• Save / Enter New Request

• Copy Comments

• Copy Followup

Figure 6 Copy Form

SEC Procedures for Staging, Loading and Migration Loading Data

DRAFT 24

5 Loading Data There are two different types of data processed by the Load team. They are:

• Imaged data (Section 4.1)

• Edocs

5.1 Loading Imaged Data

In this section, the focus is on imaged data, which usually consists of .dat, .txt, .tiff, and possibly native files (files saved in the proprietary format of an application).

5.1.1 Before Loading

An email initiates the loading process. The email has as its subject New Loading Assignments for <current date>. Use the following steps before you start the actual loading process:

1. Read the email (see sample, Figure 7). Pay close attention to the priority. If a request is listed as High priority, process it as soon as possible.

Figure 7 Sample Email

2. Access the BDMT website and navigate to the Loading Queue by selecting the Processing tab from the top of the Home page; then select Loading Queue (Figure 8) from the drop down menu.

Figure 8 Loading Queue

3. Select your loading requests from the loading queue, as follows:

a. Sort by Request ID.

b. Search for the desired RequestID.

c. When you find the RequestID, check the corresponding box in the Load column.

d. Click the Load Selected Items button at the top or the bottom of the screen to bring up the request.

4. Open the database.

5. Bring up the Load Search form by clicking Load Staff (Figure 9).

Figure 9 Load Search Form

SEC Procedures for Staging, Loading and Migration Loading Data

6. In the Load Search form, enter the request number in the RequestID field, and click the Search button.

The Load Form appears for the target request (Figure 10).

Figure 10 Load Form

7. In the request Load Form, enter the LoadID and LoadID Size Out in the appropriate fields. Get this information from the request.

8. Verify that the location information in the CopytoLocation and LiveDB Path fields is correct by testing the paths. Make sure the paths take you to the Copy To location and the Live location.

5.1.2 Loading the Data

To load imaged data, follow these steps:

1. Analyze the Data

2. Create the Database

3. Prepare for Export

5.1.2.1 Analyze the Data

Analyze the data in the Copy To location to see what you have, as follows:

1. On the Load Request screen, read and follow any special instructions (see Figure 11) related to loading the data.

Figure 11 Special Instructions

2. Remove the Read-only property from the Data folder by right-clicking it and selecting Properties.

The Data Properties dialog box appears (Figure 12).

Figure 12 Data Properties Dialog Box

3. Deselect the Read-only checkbox to remove the read-only property from the data folder.

4. Using a text editor (Ultra-Edit or TextPad), globally edit the .log (or .opt) file, so that the path(s) to the image file is complete. Figure 13 displays an unedited .log (or .opt) file.

SEC Procedures for Staging, Loading and Migration Loading Data

DRAFT 26

Figure 13 Unedited Opt File

Figure 14 displays the unedited .log (or .opt) file with IMAGES selected.

Figure 14 Unedited Opt with \IMAGES\ selected

Figure 15 displays the edited .log (or .opt) file with the full path to the images in the Copy To location.

Figure 15 Edited Opt File

Note: Be sure to test the first and the last paths in the .log (or .opt) file to verify that all the paths are correct.

5. Count the Y breaks in the .log (or .opt) file in order to get a total of the number of documents that should be in the database, as follows:

a. Select the binoculars icon from the toolbar of your text editor.

b. In the Find What field, enter ,Y, and click the Find Next button.

A message appears displaying the number of documents that are in the .log (or .opt) file (Figure 16).

Figure 16 Count the Y Breaks

6. Analyze the.dat file to make sure that:

• The file format is PC (Figure 17)

• The encoding is ANSI (Figure 17)

• There is an extra blank line at the bottom of the file (Figure 20)

Figure 17 File Format and Encoding

Also:

• Determine the date format of the file (Figure 18).

• Determine the position of the Bates fields (Figure 19).

SEC Procedures for Staging, Loading and Migration Loading Data

• Determine the position of the Imagekey field (Figure 19).

Figure 18 Date Format

Figure 19 Header Row (Bates and Imagekey)

Figure 20 Extra Blank Line

7. Once you have satisfactorily analyzed the .dat file, you can run the SAL application to create the database.

8. Proceed to the Create the Database section.

5.1.2.2 Create the Database

To create the database in the Copy To location, follow these steps:

1. Use the information you gathered from your analysis and complete the fields in the SEC Assistant Loader (SAL) Preload Analysis: Step 1 – User input screen (Figure 21), as follows:

a. Select the Select DAT file(s) to analyze checkbox.

b. Select the DAT file(s) contain header row checkbox.

c. Select the DAT file in the selection box.

d. Select the radio button that displays the correct date format.

e. In the text boxes corresponding to Beginning Bates field #, End Bates field #, and Image ID field #, enter the order of these fields as they appear in the header row of the .dat file.

f. Select the Use Concordance delims checkbox if the DAT file contains the standard Concordance delimiters. If the DAT file does not contain them, enter the appropriate delimiters for the comma, quote, and newline.

2. Scroll down to the Analyze Report button and click it.

Figure 21 SAL Preload Analysis: Step 1 User Input Screen

SEC Procedures for Staging, Loading and Migration Loading Data

DRAFT 28

The Preload Analysis Report screen appears: Top of screen (Figure 22), middle of screen (Figure 23), bottom of screen (Figure 24).

Figure 22 SAL Preload Analysis Report: Top of Screen

Figure 23 SAL Preload Analysis Report: Middle of Screen

Under the Field Names section (shown in Figure 23), SAL displays all of the fields that are present in the DAT file. Here you can alter the name, type, and format for each field.

You can also set fields to Key or Indexed, as appropriate.

3. In the New DB Name box, enter the name of your Copy To database. It is customary to leave the name as is, which is TEMP_DB.

4. In the New DB Location box (shown in Figure 24), enter the path to the SAL output folder.

SEC Procedures for Staging, Loading and Migration Loading Data

Figure 24 SAL Preload Analysis Report: Bottom of Screen

5. Once SAL is finished creating the Copy To database, open it.

6. Proceed to the Prepare for Export section.

5.1.2.3 Prepare for Export

Prepare the Copy To database for export to the Live location. The following subsections describe the tasks you need to perform.

5.1.2.3.1 Check Dates

1. Make sure there are no bad dates by running a date check. To do this select CPL from the menu bar at the top of the screen and then select Mpg - Date Check from the drop down menu (Figure 25).

SEC Procedures for Staging, Loading and Migration Loading Data

DRAFT 30

Figure 25 Date Check CPL

The Date Check Results screen appears reporting on any bad dates (Figure 26).

Figure 26 Date Check Results

2. If there are bad dates, fix them, if possible. If the bad dates cannot be fixed make a note of them.

5.1.2.3.2 Add Fields

Add the LOADID, REQUESTID, SOURCE, PROD_DATE and OCRTEXT fields. You may also need to add the DATELOADED field.

To add these fields, follow these steps:

1. Select File and then Modify.

The Modify screen appears (Figure 27).

Figure 27 Modify Screen

2. Populate the SOURCE, PROD_DATE, LOADID, REQUESTID and OCRTEXT fields with the information that is in BDMT.

3. Click OK.

5.1.2.3.3 Complete Paths to Native Files

If there are any native files, complete the paths to those files, as follows:

1. Highlight and copy the piece of information in the native field that will be targeted for global editing. In the screenshots below, the native field is named LINK_TO_NAT (Figure 28).

Figure 28 Natives 1

2. Select Edit and then Global Replace (Figure 29).

Figure 29 Natives 2

SEC Procedures for Staging, Loading and Migration Loading Data

The Global Replace screen appears (Figure 30).

Figure 30 Global Replace Screen

3. In the Global Replace dialog box, select the native field from the list of fields.

4. Enter the target of your global replace (i.e., \MCD734_154\NATIVES\) in the Target field.

5. Enter the full path to the NATIVES folder in the Replacement field.

6. Click OK.

There should now be a full path to the native files in the Copy To database (Figure 31).

Figure 31 Full Path

5.1.2.3.4 Create Hyperlinks

You must create hyperlinks for paths in the native fields described in the previous section.

To create hyperlinks, follow these steps:

1. Select CPL and then Create Hyperlinks (Figure 32).

Figure 32 Create Hyperlinks

2. When a message with instructions appears (Figure 33), follow the prompts:

a. Click Select Field and select a field with a path from the dropdown that appears.

Figure 33 Create Hyperlinks: Step 2

b. Click [G] Go!

The system returns you to the database where the new hyperlink is displayed (Figure 34).

Figure 34 Hyperlink Created

c. Click [Q] Quit and proceed to the next field that needs a hyperlink.

SEC Procedures for Staging, Loading and Migration Loading Data

DRAFT 32

5.1.2.3.5 Perform a Hyperlink Check

Run the Hyperlink Check CPL, as follows:

1. Select CPL and then Mpg – Hyperlink Check (Figure 35).

Figure 35 Hyperlink Check

2. When a message appears with instructions, follow the prompts:

a. Click [S]elect Hyperlink Field and select a field that you want to check for hyperlinks from the dropdown menu.

b. Click Select [O]utput File, browse for the output file, and click on it.

c. Click Check Hyperlinks

The results of your query appear (Figure 36).

Figure 36 Hyperlink Check Results

3. If everything is fine, click OK and E[x]it.

5.1.2.3.6 Import OCR Text

If you are missing OCR text, you need to import it.

To import OCR text, follow these steps:

1. Create an OCR text field in your database. Typically you would name it OCRTEXT.

2. Determine where the OCR text is located. Often there is an OCR folder. Sometimes the text files are in the Images folder.

3. From the Concordance menu bar, select CPL and then Readocr (Figure 37).

SEC Procedures for Staging, Loading and Migration Loading Data

Figure 37 Readocr CPL

A message appears describing the Read OCR Utility (Figure 38).

Figure 38 Read OCR Utility

4. Click OK.

The OCR Import Menu appears with instructions for importing OCR text (Figure 39).

5. Follow the prompts:

a. Skip the first two prompts [O]pen a database and [S]earch for a database and go directly to the third prompt.

b. Click [I]mage field select and select the field that contains data that matches the name of the text field; usually FIRSTBATES or IMAGEID.

c. Click O[C]R field select and select the OCR text field you created.

d. Click Directory of OCR text and insert the path to the OCR folder that contains the target file.

e. Click [G]o!

Figure 39 OCR Import Menu

SEC Procedures for Staging, Loading and Migration Loading Data

DRAFT 34

A message appears briefly notifying you that the system is processing. If the operation is successful, a file named ocroutput.LOG that contains OCR text appears in the target folder.

f. Click [Q]UIT to exit.

5.1.2.3.7 Perform a Field Comparison

You must check to see if the fields in the Copy To database exist in the Live database by running the Field Compare CPL.

To perform a field comparison between the Copy To and Live databases, follow these steps:

1. Select CPL and then Mpg – Field Compare (Figure 40).

Figure 40 Field Check: Step 1

2. When a message with instructions appears (Figure 41), follow the prompts:

a. Click Select [L]ive Database, browse for the location of the Live database and open it.

b. Click Select [T]emp Database, browse for the location of the Copy To database and open it.

c. Click Choose [O]utput File, browse for the location of the output file where you want the results to go and open it.

d. Click [C]ompare Databases.

The results of the comparison appear on the screen as well as in the output file.

Figure 41 Database Field Comparison

e. If there are no conflicts, you may now export the Copy To database to the Live location.

f. Proceed to the Export the Database to the Live Location section.

5.1.3 Export the Database to the Live Location

To export the Copy To database to the Live location, follow these steps:

1. Sort the database by FIRSTBATES (Figure 42).

SEC Procedures for Staging, Loading and Migration Loading Data

Figure 42 Sort by Panel

2. After selecting FIRSTBATES, click the Sort button.

3. On the main menu bar, select Documents, then Export, then As a Concordance database….

The Export Wizard – Database dialog displays (Figure 43).

Figure 43 Export Wizard – Database

4. In the Database field, browse to the location of the Live database where you plan to export to and click Next.

The Export Wizard – Fields screen appears (Figure 44).

SEC Procedures for Staging, Loading and Migration Loading Data

DRAFT 36

Figure 44 Export Wizard – Fields

5. Select all the fields and click Next.

The Append/Replace screen appears (Figure 45).

Figure 45 Append/Replace

6. Deselect the Copy attachments with notes checkbox and click Next.

The Export Wizard – Export screen appears (Figure 46).

Figure 46 Export Wizard – Export

7. Enter the range of documents to be exported in the First and Last fields and click Export.

8. After the export has completed, reindex the Live database by selecting File from the Concordance main menu bar and then Reindex (Figure 47).

SEC Procedures for Staging, Loading and Migration Loading Data

Figure 47 Reindex

9. Proceed to the Run a Duplicate Detection Check section.

5.1.4 Run a Duplicate Detection Check

After reindexing the Live database, the very first check that you should make is the Duplicate Detection check.

Check for duplicate documents, as follows:

1. From the toolbar at the top of the Concordance main page, select Tools and then Check for duplicates.

The Duplicate Detection screen appears (Figure 48).

2. From the list of fields in the Duplicate Detection screen, select the image key field (usually FIRSTBATES or IMAGEID).

3. Locate the previously prepared Dupe check.txt file containing the tags to be used in the duplicate check process.

4. Use the duplicate text strings in the Dupe check.txt file to populate the Original and Duplicate fields in the Duplicate Detection screen.

5. Click OK to run the duplicate check across the entire Live database.

Figure 48 Duplicate Check

6. If true duplicates are found that are from your request, do not load the images; place the request in follow-up.

7. If duplicates are found that are not from your request, you may proceed with loading the images.

8. Proceed to the Load the Images section.

5.1.5 Load the Images

To load the images, follow these steps:

1. Open the Concordance Image viewer by clicking the View Image icon .

2. From the Concordance Image menu bar, select Tools and then Imagebase Management.

The Imagebase Management screen appears.

SEC Procedures for Staging, Loading and Migration Loading Data

DRAFT 38

3. Browse for the .Dir file for the Live database and open the file.

The Imagebase Management dialog box for the .Dir file appears (Figure 49).

Figure 49 Imagebase Management Dialog Box

4. Click the Register – Load tab.

The Register – Load tab appears (Figure 50).

Figure 50 Register Load Tab

5. Click Browse and navigate to the .opt (or .log) file in the Copy To location and double-click it.

6. Click Load.

7. Once the images have completed loading, click OK.

8. Proceed to the section Isolate the Load.

5.1.6 Isolate the Load

To isolate the load, search for the LoadID and ensure that the number of documents in your load match the number in your Copy To database.

5.1.7 Perform Quality Control Checks on the Live Database

After loading all of the images, there are numerous quality checks that you must conduct.

They are:

• Check dates

• Perform a Hyperlink Check

• Perform a CPL OCR Check

• Perform a Search Check

• Check the First and Last Images

• Perform an Image Link and Path Check

• Perform a Concatenation Check

• Check Menu Lists

• Check Security Settings

SEC Procedures for Staging, Loading and Migration Loading Data

5.1.7.1 Check Dates

It is important to ensure that there are no bad dates in the database, such as dates with the wrong format.

To perform the CPL Date Check on the Live database, follow these steps:

1. Run the Date Check CPL by selecting CPL from the Concordance menu bar and then selecting Mpg - Date Check (Figure 51).

Figure 51 CPL Date Check

2. If you do not have one already, create a valid .log file for this check.

The CPL Date Check Results screen for the Live database appears (Figure 52).

Figure 52 Date Check Results

3. Click OK.

5.1.7.2 Perform a Hyperlink Check

Run the Hyperlink Check CPL, as follows:

1. Select CPL and then Mpg – Hyperlink Check (Figure 53).

Figure 53 Hyperlink Check

When a message appears with instructions, follow the prompts:

SEC Procedures for Staging, Loading and Migration Loading Data

DRAFT 40

2. Click [S]elect Hyperlink Field and select a field that you want to check for hyperlinks from the dropdown menu.

3. Click Select [O]utput File, browse for the output file, and click on it.

4. Click Check Hyperlinks

The results of your query appear (Figure 54).

Figure 54 Hyperlink Check Results

If everything is fine, click OK and E[x]it.

5.1.7.3 Perform an OCR Check

Performing an OCR check involves the following activities:

• Performing a CPL OCR check

• Performing a search check

• Checking the first and last images

5.1.7.3.1 Perform a CPL OCR Check

Ensure that there is no missing OCR text, by performing a CPL OCR Check.

To perform the CPL OCR Check on the Live database, follow these steps:

1. Run the OCR Check CPL by selecting CPL from the Concordance menu bar and then selecting Mpg - Ocr Check (Figure 55).

SEC Procedures for Staging, Loading and Migration Loading Data

Figure 55 OCR Check

The OCR Check screen appears (Figure 56).

2. Click [C]hoose OCR field to check and select the OCRTEXT field.

Figure 56 OCR Check Screen

3. Click [B]egin Processing Documents.

The OCR Check Results page appears along with an OCR Check Complete message (Figure 57).

Figure 57 OCR Check Results

SEC Procedures for Staging, Loading and Migration Loading Data

DRAFT 42

4. Click OK.

5.1.7.3.2 Perform a Search Check

Perform an OCR text quick search to ensure that the text is being highlighted correctly.

To perform a search check, follow these steps:

1. Select a word that you believe will generate hits within your load.

Words such as financial, confidential and investment tend to be good choices.

2. In the Search box at the top of the screen, type a short query which will include the LOADID for the current load and the word you have selected, such as confidential.

Note: This query ensures that you search only your load and not the other possibly many loads in the database.

Following is an example of a query you might use:

LOADID EQ 54321 AND confidential

3. Click the (Go) button to the right of the Search box.

Concordance searches the text of your load for the word. All records in your load with that search word or words should be returned with the search words highlighted in red.