Attach 1. 261008_RFI Supplemental Information_v1_1.pdf

Enterprise Modernization of the Drug Enforcement Administration Diversion Control Division’s (DC) Information Technology (IT) Systems

RFI Number: D-27-DC-91626

Supplemental Information

1. System Description There are three DC systems on which all applications within the transformation effort are housed:

The Registrant Support Network (RSN) houses the RICS, ARCOS, and CTRANS applications.

The Controlled Substance Ordering System (CSOS) houses the CSOS application and infrastructure.

The Diversion Control Contact Center (DC3) houses the DC3 application. The Diversion Control Contact Center Chatbot (DC4) resides on the same network as DC3, but is not a part of the DC3 system.

Each environment has its own development, test, and production networks.

1.1 RSN and CSOS System components:

• Microsoft Windows Server o 2012 R2 o 2016 o 2019 o 2022 o 2025 o Active directory- 2019 domain and forest

• Microsoft Windows o 10 o 11

• Microsoft Windows Server Update Services

• Manage Engine Endpoint Central

• Red Hat Enterprise Linux o 7.9 o 8.10

DCIT Supplemental Information 1

• Red Hat Satellite

• Nutanix Acropolis Hypervisor

• VMWare vSphere

• Cisco Unified Computing System (UCS)

• Network Attached Storage (NAS)

• Dynatrace

• DOJ BigFix Continuous Monitoring

• Qualys

• Crowdstrike

• Epicor IT Service Management (ITSM) NOTE: Call center representatives with CSOS expertise use ITSM to log and track CSOS tickets. ITSM provides access to registrant CSOS data that is not available to DC3 currently.

1.2 DC3 System Components

• Salesforce o Salesforce Service Cloud o Salesforce native storage o Omni-Channel

• AWS Infrastructure o API Gateway o Lambda

• AWS Connect

• Microsoft Windows o 11

1.3 DC4 Components

• AWS WAF

• AWS public load balancer

• Amazon EC2 with Chainlit Hosts

• AWS Lambda

• Amazon Bedrock (Nova Pro) Foundation model (Amazon Nova Pro) and guardrails

• Amazon RDS for PostgreSQL with pgvector

DCIT Supplemental Information 2

• Amazon S3

• Amazon CloudWatch

• AWS IAM and AWS KMS

• Amazon Athena

• AWS CloudTrail, Config, GuardDuty, Security Hub, Inspector, Systems Manager

1.4 Cloud Infrastructure

Almost all DC infrastructure has moved to an Amazon Cloud instance. The CSOS Certificate Authority is currently physical, along with RSN Virtual Desktops, and F5 load balancers. It’s anticipated that these assets will migrate to the cloud as well before the transformation effort begins.

The AWS Cloud infrastructure includes but is not limited to:

• AWS GovCloud resources, including VPCs, EC2 instances, S3 buckets, RDS databases, SES, ALB, NLB, WorkSpaces, and other services

• Use of AWS Direct Connect for secure connectivity.

• Deployment of monitoring tools such as AWS CloudWatch and AWS Config for real-time visibility and compliance tracking.

• FedRAMP High Baseline, where possible, for cloud security

1.5 Technology Stacks

The following sections list the technology stacks used by the various DC development teams.

1.5.1 Java Technology Stack

• Apache Struts

• CSS

• HTML

• Java

• Java/Jakarta Enterprise Edition

• JavaScript

DCIT Supplemental Information 3

• jQuery

• JSF

• JSP

• PrimeFaces

• Spring

• XHTML

1.5.2 RICS/ARCOS/CTRANS Technology Stack

• Bootstrap

• CSS

• HTML

• JavaScript

• jQuery

• ORA_EXCEL

• Oracle Application Express (APEX)

• PL_JSON

• PL/SQL

1.5.3 CSOS Technology Stack

• CSS

• HTML

• JavaScript

• PL_JSON

• PL/SQL

• Powershell

1.5.4 DC3 Technology Stack

• Amazon API Gateway

• AWS CloudFormation

DCIT Supplemental Information 4

• Amazon Simple Notification Service

• AWS Step Functions

• Amazon RDS for PostgreSQL

• AWS Lambda

• AWS X-Ray

• Amazon DynamoDB

• Amazon S3

• AWS Glue

• AWS CloudTrail

• AWS KMS

• AWS CloudWatch

• AWS Redshift

• AWS IAM Access Analyzer

• AWS Secrets Manager

• Amazon EventBridge

• AWS IAM

• Amazon Direct Connect

• Amazon Kinesis Data Streams

1.5.5.1 Salesforce Components

• Salesforce Service Cloud: Case management, dashboards, workflows

• Salesforce Shield: Field-level encryption and audit trails

• Native dashboards and operational reports

2. Registration Metrics Taking metrics from the past three years (Jan 2023 – Dec 2025), there have been an average of:

• 230,000 new applicants per year

DCIT Supplemental Information 5

• 480,000 renewals per year

• 2,060,000 updates per year

Updates include:

• BUSINESS ACTIVITY CHANGED

• LAST NAME CHANGED

• ADDRESS CHANGED

• DRUG CODES ADDED

• DRUG SCHEDULE CHANGED

• DRUG CODES DELETED

• LIABILITY QUESTIONS CHANGED

• EXEMPT STATUS CHANGED

• SOCIAL SECURITY NUMBER CHANGED

• APPLICANT PHONE CHANGED

• BUS ACTIVITY SUBCODE CHANGED

• BULK DRUG ADDED

• BULK DRUG DELETED

• STATE LICENSE 1 CHANGED

• STATE LIC 2/CDS CHANGED

• FIRST NAME CHANGED

• STATE LIC EXP DATE CHANGED

• POC E-MAIL CHANGED

• APPLICANT CELL PHONE CHANGED

2.1 ARCOS Metrics

ARCOS receives about 7,000,000 transaction reports monthly.

3. User Metrics

SYSTEMS

INTERNAL

USERS

EXTERNAL

USERS

ARCOS Online

2,113

ARCOS Online EDI

5,390

CSOS 751 240,000

CTRANS 1,490

DCGIS 1,070

DCIT Supplemental Information 6

SYSTEMS

INTERNAL

USERS

EXTERNAL

USERS

eSurvey (typically one time users only)

37,579

Industry Tip Reporting

Online BCMR

Online Forms: Order Forms

Online IMEX (Chemicals, Controlled Substances)

1,178

Online IMEX (Machines)

Online NTBI

Online Quotas

1,085

Online Registrants Dataset Access

19,802

Online SORS

23,392

Online TOX

Online YERS

1,599

QMS/YERS 108

REGISTRANTS

2,270,596

RICS/ARCOS 5,449

SDRS 20

DC3 112

TOX 28

GRAND TOTAL 9,028 2,603,659

3.1 Internal Accounts and Users

Most internal accounts are unique and managed by RICS/ARCOS applications. While a user may have to log in separately to access CTRANS and RICS (for example), their

DCIT Supplemental Information 7 username and password are the same. Therefore, individuals (as opposed to accounts) on the internal system is around 5500 users.

The exception to the above is DC3 and CSOS. Both have their own user lists, and are accessed through PIV authentication with DEA.

Internal iterations of the RICS, ARCOS, DC3, and CSOS applications have the most complex user role environments, as they rely heavily on role-defined workload. Generally, each has an Admin role which provides access to all functions, including application configuration (table/system settings; role creation; workload creation); an owner/manager role which provides access to all functions related to business operations; and then roles as necessary to accomplish the day-to-day mission.

Internal users log in daily.

3.2 External Accounts and Users

RICS, CTRANS, and ARCOS external accounts are all authenticated to a unique username/password, with the exception of Registration accounts. Currently, registrants provide account related information along with a PIN sent to a verified email address to log in to their registration, or to applications that provide access to registration account information. Each registration must be accessed separately. Of the 2.2 million registrations, about 1.8 million belong to unique individuals or organizations.

Most external application users log in only as required to renew their registration or make updates. (Renewal cycles are available on www.deadiversion.usdoj.gov/registration)

• ARCOS reporters typically log in to make reports either monthly or quarterly, depending on what they’ve agreed to with DEA.

• Online IMEX users may log in daily, depending on how often they report shipments.

CSOS external users access their accounts by authenticating through login.gov.

DC3 has no users external to DEA.

4. Pay.gov DC uses pay.gov to negotiate and receive payment from registrants. Two methods are used:

For transactions under $5000 (the majority of payments to DEA), external applications requiring payment establish a connection with pay.gov through an API. Once the connection is established in the application, pay.gov receives the registrant’s payment information. If the payment information is accepted by pay.gov, pay.gov sends an

DCIT Supplemental Information 8 http://www.deadiversion.usdoj.gov/registration acceptance message to DEA. Only the amount paid (which is mandated by the CFR) and the fact of the payment are retained by DEA. No other payment details are stored. DEA only accepts credit card payments for these payments.

For chain registration payments, DEA establishes a connection with pay.gov through a separate external-facing application, much as described above. Once the connection is established and payment details entered, pay.gov sends DEA notice that the payment has been entered. These are typically large sums; payment may not be verified until several days later. When payment is verified, pay.gov updates DEA. These payments are made through ACH debit cards.

5. Diversion Communication

5.1 Blast Email

If a Diversion component requires that a blast email be sent out the following steps are taken:

1. Diversion sends the body of the email, the intended recipients, and the subject line to authorized AWS SES users for fulfillment.

2. Email addresses are obtained from Diversion, or from application records

3. AWS SES users compose the email and send it out.

5.2 Website Banner

Diversion may occasionally decide to alert the public and registrants about important issues by posting a message on a banner on the website.

1) Diversion approves the language and submits a change request

2) The website administration team updates the website banner with the approved language.

5.3 Physical Mail

Almost all mail that goes out from POD is DEA Form 222 (order form), requested by registrants.

1) A registrant or a DEA employee requests an order form.

2) The request is processed and the print file is updated.

3) The print file is processed the next day. The order form is printed, folded, inserted, metered, and mailed.

DCIT Supplemental Information 9

Occasionally, Diversion will require a mass mailing to go out. A custom print file is prepared.

1) The print file contains the PDF of the mail to go out.

2) Diversion indicates which registrants the mail should go out to. The print file is updated with recipients.

3) The print file is processed the next day. The letter is printed, folded, inserted, metered, and mailed.

5.4 SMS

Note that SMS messaging is currently provided via government cell phone, not by any applications.

The call center and some field representatives use SMS as a part of authenticating individuals on the phone to verify they’re authorized to receive information about a registration.

1) Call center representatives and field operatives have a government issued cell phone.

2) If authentication is required, the representative sends a PIN in a text to the Point of Contact cell phone that’s captured in the registration record.

3) The contact reads the PIN back to the representative, and the call continues (if the PIN is valid).

If the PIN is invalid, the representative may send another PIN, or validate the caller through other means. All representatives follow standard operating procedures for identity verification.

5.5 Email Authentication

Similarly, when a registrant attempts to log in to an application that permits them to review PII or alter their registration, the application will verify their identity by sending a PIN to the email address on file in their registration. Once the PIN is input into the application, the application will proceed with validating their identity.

Failure to provide the correct PIN may result in the account being locked and access being blocked.

6. Storage The table below shows the storage size of specific tables in DC applications:

DCIT Supplemental Information 10

TABLE NAME Systems

SUM OF STORAGE

(GB)

TABLE 1 ARCOS 4635.8

TABLE 2 CSA INTERNAL 1422.47

TABLE 3 CSA WEB 449.79

TABLE 4 CSOS 107.56

TABLE 5 IMAGES 1810.6

TABLE 6 REPORT 36.6

TABLE IN GB 8463

TABLE IN TB 8

6.1 Data Migration

It is anticipated that all data will be imported into the transformation solution with 100% fidelity.

7. Record Counts Record counts:

RICS- Assuming one registration is a record: 4,747,744. (No duplicates.)

ARCOS: Assuming one transaction is a record, 1,568,017,457 (no duplicates)

CTRANS: Assuming one transaction is a record, (No duplicates)

CHIMEX IMPORT: 50,175

CHIMEX EXPORT: 196,509

CHIMEX Broker: 7,467

CHIMEX Transshipments: 685

CHEMS: 29,268

CSIMEX Declarations Import: 63,280

CSIMEX Declarations Import: 64,753

DCIT Supplemental Information 11

Zielinski, Nickalus J TB and GB are swapped

Roberts, Scott M.

Swapped – thanks.

CSIMEX Permits Imports: 23,148

CSIMEX Permits Exports: 107,572

CSOS: Assuming one participant is a record: 281,489 (no duplicates)

DC3: Assuming one case is a record, 553,117 (no duplicates)

Documents DC systems receive an average of 25,000 PDFs per year.

8. Data Quality Conditions:

Data is accurate, complete, and valid. No unwanted duplicates are present (all records counts above, for the considered systems, require uniqueness). Data which is required for decision making is timely according to business requirements. All systems control for data quality.

Note that when addressing duplicates, this document applies to duplicate records. For example, a single individual may have multiple DEA registrations within RICS, with each one being a separate record. The records are not duplicates even though there’s only one individual.

9. Interconnections Generally:

Public-Facing internet

• Provided by DOJ JUTNET

• CSOS and RSN have DMZ that host websites and applications (Https)

• DC3 hosts a few web pages and applications (Https)

1) Contact Center Chat application

2) Microsite which acts as a splash page for reaching Call Center operations.

• The chatbot (DC4) is hosted on AWS cloud servers exposed to the internet (https)

• RICS receives a daily file from First Databank for National Drug Codes. (SFTP)

• Security updates and patches are downloaded from vendor sites as required, tested, and deployed IAW DEA and DOJ security requirements.

Government-to-Government

• The IMEX applications provide a daily log of transactions to the Customs and Border Protection’s Automated Commercial Environment (ACE). The connection is secured

DCIT Supplemental Information 12 through JUTNET. Files are uploaded via SFTP to ACE, and picked up by CBP for processing.

• Diversion applications which require payment connect to pay.gov (HTTPS)

Diversion-to-DEA

• RICS provides a view of its data to various DEA components that require it for analysis outside of the Diversion environment.

• ARCOS and GIS data is provided to investigative and targeting units to build investigative packages for the field.

• The DRMS application has a connection to the DEA NFS application for referencing employee data.

Internal

• CSA data is integrated with all other RICS applications

• CSA data is provided to ARCOS to help identify registrants.

• CSA data is provided to CSOS to help identify registrants and ensure that certificates are revoked for retired registrations.

10. Recovery Point Objective (RPO) and Recovery Time Objective (RTO)

The RTO and RPO for all applications is 24 hours.

The only exception to this is that DC3 has an RTO of 12 hours.

11. Incident Response

11.1 Incident Severity Levels

Incidents shall be classified as follows:

• Priority 1 (Critical): Complete outage, material security event, or severe degradation causing a critical business system or service to be unavailable to all or substantially all users, with no reasonable workaround.

• Priority 2 (High): Significant loss of functionality or performance degradation materially affecting business operations, where a workaround may exist but business impact remains substantial.

DCIT Supplemental Information 13

• Priority 3 (Medium): Partial, non-critical loss of functionality or issue causing moderate business impact.

• Priority 4 (Low): General inquiry, cosmetic issue, routine service request, or minor defect with minimal business impact.

Incident severity may be reclassified by the government in consultation with the vendor as circumstances warrant, and as availability, functionality, and performance elements change.

11.2 Response and Escalation Obligations

Priority

24/7/365 Within 15 minutes

Immediate escalation to on-call engineer and management, as needed

Priority

24/7/365 Within 1 hour Escalation to appropriate technical personnel

Priority

Business hours unless otherwise stated

Within 4 business hours

Standard support process

Priority

Business hours unless otherwise stated

Within 1 business day

Standard support process

For Priority 1 incidents, the Contractor shall use continuous commercially reasonable efforts until service is restored or a mutually agreed workaround is implemented. For Priority 2 incidents arising during After-Hours Support Periods, the Contractor shall commence remote triage and remediation promptly upon acknowledgement.

11.3 Support Channels

The Government may initiate support requests through the Contractor’s designated support portal, email address, or emergency support telephone number. Priority 1 incidents must be reported through the emergency support telephone number or other designated urgent escalation channel to ensure immediate routing to on-call personnel.

11.4 Included Services

24/7/365 Support and After-Hours Support shall include:

• Automated monitoring and alerting for covered systems;

• Incident intake, acknowledgement, and severity assessment;

DCIT Supplemental Information 14

• Remote troubleshooting, diagnosis, and remediation efforts;

• Coordination of escalation to specialized technical resources, vendors, or subcontractors, as applicable;

• Communication of status updates to Client during active Priority 1 and Priority 2 incidents; and

• Verification of service restoration following incident resolution.

11.5 Exclusions

Unless expressly included in the applicable contract documentation (PWS, Order Form, etc.), the following are excluded from 24/7/365 Support and After-Hours Support:

• New feature development or enhancements;

• Non-urgent service requests;

• End-user training;

• On-site support;

• Support for third-party systems not under Contractor’s management; and

• Issues caused by Client-controlled infrastructure, unauthorized changes, misuse, or force majeure events.

12. Performance Baselines Performance Objective Performance Threshold

System Performance 99.9% availability. Meet RTO and RPO.

Data Performance 99.9% availability. Meet RTO and RPO.

Database Availability 99.9% availability. Meet RTO and RPO.

Network Performance 99.9% availability. Meet RTO and RPO.

System Performance 99.9% availability. Meet RTO and RPO.

Storage Performance 99.9% availability. Meet RTO and RPO.

Test Coverage Ensure test coverage of 95% for critical functionality

DCIT Supplemental Information 15

Performance Objective Performance Threshold

Application Availability Ensure applications are available 99.5% of the time.

Website Uptime The website will be operational greater than 99.5% of the time.

Accessibility Compliance 100% conformance with applicable standards.

Print on Demand daily and monthly mailings

95% of daily and monthly mailings will be printed sorted, and inserted correctly, and mailed on time.

13. Data Migration It is expected that all data will be imported into the transformation solution with 100% fidelity.

DCIT Supplemental Information 16