AIML Tech SOW.pdf

UNITED STATES

SECURITIES AND EXCHANGE COMMISSION (SEC)

Office of Acquisitions 100 F. Street NE

Washington, D.C. 20549

Statement Of Work

AIML (Artificial Intelligence and Machine Learning) Technical SME (Subject Matter Expert) Services

1. Requesting Office:

Office of Information Systems (OIT) Enterprise AIML group

2. Description of Work/General Tasks:

The U.S. Securities and Exchange Commission (SEC), Office of Information Technology (OIT), Enterprise AIML team would like to acquire the technical SME services starting in April 2025 range. The services will primarily support to mature, operate, maintain and enhance the existing production OIT AIML (Artificial Intelligence and Machine Learning) Tech Foundation and its hosted production pilot products;

to continue to implement the AIML governance services and establish the basic AIML risk management framework at SEC to manage the AIML use case portfolio and products as required; and to provide Tech SME consultancy and assessment to enable the responsible use of AIML technologies and innovative analytics to support Agency’s mission.

2.1 OIT AIML Tech Foundation Maturity and Operation

Since Q4 2022, OIT AIML Group has built and deployed this AIML technical foundation version

1.0 within SEC AWS Cloud (ACES) environment. This AIML tech foundation largely leverages the FedRAMP-Authorized services within ACES include but not limited to AWS Elastic Container Service (ECS), AWS Elastic Container Registry (ECR), AWS Relational Database Service (RDS/Postgres), AWS Elastic Map Reduce (EMR), AWS Cloud Formation, AWS Service Catalog, ACES CICD Service, and AWS Application LoadBalancer, AWS Network LoadBalancer, AWS CloudMap(Service Discovery); as well as a short list of key non-AWS software: such as open source Airflow-WebUI, Scheduler, Workers, Redis (for Airflow internal communications);

Streamlit-App for data manipulation, data visualization/customer interface.

So far, this tech foundation has enabled the operation of several main Natural Language Processing (NLP) technologies and algorithms such as data cleaning tokenization, feature engineering TF-IDF, random forest classifier, named-entity recognition, text summarization, and text classification.

Some of the main models that the tech foundation supporting to execute, infer, and evaluate include but not limited to SpaCy, FinBERT, High Performance Levenshtein Distance algorithm, and open source localized Large Language Models (LLMs): Meta-Lama 2-3 and Mixtral. In addition, this tech foundation has proven to be flexible, scalable within ACES/AWS eco-system and can easily integrate or supplement main AIML services that AWS offers such as Sagemaker and more recently Bedrock.

OIT AIML Group would like to seek for AIML tech SME services in FY25 and beyond to continue to grow and mature this technical foundation into the core platform and testing-bed for SEC to obtain the cutting-edge and indispensable technical know-hows of AIML technologies and risks, implementation feasibility analysis, model management, and meaningful technical assessment of the third-party owned or service-driven AIML products for the Agency.

2.1.1 Technical Foundation Operation, Maintenance and Enhancement

OIT AIML Group would like to maintain the existing AIML tech foundation’s capacity, small, but robust, operational, and adoptable, easily scalable, and integrable with AWS services or other vendor or third-party services. To achieve this goal, we are looking for tech SME skills and experience in Cloud computing, performance optimization, docker and containerized modular solution deployment; Airflow eco system engineering, GPU configuration and tuning, model deployment, model continuous monitoring, GenAI service deployment and management to support the following, but not limited to the below main tasks:

1) Continue to identify and consult the main areas to promote the OIT AIML tech foundation capacity growth, model risk management and operation efficiency.

2) Provide technical assessment, bench-mark testing, model explainability analysis, model monitoring analysis etc, applicable to the AIML services in use/in evaluation at the Agency:

such as AWS Sagemaker, AWS BedRock, MicroSoft Copilot etc.

3) Integrate GPU EC2 Node to support better modeling performance and cost control.

4) Enhance Airflow efficiency to automate the production monitoring and notification.

5) Automate Airflow post deployment testing and report on system metrics.

2.1.2 Pilot Product Operation, Maintenance and Enhancement

OIT AIML Group would like to continue to operate, maintain, mature, and enhance the pilot AIML products deployed on OIT AIML Tech Foundation: namely, the Comment Analysis & Review System (CARS); and Artificial Intelligence Disclosure Analytics (AIDA).

2.1.2.1 Comment Analysis & Review System (CARS) O&M and Enhancement

The Comment Analysis and Review System (CARS) project was an OIT-CF collaborative initiative starting in FY22 in response to the Chair’s directive to immediately enhance analytic capacities to facilitate the SEC’s rule-making mission and accelerate the comment analysis process.

In FY23, the project team has completed the primary build of the CARS CF Minimum Viable Product (MVP) and obtained the Authorization to Operate (ATO). In FY24, SEC rolled out this Comment Letter Review and Analysis System (CARS) Minimum Viable Product into production; also, further explored and proof built some modeling features/solutions that should benefit the overall comment analysis. In FY25, OIT AIML group seeks for Tech SME services to achieve the below primary goals:

1) To maintain and operate CARS MVP build to enable its steadily increasing user adoption within

Divisions and Offices at SEC, to support their active rule-making comment analysis.

The main features delivered by CARS should include but not limited Comment Letters assignment, labelling, annotation, search and model-generated classifier and summarization.

In specific, CARS should be

Able to locate, review and annotate, and label comment letters from a centralized location.

Able to assign, unassign, claim, list the review status of a comment letter.

Able to create, update, manage customized labels of a comment letter.

Able to enter review notes of a comment letter, and summarize, share and download the review notes.

Able to annotate a comment letter and compile the annotation summary.

Able to model, train, refine the commenter type based on the predefined labels.

Able to search across comment letters with key words and full text.

Able to assist rulemaking reviewers/writers with the comment summary preparation.

Note: (See CARS MVP Requirement Specification for details)

2) CARS solution should create the centralized CARS data repository residing in ACES EDW 2.0

To completely sync with upstream data source Comment Letter Log intake system (CLL 2.0) from ACES cloud; which is the system of record of comment letters; to capture the comment letters, relevant documents and hyperlinks at sec.gov.

To make comment letters accessible to Divisions and Offices in a searchable and downloadable format as soon as they are approved by Office of the Secretary (OS).

3) In FY25, CARS product plans to roll out the modeling features/solutions that built in FY24 and are proven to be able to benefit overall comment letter analysis. The potential features to roll out to operate include but not limited to:

Generative Question and Answer RAG search and semantic matching.

Ability to tag and categorize comment letters by predicated/customized labels and annotate them.

Automated initial identification and grouping/classifications (e.g., topics, themes, submitters) Interactive dashboard with keyword search, category filters, summaries, analysis, trends Ability to automate the identification and grouping of statues and rules cited in the submitted comments and allow for manual modification.

2.1.2.2 Artificial Intelligence Disclosure Analytics (AIDA) O&M and Enhancement

The Artificial Intelligence Disclosure Analytics (AIDA) project was an OIT-IM and OIT-CF collaborative initiative starting in FY22 to leverage the Machine Learning algorithms and AI localized large language model (LLMs) to augment the non-compliance checks and content analytics. In FY23, three modular solutions have been built including AIDA-Certification; AIDA-Signature, and AIDA- Earning Call Transcript to assist IM/CF business to enhance the disclosure review efficiency of heavy-duty EDGAR filings. In FY24, the additional AIDA-Audit Report solution has been built by largely leveraging local LLMs and Retrieval Augmented Generation (RAG) technology while to assist Divisions and Offices in exploring the GenAI features and adoption. AIDA is expected to obtain the Authorization to Operate (ATO) in January 2025.

In FY25, OIT AIML group would like to continue to operate, maintain the innovative edge that AIDA offers to analyze in-depth EDGAR filings. The primary goals of AIDA O&M and enhancement include but not limited to the below:

1) Support the production operation and monthly delivery of AIDA certification and Audit Report to IM. The operation scope should include the end-to-end model pipeline maintenance via Airflow;

basic model registration, scanning, and risk control via MLflow model repository service; and basic user interfaces and visualization of model results built by using Streamlit APIs.

2) Trouble-shoot, fine-tune, and maintain the production algorithms to maintain the high model inference successful rate.

3) Conduct data/technology assessments to evaluate suitability of MVPs to scale for cross-divisional use cases in Disclosure Analytics primarily applicable to EDGAR Filings, which have the highest business values and best data/technology readiness.

4) To assist the further technical solution exploration and deployment for other business high priority use cases: such as name rule/portfolio for holding use case for IM.

2.1.3 AIML New Technical Know-How Exploration and Discovery

OIT AIML group would like to acquire the continuous Tech SME services in the innovative analytics and discovery area to promote the feasible and efficient adoption of the Artificial Intelligence and Machine Learning new features at the Agency. In FY24, OIT AIML group has deployed several localized Large Language Models with the basic risk management guardrails implemented on OIT AIML Tech Foundation within SEC ACES environment. These LLMs allowed the build and delivery of a couple of GenAI Retrieval Augmented Generation (RAG) enabled solutions such as AIDA-Audit Report; IM-Investment-Prospectus; and the enhanced Question and Answer solution for Comment Letter Analysis (CARS).

In FY25 and beyond, OIT AIML group plans to continue to closely follow up with the latest development of GenAI foundation model usage and related Large Language Model (LLM) and Natural Language Processing (NLP) landscape and technologies and seek for the Tech SME services to focus on, but not limited to the below tasks:

1) Continue to maintain the advanced Tech SME knowledge and know-hows to understand the Large Language Model (LLM) landscape and best practices to guide the Agency’s effective adoption of GenAI services and products.

2) Continue to research and build up the hands-on technical SME knowledge and know-hows in model deployment, integration, technical prototype solution delivery, model performance monitoring strategy of using AWS and other GenAI services (such as BedRock or Copilot) and to provide tech know-how support as needed to enable the Division and Office’s GenAI initiatives.

3) Continue to maintain the Tech SME knowledge and know-hows in more classical Natural Language Processing (NLP) technologies such as regular expressions(regex) and other ML models to support the growth of SEC’s unstructured data/document analytics.

4) Continue to explore the new GenAI RAG features and help build the Agency’s RAG feature roadmap: including the optimizing Retrieval and Generation strategy, prototyping GraphRAG use cases, and Vector DB technical implementation and tuning, and partnering with existing project/product teams to roll out the matured RAG features in enhanced search, unstructured text and content analytics applicable to EDGAR filings and other SEC documents with high business values.

5) Continue to operate and mature the modular RAG-based solutions developed in FY24 with the proven business values; with integrated and scalable AWS services, if needed.

2.2 OIT AIML Governance and Risk Management Framework

While SEC has been working to establish the AIML governance practices and the basic risk management framework, OIT AIML group seeks for Tech SME services to focus on the below two areas, but not limited to the two:

2.2.1 AIML Governance Services:

In FY25 and beyond, we would like to have the AIML Tech SME services support the below main tasks in AIML governance space:

1) Provide the technical consultancy, risk assessment, tech SME advice to the AIML governance bodies such as SEC AIML Steering Committee and Office of Chief AIML Officer (CAIO), and Division & Offices’ AIML use case stakeholders regarding the AIML use case inventory management, risk compliance implementation, third-party AIML services usage and vendor AIML services such AWS Beckrock and MicroSoft CoPilot.

2) Work with OIT Security Assessment Team, OIT Technical Review Board (TRB) to implement the basic AIML, especially GenAI risk management guardrails and integrate the controls in the existing SEC Software Development Framework (SDF) design requirement and workflow. The basic guardrails focusing on mitigating the model and technical risks include but not limited to using security group control to eliminate the localized LLM data egress; the mandatory model registration and continuous monitoring via MLflow service (will be illustrated further in the 2.2.2 section); and the mandatory application/prompt logging of AIML products/applications.

3) Provide the best practices in AIML code/data check-in and versioning in the centralized SEC Enterprise Gitlab as the model source code/versioning repository and Enterprise Data Warehouse (EDW2) as the AIML data repository.

2.2.2. AIML Risk Management Framework Establishment

Since late FY23, OIT AIML group has experimented to establish a framework that can operationalize the lifecycle management of the deployed Artificial Intelligence and Machine Learning models. An open-source Machine Learning Operations framework- MLFlow framework has been built in the lower environment and has test-registered the existing models utilized in AIDA and CARS projects.

In FY25 and beyond, OIT AIML group seeks for Tech SME services to continue to operate, maintain and mature this MLflow service framework and other continuous model monitoring as well as to support the model management efficiency by focusing on, but not limited to the below tasks:

1) Deploy the MLflow framework in production via AIDA SDF process and implement MLflow model repository/service to fully track the model registration, execution, and to build an enforceable and auditable AIML model run-time service/repository. This MLFlow should standardize the packaging of ML code, workflows, and artifacts, and facilitate end-to-end AI/ML solution deployment including both SaaS and OSS LLM models on-boarding; at the same time, offers a centralized model run-time service and meta-data store, APIs, and UI to ensure the smooth model registry and monitoring.

2) Experiment and deploy the ML model explainability features.

3) Experiment and assess the recently available AWS MLflow new services for potential adoption.

4) Collaborate with other OIT or Div/O user groups to develop the best practices in model management (especially in GenAI implementation/initiatives) and operation at SEC and to provide the practical guidance in the main aspects of model management including model evaluation and validation; deployment and tuning; maintenance and updates; and continuous model performance improvement and monitoring based on business feedback and inputs.

5) Conduct market research, exploration to help establish model risk management workflow by either self-building or leveraging the third-party commercial products based on AIML Steering Committee’s direction and decisions.

2.3 OIT AIML Tech SME Consultancy and Collaboration

OIT AIML Tech SME services are expected to provide two supporting models to provide the technical know-hows to Division and Offices to enable their advancement in exploring AIML capacities.

1) Partner with Division and Offices to provide the analysis to identify the technology and data readiness, design and implementation feasibility, feature roadmap planning, prototype analysis, and model registration and monitoring. OIT AIML group is expected to work with major stakeholders from OCDO, EXAM, CF, IM, ENF, DERA etc to support their on-going new AIML initiatives.

2) Partner with Division and Offices to build a small-scoped end-to-end solution if needed, such as to support EDGAR Business Office (EBO)’s pursuit to use GenAI to streamline their EDGAR FormID registration process.

3 Estimated Period of Performance:

04/21/2025-04/22/2026 (as the base year; with optional years to award)

4. On-site or Off-site:

Off-site.

5. Tasks and Deliverables:

No.

Tasks and Deliverable(s)

1 Integrated Project Schedule 2 Risk, Issue, Action (RIA) Log(s) 3 Successful deployments of product releases 4 Source codes, tech design documents checked in SEC Gitlab (ACES)

5 Requirements and feature workshops and artifacts (including but not limited

to) stakeholder presentations, demos and/or elicitation sessions

Service Delivery Framework (SDF) artifacts as required by OIT including but not limited to the updated Security System Plan (SSP), Standard Operation Procedures (SOP) and Detailed Technical Design Document (DAD).

∗ These Tasks and Deliverables are in addition to the specific ones listed above.

6. Tasks and Deliverable delivery timing?

The SEC COR and the Contractor will work out the timing of the Tasks and Deliverables outside of this document.

7. How will this be funded and What CLIN Type will be used (e.g. TM, FFP, Etc.)?

Firm Fixed Pricing (Estimated April 22, 2025- April 21,2026 as the base year with optional years award)

8. What LCATs and the LOE for all?

Seek for 1 AIML Tech Lead; 1 Primary Cloud Engineer; 1 GenAI SME, 1 AIML Feature Presentation Lead/Developer; 1 AIML Governance SME

9. How will performance be evaluated?

1. Successful delivery of new releases in 6–8-week interval or upon business requirements.

2. Successful delivery of the assigned solutions based on the agreed Level of Efforts (LOEs) on time within budget.

3. Weekly status reporting:

a. To track the status, manage deliverables, and address issues.

4. Customer Satisfaction:

a. To manage customer expectation by soliciting the feedback on performance on the regular basis; and

b. To obtain the sign off on the User Acceptance Testing (UAT)

5. SLA compliance:

a. To meet with the Contract operation SLAs agreed upon with stakeholders.