GenAI-draft PWS_RFI.docx

Draft Work Statement

SPS

Generative AI (GenAI) Services DRAFT v1.0 11/08/2024 Information Technology Services (ITS) Gen AI Support Services

Table of Contents

INTRODUCTION

The Office of the Comptroller of the Currency (OCC) is a Bureau of the Department of the Treasury responsible for ensuring that national banks and federal savings associations operate in a safe and sound manner, provide fair access to financial services, treat customers fairly, and comply with applicable laws and regulations.

In support of this mission, the Office of the Comptroller of the Currency (OCC), Information and Technology Services Division (ITS) has been evaluating and implementing the use of Generative Artificial Intelligence (GenAI or AI) and Machine Learning (ML). The capabilities and advantages offered by these technologies to enhance the value, speed, and efficiency of agency business processes has led to the implementation of proprietary tools and models in Azure OpenAI and Machine Learning platforms.

Because of the advantages created by AI/ML in helping the OCC to fulfill its mission—including advanced risk assessments, fraud detection, and improved regulatory oversight through data-driven insights--the agency is seeking to mature and develop its AI/ML capabilities.

BACKGROUND

ITS has developed OCC-Chat, a text-to-text AI tool similar to ChatGPT, using Azure OpenAI and the GPT-4o model. This tool is scheduled to move to production in December of 2024. Additionally, ITS oversees several AI/ML applications currently in the prototype phase, such as OCC-infoAssist, Document Insights, Comment Analytics, and Issue Analytics, which are set to transition to production and utilize technologies from Microsoft, AWS, and Open Source. The OCC aims to scale its AI/ML infrastructure to support additional models, incorporate innovative functionalities, and improve the generated information to ensure robust and reliable AI-driven solutions.

OCC AI/ML efforts are currently split between two teams - the OCC Solutions Lab and the GenAI Service Delivery Team. The OCC’s Solutions Lab assembles prototypes of Gen AI tools. Once prototypes are approved for broader use, the GenAI Service Delivery Team is responsible for implementation, operational maintenance, decommissioning (Section 4.2), and tailoring them for specific uses cases in various Business Units (BUs) (Section 4.3). The OCC’s tools currently use technologies from Microsoft and AWS. Over the next five years, there are plans for substantial growth and integration of these and other developing technologies.

SCOPE

The scope of this contract includes specialized support services for AI/ML production systems to assist ITS in the ongoing management, optimization, and maintenance of advanced artificial intelligence technologies and decommissioning of tools within the purview of the GenAI Service Delivery Team.

The scope of work for Gen AI support includes:

· Program Support to mature AI/ML initiatives.

· GenAI/ML Implementation and Operational Maintenance technical services.

· GenAI Capability Tuning to improve the utility of AI for OCC business functions.

TASKS

Program Support The OCC requires contractor support for enhancing policy, processes, and procedures for AI/ML tools. The Contractor shall analyze the OCC environment, make recommendations for OCC policies within existing Federal guidelines, and make recommendations for continuous improvement and best practices. The contractor shall provide advice and recommendations on where and how the OCC can or should mature their use of AI.

· Work with OCC Security to ensure AI information retrieval is secure and free from unwanted penetration.

· Collaborate with OCC IT and business units and other OCC stakeholders, such as the Solutions Lab, to ensure AI/ML program alignment with overall organizational goals and IT infrastructure. The contractor shall coordinate with external vendors and partners, as necessary, for AI/ML tool integration and support.

· Research and stay apprised of industry changes and trends and make recommendations to OCC ITS on a variety of topics related to growth and expansion of the AI/ML programs and tools.

· Develop recommendations for a monitoring program for all AI/ML tools in ITC. The contractor shall then implement approved monitoring.

· Provide user support services to the OCC community to improve AI responses and provide assistance supplying AI requests.

· Develop a Continuous Improvement Program Plan and implement the planned improvements to enhance program effectiveness, including a process for regularly reviewing and assessing the AI/ML program and processes for improvement opportunities. The plan will be created in collaboration with the OCC and should blend the needs and objectives of the OCC with industry best practices and movements in AI technology and usage. To stay current with industry movements, the Continuous Improvement Program Plan will be assessed and updated annually.

Table 1 - Program Support Deliverables

GenAI/ML Implementation and Operational Support

· Make recommendations for AI and ML Architecture to integrate tools with AI, Large Language Models (LLMs) (i.e., LLAMA and ChatGPT-4) and implement approved architecture.

· Manage changes to AI/ML systems and processes, including documentation, approval, and implementation of changes. The contractor shall communicate these changes to relevant stakeholders prior to implementation and as directed by the COR.

· Regularly update AI/ML offline model software and tools to the latest versions, conduct necessary API updates and patches, and apply security updates to maintain system integrity. These updates need to be tested prior to implementation to ensure the updates/installations will be completed successfully, perform in accordance with the host platform’s vendors documentation and specifications, and does not introduce errors. This includes ensuring the OCC’s AI/ML tools remain compliant against any updates the host platform undergoes, such as those from Azure, and supporting system version roll-back as needed. While Azure may be responsible for updates to the underlying infrastructure and platform services, the contractor is responsible for updates and maintenance of the specific AI/ML applications and tools deployed on the Azure (or any other host) platform. Comment by Helman, Casey W: Prior to implementing or after? Comment by Smith, Stephen (Contractor): Made this “prior to implementation” as it’s usually the much preferred. Venkat, you may be able to confirm if this is feasible given what you know technically about the system architecture. Comment by Bangalore, Venkatakrishna: Yes, we can handle this ahead of implementation since we are informed in advance when new versions will become available and when the old version will be deprecated by the service provider. Comment by Helman, Casey W: The OEM vendor or this contractor?

· Promptly identify and resolve issues or errors in AI/ML systems. Additionally, the contractor shall document and track the issue resolution progress and outcomes.

· Work with the OCC to implement approved recommendations identified in Task 4.3.

· Work with the OCC and its contractors to decommission AI-related tools that have become obsolete or have reached the end of life.

· Develop and maintain documentation. Due to the technical nature of this task, the OCC envisions this be incorporated in a system operation and maintenance handbook. Other documentation is created in the other tasks. The operation and maintenance documentation shall cover configuration and utility updates, report/dashboard creation, report/dashboard generation and maintenance, baseline creation as applicable, and guidance on performing interpretive analysis of data. Updates to the handbook are most likely driven by GenAI/ML product ingestions, changes to AI management processes, AI system configurations, and other elements impacting operation and maintenance. The contractor is expected to use their expertise to formulate and propose an operation and maintenance handbook that contains information needed for ongoing maintenance. The handbook shall become the property of the OCC to be used in continuity of operations, knowledge transfer, and by successors to this contract.

Table 2 - GenAI/ML Implementation and Operational Support Deliverables

AI Capability Tuning The goal of this task is to enable the GenAI system(s) to return information that supports the business objectives of the OCC. This involves training the AI through pre-generative training to teach the AI about the OCC’s business operations, terminology, mission objectives, and other aspects that support the OCC’s needs. The contractor will leverage their expertise to recommend aspects and techniques that will produce the most effective results. The determination of effective results will be in collaboration and in concurrence with the OCC. The most effective methods of natural language inquiries will need to be captured and communicated to the OCC user community.

Below are anticipated performance actions of the contractor; however, it is not an exhaustive list of all activities. Meeting the overall goals of the task is the objective, and the contractor is expected to bring their expertise and execute as needed to support the goal. The contractor will:

· Design and execute Prompt Engineering on the AI systems. This may include the assembly of Large Language Models of terms, phrases, and questions established as human language that will give the most accurate output, and customizing the tool using OCC proprietary data.

· Assess the AI Pre-Generative Report construct and incorporate updates and improvements on the information. These updates will be influenced by a variety of stimuli such as movements in the industry on measuring AI capability improvement, needs by the AI program to better understand and interpret the results, and external stakeholders that need specific information to perform their business determinations and functions regarding AI use in the OCC.

· Foster the use of GenAI information retrieval by the OCC user community through user education. Encapsulated here as “user education documentation,” this documentation may take the form of user guides, job aids, Knowledge Base Articles (KBAs), instructional videos, and other forms deemed appropriate for user education. In collaboration with the OCC, the contractor shall leverage their experience to propose a set of user documentation that is tailored to the OCC user community and aligns with industry practices. The type, number, and specific content of the documentation will be outlined in the user documentation plan.

Table 3 - AI Capability Tuning Deliverables

Knowledge Transfer The contractor shall schedule and conduct knowledge transfer sessions with the OCC and other contractors as needed and prepare supporting material. The goal of the knowledge transfer is to prepare selected OCC and contractor personnel in the operation, maintenance, and data analysis of AI utilities on a proactive and ongoing basis.

Background for the Curious: I think we can at least establish that 2 times a months the contractor will need to address and perform some degree of knowledge transfer just to keep the ‘ongoing’ concept visible in the forefront.

‘How often’ has some dependency on upon the methods and artifacts proposed by the contractor as elements required in the Knowledge Transfer Plan. The Knowledge Transfer Plan is also a formal deliverable that must be deemed acceptable and approved by the OCC. Comment by Helman, Casey W: would it be more appropriate to say at least one a month, with the exact frequency/cadence to be established in the knowledge transfer plan? Comment by Bangalore, Venkatakrishna: We should also highlight this as needed and maintain a monthly cadence.

An acceptable Knowledge Transfer Plan will contain or reference:

· The methods that will be employed to perform knowledge transfer.

· The supporting artifacts produced for knowledge transference.

· The schedule for knowledge transference.

Table 4 - Knowledge Transfer Deliverables

Stakeholder Communications The contractor shall provide communications to keep stakeholders informed on key activities related to the AI system(s). These may include communications such as planned maintenance, notifying stakeholders of major evolutions of AI learning, updates that could potentially impact Security, and AI results that are either reported or detected that could lead to the dissemination of misinformation. Communications may take the form of emails or news clips provided to the OCC communications team for inclusion in newsletters.

STAFFING

The contractor shall propose a staffing plan that would meet the requirements in this work statement. The OCC recommends that a backup to the primary resource is also proposed, to ensure continuity of services should the primary resource be unavailable for an extended period. To ensure readiness, the backup resource should also be submitted for the security onboarding process and participate in ramp-up activities at the outset of the contract.

GenAI Engineering Comment by Helman, Casey W: Suggest also mapping to the task area for which this skill is applicable. Comment by Smith, Stephen (Contractor): Mappings provided.

Focus Task Sections: 4.2, 4.4

· Strong background in machine learning and deep learning algorithms.

· Proficiency in Python, TensorFlow, and PyTorch for developing AI models.

· Ability to design and implement scalable and efficient AI systems.

· Advanced knowledge of natural language processing for text generation tasks.

· Familiarity with computer vision and image generation using AI.

· Skills in data preprocessing and feature engineering for AI model training.

· Strong understanding of neural network architectures and optimization techniques.

· Experience in deploying AI models into production environments.

· Ability to stay updated with the latest advancements in generative AI research and incorporate them into work.

GenAI Tuning Specialization Focus Task Sections: 4.3, 4.4

· Utilize advanced machine learning techniques to develop and train generative AI models.

· Collaborate with cross-functional teams to prepare and preprocess large datasets for model training.

· Research and implement cutting-edge algorithms and architectures for generative AI applications.

· Optimize model performance and scalability for real-time inference and deployment.

· Stay current with industry trends and advancements in generative AI technology to drive innovation.

· Experiment with different hyperparameters and model configurations to improve generative AI model quality.

· Provide technical expertise and guidance to support the integration of generative AI solutions into various products and services.

Technical Writing/Training Focus Tasks Sections: Global

· Experience conveying complex technical execution of AI utilities in simple terms to other personnel, exhibiting good communication skills.

· Experience developing user and technical documentation, preferably leveraging visual, audio, and textual media.

PLACE OF PERFORMANCE AND BUSINESS HOURS

Remote work on this contract is permissible, though all staff supporting this contract must be in the United States. Remote work from international locations is prohibited. Although the majority of the work will be performed remotely, the OCC may request an on-site presence at any OCC facility to perform the described services of this Order. Should the contractor choose to meet at or work from OCC HQ, Llocal travel and parking costs at the OCC HQeadquarters is not separately reimbursable to the Contractor.

The OCC’s core business hours are 10:00 a.m. to 3:00 p.m. ET. To avoid interruption to OCC business activities, the contractor will also need to perform some services during non-core business hours and weekends. This includes support and testing for software updates, installations, security software patches and other activities that have the potential of interrupting OCC business. Non-business hour work will be requested by and coordinated with the OCC as needed to maintain a high level of service across the OCC.

Government personnel observe the following Federal holidays and OCC will be closed:

* If the date falls on a Saturday, the Government holiday is the preceding Friday. If the date falls on a Sunday, the Government holiday is the following Monday.

In addition to the days designated as holidays, the Government observes the following days:

· Any other day designated by Federal Statute

· Any other day designated by Executive Order

· Presidential Inauguration Day

· Any other day designated by the President’s Proclamation Performance Requirements Below are the performance requirements for this work statement.

Table 6 – Performance Requirements

Improvement, as defined by the report construct and incorporated in the QASP

Deliverable Quality Assurance Process The PWS specifies deliverables which require formal review. All other deliverables are informal deliverables not requiring review, unless otherwise specified. All deliverables (formal and informal) will be submitted to the COR via email. Final versions of all deliverables shall be saved in a designated repository associated with the task order.

Formal Review:

The Contractor must produce two distinct versions of documents requiring review:

1. The draft version (by the date specified in the deliverable table)

2. The final version (in accordance with the process below)

The COR will review the draft version within 10 business days and may add other OCC reviewers as they deem necessary. The COR will send comments and changes back to the contractor in writing. The COR is responsible for delivering clear, concise, and actionable feedback to the Contractor. The Contractor will incorporate those requested changes into the final version of the deliverable. If the COR requires more time to review the draft deliverable, he/she must document, via email, the date by which comments will be provided. If the deadline passes without the COR providing written comment, the draft version of the document will automatically be deemed acceptable, and the Contractor shall finalize the document for delivery. The Contractor will have five (5) business days to revise the draft version of a deliverable after receipt of comments. If the OCC has taken more than 10 days to review the draft, the same extension is afforded to the Contractor to finalize the deliverable. Comment by Helman, Casey W: This is language from a different task order. Please insert a timeframe appropriate to your BU. Comment by Smith, Stephen (Contractor): @Bangalore, Venkatakrishna I’m familiar with this kind of deliverable acceptance language in Section 8 (though unsure of it’s origins), but it’s somewhat standard. I do suggest though that you read it carefully as it applies very specific timelines for both the OCC and contractor, with auto-acceptance being the default in some cases if OCC responses are delayed. I suggest you make sure you are comfortable with it - Casey can probably provide deeper insights into any questions. Comment by Bangalore, Venkatakrishna: To maintain consistency across the projects. I am ok keeping it as-is

The COR is responsible for the review and acceptance of all formal deliverables. The OCC intends to conduct inspections as described in this section, however, this section shall not limit the OCC’s inspection rights set forth in Federal Acquisition Regulation (FAR) 52.212-4(a). The OCC may modify or refine the inspection process in subsequent option periods, if necessary.

Remediation of Unacceptable Deliverables The OCC will require the Contractor to re-work and re-submit any final version of a deliverable that are deemed unacceptable. An "unacceptable" final version is a document that does not accurately and completely reflect the revisions identified by the COR in his/her review of the draft version. The COR will not accept as final any version of a deliverable that does not accurately and completely incorporate all changes identified by the COR in the draft deliverable. For the purposes of assessing the timeliness of a deliverable, unacceptable deliverables will be considered late if they are judged by the COR, upon submission, not to reflect all the changes identified during the draft review process.

Consistent with FAR 52.212-4, the Contractor shall correct deliverables without additional cost to the OCC.

APPENDIX A

image1.jpeg