36C10B26Q0286_1.docx

SUBJECT*

Abacus Visual Playbook Builder and Automation Orchestrator (VA-26-00041469)

GENERAL INFORMATION

BASE NOTICE TYPE

SET-ASIDE

Technology Acquisition Center

23 Christopher Way Eatontown NJ 07724

POINT OF CONTACT*

Contract Specialist Gary Throckmorton gary.throckmorton@va.gov 848-377-5269

PLACE OF PERFORMANCE

ADDRESS

POSTAL CODE

COUNTRY

ADDITIONAL INFORMATION

AGENCY’S URL

URL DESCRIPTION

AGENCY CONTACT’S EMAIL ADDRESS

EMAIL DESCRIPTION

DESCRIPTION

The purpose of this amendment is to advise interested parties that the RFI Responses Due Date has been extended.

Responses are now due no later than 3:00pm est, April 15, 2026

Request for Information

Title: Abacus Visual Playbook Builder and Automation Orchestrator

VA-26-00041469

1. Introduction

This Request for Information (RFI) is for planning purposes only and shall not be considered an Invitation for Bid, Request for Task Execution Plan, Request for Quotation or a Request for Proposal. Additionally, there is no obligation on the part of the Government to acquire any products or services described in this RFI. Your response to this RFI will be treated only as information for the Government to consider. You will not be entitled to payment for direct or indirect costs that you incur in responding to this RFI. This request does not constitute a solicitation for proposals or the authority to enter into negotiations to award a contract. No funds have been authorized, appropriated or received for this effort. Interested parties are responsible for adequately marking proprietary, restricted or competition sensitive information contained in their response. The Government does not intend to pay for the information submitted in response to this RFI.

The North American Industry Classification System (NAICS) for this requirement is 541512 with a size standard of $34.0 million.

2. Submittal Information:

All responsible sources may submit a response in accordance with the below information.

There is a page limitation for this RFI of 15 pages. The Government will not review any other information or attachments included, that are in excess of the (15) page limit. NO MARKETING MATERIALS ARE ALLOWED AS PART OF THIS RFI. Generic capability statements will not be accepted or reviewed. Your response must address capabilities specific to the services required in the attached PWS and must include the following:

a. Interested Vendors shall at a minimum, provide the following information in the initial paragraph of the submission:

Name of Company Address Point of Contact Phone Number Fax Number Email address Company Business Size and Status For VOSB and SDVOSBs, proof of SBA certification. SDVOSB/VOSB vendors that have not been certified by SBA should not respond to this notice.

NAICS code(s) Socioeconomic data Unique Entity Identifier (UEI) Number Existing Contractual Vehicles (GWAC, FSS, or MAC)

b. Provide a summary of your capability to meet the requirements contained within the attached “Draft Statement of Objectives” for the following areas:

i. Describe your organization’s experience and capabilities as they relate to the following. Include specific project references, case studies, or contract examples where available:

(1) Contractor developed web application and delivery for federal agencies, with emphasis on VA Medical Device Virtual Local Area Network (Vlan) healthcare environments with emphasis on Visn 23 Healthcare Technology Management (HTM) usability.

(2) Systems integration involving Red Hat Ansible Automation Platform, Red Hat OpenShift, and enterprise infrastructure components under a single Prime contractor accountability model.

(3) Coordination of multi-vendor professional services engagements (e.g., Red Hat Professional Services, HCI vendor professional services) as subcontractors or directed partners. Address how you would manage the infrastructure dependency chain where application deployment cannot begin until platform installation is complete, and how you would handle accountability when integration issues span vendor boundaries.

(4) Development and deployment of containerized applications in air-gapped (dark-site) environments, including release bundle assembly, offline container registry management, and air-gapped CI/CD pipelines.

(5) Building user interfaces for non-technical user populations who lack scripting, command-line, or automation background.

(6) Experience with Ansible module or collection development, particularly for Windows targets using WinRM.

ii. Provide a narrative describing your proposed technical approach to the following areas. The Government is particularly interested in responses that demonstrate understanding of the system’s novel integration challenges rather than generic capability statements.

(1) Dark-Site Delivery and Sustainment Strategy. Describe your approach to delivering and maintaining containerized applications in an air-gapped environment. Address how release bundles will be assembled, how container images will be mirrored to internal registries, how offline vulnerability databases will be maintained for CI/CD scanning, how Ansible collection updates will be distributed through Private Automation Hub, and how future updates (including UBI base image security patches) will be packaged and delivered. Identify the specific tooling you would use. (This contract is scoped just for VISN 23 Healthcare Technology Management (HTM) and does not need to accommodate enterprise-wide compatibility for VHA.)

(2) Technology Stack and Container Strategy. Identify the development frameworks, languages, and runtimes you would propose for each contractor developed component (Abacus Application frontend and backend, Asset Ingress Service, abacus.retrieve Ansible collection, playbook generation logic). Explain how your selections align with Red Hat UBI requirements, OpenShift-native deployment patterns, and long-term maintainability. Address how the Government would maintain the system independently post-delivery.

(3) User Experience for Non-Technical Users. The primary users of this system are clinical site administrators who are operationally busy, have limited or no automation experience, and have no scripting or command-line proficiency. Describe your approach to building a complex web application (with concepts such as project/role/card hierarchies, editable and stable environments, and a content storefront with provenance tracking) that is navigable and productive for this population.

(4) Cybersecurity and Software Supply Chain. Briefly describe how you would maintain cybersecurity best practices across the software supply chain in a dark-site deployment. Address container vulnerability scanning in disconnected environments, Software Bill of Materials (SBOM) delivery and maintenance, and your approach to security patching when base image updates must be transferred physically.

iii. Provide a rough order of magnitude (ROM) for the following. The Government understands these are preliminary estimates subject to refinement during a formal solicitation. ROM estimates will not be binding.

(1) Contractor will propose all remaining technologies and include all applicable licenses and rough order of magnitude of annual cost or one time cost where applicable

(2) Estimated timeline for MVP delivery (the Performance Work Statement anticipates 12 months)

(3) Estimated level of effort (labor categories and approximate hours or FTEs)

(4) Estimated cost range for the full 12-month period of performance for MVP and separate cost schedule for the 4 option year Service Level Agreement (SLAs)

(5) Whether your organization would self-perform all software development or use subcontractors, and if so, for which components.

c. Corporate experience or expertise in performing these services and specific examples or references. Specific examples or references provided must include the agency, point of contact, dollar value, and contract number.

d. For SDVOSB/VOSB vendors: Your company’s intent and ability to meet the set aside requirement in accordance with VAAR 852.219-73 (JAN 2023) (DEVIATION) VA Notice of Total Set-Aside For Certified SDVOSBs and 13 CFR §125.6, which states the contractor will not pay more than 50% of the amount paid by the government to the prime for contract performance to firms that are not certified SDVOSBs listed in the SBA certification database (excluding direct costs to the extent they are not the principal purpose of the acquisition and the SDVOSB/VOSB does not provide the service, such as airline travel, cloud computing services, or mass media purchases). When a contract includes both services and supplies, the 50 percent limitation shall apply only to the service portion of the contract.

Your response shall include information as to available personnel and financial resources; full names of proposed team members and the PWS requirements planned to be subcontracted to them, which must include the prime planned percentage or the names of the potential team members that may be used to fulfill the set aside requirement.

e. Has the draft PWS provided sufficient detail to describe the technical requirements that encompass the software development and production operations support services to be performed under this effort.

____YES or NO_______ (if No, answer question f)

f. If “NO”, please provide your technical comments/recommendations on elements of the draft PWS that may contribute to a more accurate proposal submission and efficient, cost effective effort.

Responses are due no later than 3:00pm est, April 15, 2026 via email to Robert Robinson, Contract Specialist at Robert.Robinson7081@VA.gov and Christopher Minetti, Contracting Officer at Christopher.Minetti@VA.gov. Please note “Abacus Visual Playbook Builder” in the subject line of your response. Mark your response as “Proprietary Information” if the information is considered business sensitive. The email file size shall not exceed 5 MB.

Modification to a Previous Notice Modification to a Previous Notice

Modification to a Previous Notice

Draft Statement Of Objectives for: Request for Information (RFI)

Title: Abacus Visual Playbook Builder and Automation Orchestrator

VA-26-00041469

Department of Veterans Affairs Veterans Health Administration (VHA) Veterans Integrated Service Network 23 (VISN 23) Healthcare Technology Management / Biomedical Engineering

1.0 Background

1.1 Operational Problem

VISN 23 Healthcare Technology Management manages clinical workstations and servers across multiple geographically distributed medical facilities. These systems run specialized medical software with vendor-imposed constraints that prohibit agent-based management tools and restrict the remote management protocols available. The environment operates as an air-gapped (dark-site) network with no internet connectivity.

Currently, configuration management and software deployment across these endpoints is performed through manual processes or fragmented scripting, resulting in inconsistent configurations, slow deployment cycles, and limited auditability. The Government requires a system to be titled: “Abacus System” that enables non-technical clinical site administrators to build, share, and execute automation without requiring scripting proficiency, command-line expertise, or deep knowledge of the underlying automation platform.

1.2 Overview

The Abacus System shall leverage commercial platform components, but the integrated solution does not exist as a single commercial product. Respondents should understand that this effort requires contractor-developed software and systems integration, not solely configuration of an existing product.

The Contractor developed software shall include: a browser-based visual playbook builder (the Abacus System’s Web User Interface) with a card-based automation design interface, project/role/card hierarchy management, an editable/stable two-environment workflow model, and a storefront for cross-site content sharing; an Asset Ingress Service enabling browser-based file and folder uploads (which first compress to a single .zip file for storing) to S3-compatible object storage; a custom Ansible collection (abacus.retrieve) that executes on target hosts to download and unarchive/extract .zip assets from presigned S3 URLs during playbook execution; and playbook generation logic that traverses user-created project structures and produces executable Ansible playbooks.

The VA anticipates the following platform components will need to be leveraged (not developed) in order to meet all of VA’s requirements: Red Hat Ansible Automation Platform (AAP) for orchestration and execution; Red Hat OpenShift for container hosting; hyperconverged infrastructure (HCI) for compute and storage; and S3-compatible object storage for asset management.

No vendor currently offers a visual playbook builder that combines card-based Ansible automation design, presigned URL-based asset retrieval, a content storefront with provenance tracking, and dark-site delivery as a unified product. Respondents proposing an existing COTS product must demonstrate how that product satisfies the specific integration requirements outlined in this RFI to include this draft Statement of Objectives.

2.0 Scope

2.1 Objective

The Abacus System is a visual playbook builder and automation orchestrator that allows Government operators to construct, share, and execute Ansible automation through a graphical web interface. The system shall integrate commercial platform components into a unified dark-site solution with the following high-level capabilities. This procurement requires the hardware and hyperconverged infrastructure as part of this contract (see Attachment B for details):

1. Visual Playbook Building: A card-based web interface where users construct automation by selecting modules, configuring parameters, and arranging execution order through drag-and-drop or equivalent interactions, without authoring “Yet Another Markup Language” (YAML) directly.

1. Content Storefront: A centralized repository where published roles, playbooks, and bundled configurations are shared across sites with provenance tracking, version management, maintainer permissions, and site-scoped visibility.

1. Asset Management: Browser-based upload of files and folders (up to 12 GB), compressing assets to .zip before storage, then storing to S3-compatible object storage, with integrity verification and presigned URL-based retrieval and unarchiving/extraction during playbook execution on target hosts.

1. VM Provisioning: User-initiated virtual machine creation on existing Nutanix HCI infrastructure (Visn 23 currently has Nutanix for this VM provisioning workflow (See Attachment B)) through the web interface, with automated onboarding into AAP inventory and optional immediate playbook execution.

1. Multi-Site Operations: Eight (8) physical sites with site-scoped access control, Active Directory integration, and a ninth administrative site (VISN 23) which has cross-site access.

1. Dark-Site Delivery: All components operate in an air-gapped environment with no internet connectivity. Release bundles, container images, vulnerability databases, and Ansible collections must be transferable and installable offline.

2.2 Operational Viewpoint (OV)

The following OV-1 diagram depicts the operational concept of the Abacus System, showing how operators interact with the system, how automation flows from the web interface through AAP to target hosts, and how assets are retrieved directly from S3-compatible storage via presigned URLs during execution.

2.3 Key Architectural Points

1. AAP does not deliver asset payloads. AAP generates presigned URLs and passes them as execution variables. Target hosts shall be able to use the custom abacus.retrieve module (to be developed via this contract) to fetch assets directly from S3-compatible storage via HTTPS. This separates the control path (WinRM) from the data path (HTTPS to S3).

1. All application containers run on Red Hat OpenShift and must be built on Red Hat Universal Base Image (UBI). No non-Red Hat base images are permitted.

1. The Hyperconverged Infrastructure (HCI) platform must be Red Hat OpenShift-certified per the Red Hat Ecosystem Catalog, with demonstrated support for air-gapped OpenShift installation and S3-compatible object storage.

1. Windows hosts are managed via WinRM with certificate-based authentication. No agents may be installed on managed endpoints. OpenSSH cannot be enabled on Windows targets.

2.4 Scale

The Abacus System shall support eight (8) physical sites, up to 50 projects per site (400 total), up to 1,500 managed hosts across all inventories, and at least 10 concurrent users. Individual projects may manage between 1 and 60 hosts. The system must support projects containing 15–35 roles, each with 20–60 actions. The host count of 1,500 is the entire VISN 23 medical device inventory. VA anticipates no more than 100 managed nodes of Red Hat Ansible Automation Platform licenses in the base year. Option years can quote additional license amounts. The onboarding cadence of new medical systems to the Abacus System is dependent on VA. VISN 23 will wait for a project upgrade to initiate the additional work needed to automate a new or existing system. Comment by Minetti, Christopher: Can we just say- VA anticipates no more than 100 managed nodes of Red Hat Ansible Automation Platform licenses in the base year, and an additional ……… in the option periods or something like that. Just to cement the quantities more. Is the onboarding cadence depending on VA or the contractor? Just asking because if it’s the contractor and they say they can do it faster, I assume we wouldn’t want to slow them down. Comment by Biddle, Adam J.: Agreed, stated as such and clarified that it is VA that is the bottleneck here, and we wait for new projects to occur and then plan to convert our inventory via attrition over several years, and therefore do not know the conversion cadence.

3.0 Technology Selection: Red Hat Ansible Automation Platform and Red Hat Openshift The Government has determined that AAP is the only automation platform that satisfies all operational constraints of this environment. Ansible is the only production-proven automation platform that is agentless by design, uses WinRM natively for Windows management with certificate-based authentication, provides comprehensive cross-platform coverage (70+ certified Windows modules), offers an enterprise orchestration platform (AAP) with stable REST API, and supports air-gapped content distribution through Private Automation Hub. Furthermore, this AAP requires Red Hat OpenShift as the certified deployment platform (AAP Operator) to include an OpenShift-certified HCI platform for dark-site deployment. This determination is based on the following six hard constraints, each of which independently eliminates one or more alternative technologies.

3.1 Operational Constraints

3.2 Alternative Technologies Eliminated

4.0. Referenced Documents

The following documents provide detailed requirements and are referenced throughout this RFI. These documents will accompany any subsequent solicitation:

5.0. Contractor Software Development Scope Summary

To assist respondents in understanding the scope and complexity of software development work, the following table summarizes the major functional areas and their approximate requirement counts from the forthcoming Performance Work Statement (PWS), which will be refined as a result of information derived from the RFI. This is not an exhaustive list but represents the primary capability domains.

Total approximate requirement count: ~760 items. The PWS shall be structured to be comprehensive and auditable.

Department of Veterans Affairs Page 16

ATTACHMENT A

Red Hat Professional Services

Ansible Automation Platform Integration for:

Abacus Visual Playbook Builder and Automation Orchestrator

Department of Veterans Affairs Veterans Health Administration (VHA)

Document Purpose This document serves as Attachment A to the Performance Work Statement (PWS) for the Abacus Visual Playbook Builder and Automation Orchestrator acquisition. It defines the scope and requirements for Red Hat Professional Services engagement.

This document covers the interconnected technical areas requiring Red Hat expertise:

1. VM Provisioning and VLAN Management — API-driven infrastructure automation across multiple Nutanix sites

1. Windows Certificate Authentication — Secure, scalable authentication for managing Windows hosts

1. Certificate Management Workflows — Git-based certificate lifecycle management with four (4) distinct workflows

1. Presigned URL Workflow — S3 asset retrieval for playbook execution without direct AWS credentials

1. Real-Time Job Output Streaming — WebSocket integration for live job status in external UI

1. AAP Infrastructure and Self-Management — Operational best practices for running AAP

Abacus: The Orchestration Application Abacus is the external application that orchestrates all user interaction and workflow coordination. Understanding Abacus's role is essential context for the AAP integration design.

Abacus Responsibilities

1. User interface for all VM provisioning and configuration requests

1. Determines which AAP job templates to invoke based on user actions and site context

1. Specifies which playbooks to execute (via Git commit reference)

1. Manages S3 asset references and coordinates presigned URL workflow

1. Handles user authentication and authorization before calling AAP APIs

1. Displays real-time job execution output streamed from AAP

1. Retrieves workstation onboarding script from Git AAP's Role AAP serves as the automation backend — it does not provide user-facing interfaces for end users. All user interaction happens in Abacus; Abacus calls AAP APIs; AAP runs playbooks against target infrastructure.

Key Architectural Principle Abacus orchestrates, AAP executes. Users interact with Abacus; Abacus calls AAP APIs; AAP runs playbooks against target infrastructure.

VM Provisioning and VLAN Management This section describes the requirements for automating Windows VM provisioning and network management across multiple Nutanix sites.

Multi-Site Architecture The Government has eight (8) geographically distributed sites, each with its own Nutanix Prism Element cluster. A single AAP installation serves all sites with strict security boundaries between them.

Multi-Site Security Model The primary security requirement is strict site isolation. Users at Site A shall not have access to, view, or modify resources at Site B — even through misconfiguration.

Implementation approach:

1. One (1) AAP Organization per site (e.g., Minneapolis, Omaha, Fargo)

1. Organization-scoped Nutanix credentials (each credential can only access its site's cluster)

1. AD group to AAP Team mapping for RBAC

1. Site-specific VLAN configuration files in Git (vars/sites/{site}_vlans.yml) Physical Sites:

Fort Meade, Fargo, Sioux Falls, Saint Cloud, Minneapolis, Omaha, Iowa City, Central Iowa Logical Site:

VISN23 (administrative; no dedicated cluster; staff can work across all physical sites) Each physical site has its own Prism Element cluster and AD security groups. VISN23 staff are granted cross-site access via AD group membership.

VLAN/Network Configuration Support VLAN and network configuration management requirements are specified in Category LV [LV1-LV22]. The implementation mechanism (Abacus-native functionality, AAP workflows, dedicated service, or equivalent) is at Contractor discretion.

Red Hat Scope for VLAN-Related Work:

Red Hat's involvement in VLAN management is limited to technical guidance and integration support if the Contractor selects an AAP-based implementation approach:

Critical Safety Requirement VLAN ID modification restrictions per [LV14] shall be enforced regardless of implementation mechanism. This prevents accidental disconnection of VMs using existing VLANs.

API-Driven VM Provisioning VM creation is triggered by Abacus via AAP's REST API — not through AAP's self-service portal. Abacus specifies the site, and the appropriate job template is invoked.

API Design:

1. One (1) Job Template per site (eight (8) total): minneapolis-vm-create, omaha-vm-create, etc.

1. Abacus determines which endpoint to call based on user's site

1. Site identifier implicit in endpoint (not passed as parameter) VM Creation Parameters:

VM Creation Workflow Steps The VM creation workflow shall execute the following steps:

1. Load VLAN configuration from Git for the target site

1. Perform "subnet math" to determine which VLAN the IP address belongs to

1. Validate IP is within usable range (not gateway, broadcast, etc.)

1. Check for IP conflicts (query existing VMs in Nutanix)

1. Create VM in Nutanix using specified template

1. Configure disks and CD-ROMs

1. Configure Windows hostname and network settings (mechanism at Contractor discretion per [L25])

1. Power on VM

1. Onboard VM for Windows authentication (see Section 3.6)

1. Add host to AAP inventory based on site, project, and group

1. Return result to Abacus Automated VM Onboarding After a VM is created and powered on, it must be onboarded into the Windows certificate authentication system before it can be managed by Ansible. This onboarding is an extension of the VM creation workflow.

Workflow continuation after VM power-on:

1. Wait for VM to complete initial Windows setup (sysprep completion)

1. Execute onboarding configuration (either via sysprep or initial connection)

1. Validate WinRM connectivity using certificate authentication

1. Add host to AAP inventory with correct Organization, Inventory, environment group, and component group memberships

1. Tag host with appropriate metadata (creation date, template used, etc.)

DRY Architecture Requirement All eight (8) sites shall use identical playbook code — a single Git repository with no per-site playbook copies. Site-specific differences shall be handled via:

1. Organization-scoped credentials

1. Job Template extra variables (site_name, nutanix_cluster)

1. Site-specific variable files in Git (vars/sites/{site}_vlans.yml) Benefit: Update playbook logic once; all eight (8) sites automatically receive the update.

VM Template Registry (Admin-Only) Infrastructure administrators need the ability to register new Windows VM templates so they become available for VM provisioning. This shall be restricted to admin AD groups only.

Windows Certificate Authentication This section describes the certificate-based authentication design for managing Windows hosts with Ansible.

Problem Statement The Government needs to manage a mixed environment of Windows machines where some are domain-joined and others exist in workgroups. Several critical constraints eliminate traditional authentication methods.

Why Traditional Methods Do Not Work:

Solution: Two-Certificate Architecture The solution is a certificate-based authentication system where Ansible uses WinRM over HTTPS with client certificate authentication. This approach uses two (2) separate certificates with distinct purposes.

Certificate 1: Windows Host Server Certificate

1. Purpose: Encrypts the WinRM communication channel (like HTTPS for a website)

1. Location: Each Windows host generates its own self-signed certificate

1. Private key: Never leaves the Windows host; never shared with Ansible

1. Validation: Ansible ignores validation (ansible_winrm_server_cert_validation: ignore) Certificate 2: Ansible Client Certificate

1. Purpose: Proves Ansible's identity; replaces username/password authentication

1. Location: Single CA-signed certificate from Venafi on Ansible controller

1. Private key: Stored on Ansible controller only

1. Public key: Distributed to all Windows hosts (Trusted People store)

1. Mapping: Windows hosts map this certificate to local ansible_admin user Key Design Decision: Why Self-Signed for Server Certs?

Using CA-signed certificates for each Windows host would require managing hundreds of individual certificates through the manual Venafi portal with annual renewals. Self-signed certificates provide the encryption needed without this operational burden. Security risk is acceptable because Ansible initiates all connections to explicitly configured IP addresses.

Git Repository for Platform Configuration All certificate materials, root CAs, and generated scripts are stored in a dedicated Git repository. This design provides several benefits:

1. No TLS trust dependency: Git access via SSH requires no TLS certificate validation

1. Version history: All changes are tracked with full audit trail

1. Single source of truth: One authoritative location for all certificate-related materials

1. Resilience: If S3 or other infrastructure becomes untrusted due to CA rotation, Git remains accessible Certificate Management Workflows Overview Certificate management is divided into four (4) distinct workflows, each with a specific purpose and scope:

Initial Host Onboarding Workflow When adding a new Windows host to Ansible management, an administrator performs an initial RDP session to execute a PowerShell setup script. This is the "moment of physical presence" to establish trust relationships.

The onboarding script performs these steps:

1. Generate self-signed certificate and configure WinRM HTTPS listener on port 5986

1. Import the root CA certificate that signed the Ansible controller certificate into Trusted Root store

1. Read Ansible controller certificate thumbprint

1. Create local ansible_admin user account with strong random password (generated locally, never transmitted)

1. Create certificate mapping: "when this certificate is presented, authenticate as ansible_admin" After onboarding, Ansible can connect using certificate authentication without passwords transmitted over the network.

Red Hat Deliverable: Bootstrap Script Generation Red Hat shall develop a job template that generates the PowerShell onboarding script. The script is dynamically generated by reading the current controller certificate thumbprint, public certificate, and signing root CA from the ansible-platform-config Git repository, then rendering the templates/onboarding-workstation.ps1.j2 template. The generated script is committed to generated/onboarding-workstation.ps1 in the same repository. This generation is part of Workflow 1 (Certificate Preparation & Staging).

Workflow 1: Certificate Preparation and Staging This workflow extracts certificate materials from a Venafi export, stages them in Git, and generates the onboarding script. It is a "safe" operation that does not touch any Windows hosts.

Trigger:

Admin uploads venafi-export.pfx (or ZIP containing PFX) to controller-certificate/ folder in Git, then runs the job template with the decryption password.

Steps:

1. Pull latest from Git repository

1. Detect file format (PFX directly, or ZIP containing PEM files, etc.)

1. Decrypt/extract using provided password: Private key → Store in AAP credential storage; Public certificate → Write to controller-certificate/public-cert.pem; Thumbprint → Extract and write to controller-certificate/thumbprint.txt; Signing root CA → If not already in root-ca/ folder, add it

1. Render onboarding-workstation.ps1 from Jinja2 template, embedding the root CA, thumbprint, public certificate, and logic for local user creation and certificate mapping

1. Write rendered script to generated/onboarding-workstation.ps1

1. Commit and push all changes to Git

1. Job completes successfully Red Hat Deliverable: Certificate Preparation Job Template Red Hat shall implement Workflow 1 as a job template that extracts certificate materials from the Venafi export, handles various export formats, securely manages the decryption password (not persisted after job completion), and commits all extracted materials to Git.

Workflow 2: Certificate Rotation to Existing Hosts This workflow pushes the new certificate to all managed Windows hosts. It requires careful design for reliability and rollback safety.

Pre-Conditions:

1. Workflow 1 has completed — all materials staged in Git

1. The old certificate is still valid (running 45-60 days before expiration)

1. AAP currently authenticates to Windows hosts using the old certificate

1. Both old and new credentials are available to AAP Safety Mechanism: Scheduled Task Rollback Before changing anything on a host:

1. Create a Windows scheduled task that will restore the old certificate mapping

1. Task triggers either on system reboot OR after a timeout (e.g., 10-15 minutes) Then make the change:

1. Apply new certificate mapping (new thumbprint → ansible_admin) Validate immediately:

1. Attempt to connect using the NEW certificate Based on result:

1. Success: Delete the scheduled task (disarm the rollback), mark host as rotated

1. Failure: Leave scheduled task in place, mark host as failed — the host will auto-recover when the task fires or on next reboot Red Hat Deliverable: Certificate Rotation Workflow Red Hat shall implement Workflow 2 with robust status tracking, automatic rollback mechanisms, support for re-runs against failed hosts, and clear operational visibility. Red Hat shall devise the optimal approach for dual-certificate handling during the transition period.

Workflow 3: Root CA Distribution to Windows Hosts This workflow pushes all root CA certificates from the Git repository to all managed Windows hosts. It is idempotent — Windows handles "already present" gracefully.

Purpose:

Ensure Windows hosts trust all internal root CAs. This is needed for presigned URL downloads from Nutanix Objects S3 endpoint and any other internal HTTPS services Windows hosts may need to access.

Trigger:

Admin runs job template after adding new CA to root-ca/ folder in Git.

Steps:

1. Pull latest from Git repository

1. Read all .crt files from root-ca/ folder

1. For each managed Windows host: Push all CA certificates to Trusted Root Certification Authorities store (Windows handles idempotency natively)

1. Track per-host status: success, failed, unreachable

1. Persist state so re-runs can target only incomplete hosts

1. Report summary Red Hat Deliverable: Root CA Distribution Job Template Red Hat shall implement Workflow 3 with the same status tracking and re-run support as Workflow 2.

Workflow 4: Execution Environment CA Trust Update This workflow updates the AAP Execution Environment's CA trust store without requiring a container image rebuild.

Purpose:

Ensure the Execution Environment trusts all internal root CAs. This is required for amazon.aws.aws_s3 collection connecting to Nutanix Objects (presigned URL generation) and any other collections/modules that connect to internal TLS endpoints.

Desired Outcome:

1. Pull latest from Git repository

1. Read all .crt files from root-ca/ folder

1. Transform to format required by EE (Red Hat to recommend appropriate format)

1. Update the EE's CA trust source via the recommended mechanism

1. Result: Collections use system CA trust, which now includes new CAs — without rebuilding the EE image Red Hat Deliverable: EE CA Trust Update Automation Red Hat shall implement Workflow 4 to update the Execution Environment's CA trust without requiring image rebuilds, reading from the Git repository as the source of truth. Red Hat shall recommend and implement the optimal mechanism for the OpenShift environment.

Administrator Certificate Renewal Workflow The annual certificate renewal should be as simple as possible for the administrator.

Desired admin experience:

1. Admin requests new certificate from Venafi portal, downloads the export (typically a ZIP containing an encrypted PFX)

1. Admin uploads the raw Venafi export (unmodified) to the Git repository (controller-certificate/ folder) via the Git web interface

1. Admin triggers Workflow 1 through AAP

1. Admin provides the PFX decryption password when prompted (this password shall be treated as sensitive data and not persisted after the workflow completes)

1. Admin reviews the generated onboarding script and staged materials

1. Admin triggers Workflow 2 to rotate existing hosts

1. If a new root CA was added, admin triggers Workflow 3 and Workflow 4 (or these may be automatically chained) Key requirement:

The administrator should not need access to Linux command-line tools, OpenSSL, or manual file manipulation. The goal is that an administrator can complete annual certificate renewal by performing two (2) primary actions: uploading a file and providing a password.

CA Certificate Updates When a new internal root CA is issued (for example, when the certificate authority used to sign the Nutanix Objects TLS certificate rotates), the administrator commits the new CA certificate file to root-ca/ in the Git repository.

Required propagation:

1. Workflow 3: Push new CA to all existing managed Windows workstations

1. Workflow 4: Update Execution Environment CA trust via ConfigMap/Secret

1. Workflow 1 (if re-run): Regenerate onboarding script (though new hosts only need the controller certificate's signing CA) Operational Visibility and Recovery All certificate management workflows involve changes across hundreds of hosts or critical infrastructure. Red Hat shall design these workflows with operational visibility as a first-class concern.

Requirements:

1. Clear reporting on which hosts were updated successfully, which failed and why, and which were unreachable

1. Support for targeted re-execution against only the hosts that need attention, without re-processing hosts that already succeeded

1. A mechanism for administrators to verify the current state of any host (which certificate thumbprint is it trusting? which CAs are in its trust store?)

Presigned URL Workflow for Asset Retrieval Overview Abacus needs a two-job AAP Workflow that:

1. Job 1: Receives a list of S3 asset references, generates presigned download URLs for each, and stores them for the next job

1. Job 2: Executes the actual playbook, with presigned URLs available as variables for download tasks Assets are ZIP files that may contain a single file or a folder structure when extracted.

Job 1: Presigned URL Generation Trigger:

API request from Abacus to AAP.

Input (via extra vars):

Abacus provides a list of S3 asset references, each containing:

1. asset_id: Unique identifier for the asset

1. s3_bucket: Bucket name (e.g., "abacus-playbook-assets")

1. s3_key: Object key path (e.g., "assets/my-asset-key")

1. s3_version_id: Specific version of the object Processing:

1. Iterate over the assets_to_fetch list

1. Use amazon.aws.aws_s3 collection with mode: geturl

1. Generate presigned URL with twelve (12) hour TTL (if configurable)

1. Build output dictionary mapping asset_id to presigned URL Critical TLS Requirement Job 1 MUST use the FQDN TLS endpoint for Nutanix Objects (e.g., objects.abacus.yourdomain.com) and MUST validate the TLS certificate. Presigned URLs contain whatever hostname was used to generate them. Target workstations trust the CA-signed FQDN certificate, not a static IP.

Output (stored for Job 2):

A dictionary (presigned_url_download_array) where keys are asset identifiers and values are the presigned download URLs.

Job 2: Playbook Execution Input:

1. Git commit reference specifying which playbook to execute (Abacus determines this)

1. presigned_url_download_array from Job 1 Processing:

1. Pull playbook from Git at specified commit

1. Execute playbook where abacus.retrieve tasks reference URLs via: {{ presigned_url_download_array['asset_id'] }}

1. abacus.retrieve downloads the asset, saves to destination, handles unzipping The abacus.retrieve Collection abacus.retrieve is a custom Ansible module/collection that the Prime Contractor shall build. It handles downloading assets via presigned URL, saving to a destination path defined in Abacus, and extracting ZIP contents.

This collection is outside the scope of Red Hat's development responsibility.

Black Box Contract From AAP/Red Hat's perspective:

1. They receive: A dictionary containing all S3 asset identifiers plus versions needed

1. They produce: A structured map of asset identifiers to presigned download URLs, available to the subsequent playbook execution job From Abacus's perspective:

1. Abacus handles: Batching all required assets into the input dictionary, authoring playbooks that reference the output map, and determining which playbook (Git commit) to execute

Real-Time Job Output Streaming Abacus needs to display real-time job execution output to users as playbooks run. This requires integrating with AAP's job output streaming capabilities.

Requirements

1. Stream job output in real-time to Abacus UI as tasks execute

1. Display task status (running, success, failed, skipped) as it happens

1. Show playbook progress without requiring page refresh or polling

1. Support multiple concurrent users viewing different job outputs Proposed Approach: WebSocket Integration AAP provides WebSocket endpoints for streaming job output. Abacus (or its backend) needs to connect to these WebSockets to receive real-time updates.

Expected workflow:

1. Abacus launches a job via AAP API

1. AAP returns job ID

1. Abacus establishes WebSocket connection to AAP for that job ID

1. AAP streams job events (task start, task complete, output lines) via WebSocket

1. Abacus UI renders events in real-time

1. WebSocket closes when job completes WebSocket Integration Guidance Red Hat shall provide documentation and guidance sufficient for the Prime Contractor to implement real-time job output streaming from AAP to the Abacus application UI. This guidance shall enable an external development team to properly implement the streaming integration.

Requirements: