ATTACHMENT_A_v2 _ final.docx

DOCX document 56 KB Posted

Attached to
7D20--Abacus Visual Playbook Builder and Automation Orchestrator (VA-26-00041469)
Solicitation number
36C10B26Q0286
Issued by

About this file

Attachment A is a Performance Work Statement defining the scope and technical requirements for Red Hat Professional Services engagement in integrating Ansible Automation Platform (AAP) with the Abacus Visual Playbook Builder and Automation Orchestrator system for the Department of Veterans Affairs Veterans Health Administration (VHA) Veterans Integrated Service Network 23 (VISN 23).

The statement outlines six interconnected technical areas requiring Red Hat expertise: (1) VM provisioning and VLAN management via API-driven infrastructure automation across eight geographically distributed Nutanix Prism Element clusters; (2) Windows certificate authentication using a two-certificate architecture (self-signed server certificates on Windows hosts and CA-signed client certificates on the Ansible controller) to support both domain-joined and workgroup machines; (3) certificate management workflows including certificate preparation and staging from Venafi exports, certificate rotation to existing hosts with rollback safety mechanisms, root CA distribution to Windows hosts, and Execution Environment CA trust updates via OpenShift ConfigMap/Secrets; (4) presigned URL workflow for S3 asset retrieval from Nutanix Objects with two-job orchestration pattern; (5) real-time job output streaming via WebSocket integration for displaying live playbook execution status in the Abacus UI; and (6) AAP infrastructure and self-management operational best practices. Red Hat's deliverables include installation and configuration of OpenShift and AAP, nine Organizations (one per physical site plus VISN23), Active Directory/LDAPS integration, credential management configuration, comprehensive playbooks and roles for all technical areas, PowerShell onboarding script generation, documentation including architecture diagrams and admin runbooks, WebSocket integration guidance for external developers, and knowledge transfer sessions. The engagement emphasizes a DRY (Don't Repeat Yourself) architecture where all eight sites use identical playbook code with site-specific differences handled through Organization-scoped credentials, job template variables, and Git-based variable files, ensuring single-point updates propagate to all locations.

View the file

Other files

File Type Posted
36C10B26Q0286 0003.docx DOCX document
RFI Follow-up _ Questions Answers_final.pdf PDF
36C10B26Q0286 0002.docx DOCX document
36C10B26Q0286_1.docx DOCX document
RFI_Statement of Objectives_v1.3_final.docx DOCX document
ATTACHMENT_B_v2 _ final.docx DOCX document
36C10B26Q0286.docx DOCX document

On GovTribe

Work with this file

  • Download the original file
  • Contacts named in this file
  • Similar government files
  • Ask GovTribe AI about this file

Text version

ATTACHMENT A

Red Hat Professional Services

Ansible Automation Platform Integration for:

Abacus Visual Playbook Builder and Automation Orchestrator

Department of Veterans Affairs Veterans Health Administration (VHA) Veterans Integrated Service Network 23 (VISN 23) Healthcare Technology Management / Biomedical Engineering

Document Purpose This document serves as Attachment A to the Performance Work Statement (PWS) for the Abacus Visual Playbook Builder and Automation Orchestrator acquisition. It defines the scope and requirements for Red Hat Professional Services engagement.

This document covers the interconnected technical areas requiring Red Hat expertise:

1. VM Provisioning and VLAN Management — API-driven infrastructure automation across multiple Nutanix sites

1. Windows Certificate Authentication — Secure, scalable authentication for managing Windows hosts

1. Certificate Management Workflows — Git-based certificate lifecycle management with four (4) distinct workflows

1. Presigned URL Workflow — S3 asset retrieval for playbook execution without direct AWS credentials

1. Real-Time Job Output Streaming — WebSocket integration for live job status in external UI

1. AAP Infrastructure and Self-Management — Operational best practices for running AAP

Abacus: The Orchestration Application Abacus is the external application that orchestrates all user interaction and workflow coordination. Understanding Abacus's role is essential context for the AAP integration design.

Abacus Responsibilities

1. User interface for all VM provisioning and configuration requests

1. Determines which AAP job templates to invoke based on user actions and site context

1. Specifies which playbooks to execute (via Git commit reference)

1. Manages S3 asset references and coordinates presigned URL workflow

1. Handles user authentication and authorization before calling AAP APIs

1. Displays real-time job execution output streamed from AAP

1. Retrieves workstation onboarding script from Git AAP's Role AAP serves as the automation backend — it does not provide user-facing interfaces for end users. All user interaction happens in Abacus; Abacus calls AAP APIs; AAP runs playbooks against target infrastructure.

Key Architectural Principle Abacus orchestrates, AAP executes. Users interact with Abacus; Abacus calls AAP APIs; AAP runs playbooks against target infrastructure.

VM Provisioning and VLAN Management This section describes the requirements for automating Windows VM provisioning and network management across multiple Nutanix sites.

Multi-Site Architecture The Government has eight (8) geographically distributed sites, each with its own Nutanix Prism Element cluster. A single AAP installation serves all sites with strict security boundaries between them.

Component
Details
Ansible Automation Platform
Single installation, multi-Organization architecture
Nutanix Environment
Eight (8) Prism Element clusters plus Prism Central; Windows VM templates in image library
Source Control
On-premise Git (Gitea or GitLab)
Identity Provider
Active Directory via LDAPS
Container Platform
Red Hat OpenShift (deployed on Nutanix)
Object Storage
Nutanix Objects (S3-compatible)

Multi-Site Security Model The primary security requirement is strict site isolation. Users at Site A shall not have access to, view, or modify resources at Site B — even through misconfiguration.

Implementation approach:

1. One (1) AAP Organization per site (e.g., Minneapolis, Omaha, Fargo)

1. Organization-scoped Nutanix credentials (each credential can only access its site's cluster)

1. AD group to AAP Team mapping for RBAC

1. Site-specific VLAN configuration files in Git (vars/sites/{site}_vlans.yml) Physical Sites:

Fort Meade, Fargo, Sioux Falls, Saint Cloud, Minneapolis, Omaha, Iowa City, Central Iowa Logical Site:

VISN23 (administrative; no dedicated cluster; staff can work across all physical sites) Each physical site has its own Prism Element cluster and AD security groups. VISN23 staff are granted cross-site access via AD group membership.

VLAN/Network Configuration Support VLAN and network configuration management requirements are specified in Category LV [LV1-LV22]. The implementation mechanism (Abacus-native functionality, AAP workflows, dedicated service, or equivalent) is at Contractor discretion.

Red Hat Scope for VLAN-Related Work:

Red Hat's involvement in VLAN management is limited to technical guidance and integration support if the Contractor selects an AAP-based implementation approach:

If Contractor Selects
Red Hat May Provide
AAP-based VLAN management
Job template design guidance, playbook structure recommendations, Git integration patterns
AAP workflows for Nutanix queries
Integration guidance for nutanix.ncp collection usage
Non-AAP implementation
No Red Hat deliverables for VLAN management

Critical Safety Requirement VLAN ID modification restrictions per [LV14] shall be enforced regardless of implementation mechanism. This prevents accidental disconnection of VMs using existing VLANs.

API-Driven VM Provisioning VM creation is triggered by Abacus via AAP's REST API — not through AAP's self-service portal. Abacus specifies the site, and the appropriate job template is invoked.

API Design:

1. One (1) Job Template per site (eight (8) total): minneapolis-vm-create, omaha-vm-create, etc.

1. Abacus determines which endpoint to call based on user's site

1. Site identifier implicit in endpoint (not passed as parameter) VM Creation Parameters:

Parameter
Required?
Notes
vm_name
Yes
Name for VM in Nutanix
ip_address
Yes
Static IPv4 address (used to determine VLAN via subnet math)
template_name
Yes
Nutanix template to clone from
site
Yes
Site identifier (e.g., "minneapolis")
project
Yes
Project identifier for inventory organization
group
Yes
Environment group: development, staging, or production
cpu
No
Number of vCPUs (default TBD)
ram_gb
No
Memory in GB (default TBD)
disks
No
Array of disk configs: drive_letter, size_gb, partition_block_size_kb
cdrom_count
No
Number of blank CD-ROM drives (default: 0)

VM Creation Workflow Steps The VM creation workflow shall execute the following steps:

1. Load VLAN configuration from Git for the target site

1. Perform "subnet math" to determine which VLAN the IP address belongs to

1. Validate IP is within usable range (not gateway, broadcast, etc.)

1. Check for IP conflicts (query existing VMs in Nutanix)

1. Create VM in Nutanix using specified template

1. Configure disks and CD-ROMs

1. Configure Windows hostname and network settings (mechanism at Contractor discretion per [L25])

1. Power on VM

1. Onboard VM for Windows authentication (see Section 3.6)

1. Add host to AAP inventory based on site, project, and group

1. Return result to Abacus Automated VM Onboarding After a VM is created and powered on, it must be onboarded into the Windows certificate authentication system before it can be managed by Ansible. This onboarding is an extension of the VM creation workflow.

Workflow continuation after VM power-on:

1. Wait for VM to complete initial Windows setup (sysprep completion)

1. Execute onboarding configuration (either via sysprep or initial connection)

1. Validate WinRM connectivity using certificate authentication

1. Add host to AAP inventory with correct Organization, Inventory, environment group, and component group memberships

1. Tag host with appropriate metadata (creation date, template used, etc.)

DRY Architecture Requirement All eight (8) sites shall use identical playbook code — a single Git repository with no per-site playbook copies. Site-specific differences shall be handled via:

1. Organization-scoped credentials

1. Job Template extra variables (site_name, nutanix_cluster)

1. Site-specific variable files in Git (vars/sites/{site}_vlans.yml) Benefit: Update playbook logic once; all eight (8) sites automatically receive the update.

VM Template Registry (Admin-Only) Infrastructure administrators need the ability to register new Windows VM templates so they become available for VM provisioning. This shall be restricted to admin AD groups only.

Windows Certificate Authentication This section describes the certificate-based authentication design for managing Windows hosts with Ansible.

Problem Statement The Government needs to manage a mixed environment of Windows machines where some are domain-joined and others exist in workgroups. Several critical constraints eliminate traditional authentication methods.

Constraint
Impact
NTLM Deprecation
Microsoft is actively sunsetting NTLM; building on it would require future migration
Mixed Environment
Must support both domain-joined and workgroup machines with unified approach
Manual Certificate Generation
Certificates must be requested through Venafi portal; no automated enrollment
One-Year Certificate Validity
Annual renewal cycle creates operational overhead at scale
Hundreds of Windows Hosts
Per-host certificate management would be unsustainable
No Additional Software
Cannot modify base Windows installation beyond Windows 11 / Server 2019+

Why Traditional Methods Do Not Work:

Method
Why Rejected
NTLM
Being deprecated by Microsoft; GPO restrictions on domain machines
CredSSP
Sends credentials to remote host; Ansible docs warn against production use
Basic Auth
Transmits password with each request; intended for dev/training only
Kerberos
Requires AD domain membership; does not work for workgroup machines

Solution: Two-Certificate Architecture The solution is a certificate-based authentication system where Ansible uses WinRM over HTTPS with client certificate authentication. This approach uses two (2) separate certificates with distinct purposes.

Certificate 1: Windows Host Server Certificate

1. Purpose: Encrypts the WinRM communication channel (like HTTPS for a website)

1. Location: Each Windows host generates its own self-signed certificate

1. Private key: Never leaves the Windows host; never shared with Ansible

1. Validation: Ansible ignores validation (ansible_winrm_server_cert_validation: ignore) Certificate 2: Ansible Client Certificate

1. Purpose: Proves Ansible's identity; replaces username/password authentication

1. Location: Single CA-signed certificate from Venafi on Ansible controller

1. Private key: Stored on Ansible controller only

1. Public key: Distributed to all Windows hosts (Trusted People store)

1. Mapping: Windows hosts map this certificate to local ansible_admin user Key Design Decision: Why Self-Signed for Server Certs?

Using CA-signed certificates for each Windows host would require managing hundreds of individual certificates through the manual Venafi portal with annual renewals. Self-signed certificates provide the encryption needed without this operational burden. Security risk is acceptable because Ansible initiates all connections to explicitly configured IP addresses.

Git Repository for Platform Configuration All certificate materials, root CAs, and generated scripts are stored in a dedicated Git repository. This design provides several benefits:

1. No TLS trust dependency: Git access via SSH requires no TLS certificate validation

1. Version history: All changes are tracked with full audit trail

1. Single source of truth: One authoritative location for all certificate-related materials

1. Resilience: If S3 or other infrastructure becomes untrusted due to CA rotation, Git remains accessible Certificate Management Workflows Overview Certificate management is divided into four (4) distinct workflows, each with a specific purpose and scope:

Workflow
Purpose
Trigger
What It Touches
Workflow 1: Certificate Preparation & Staging
Extract Venafi export, stage materials in Git, generate onboarding script
Admin runs after uploading PFX to Git
Git repository only (safe operation)
Workflow 2: Certificate Rotation to Existing Hosts
Push new certificate to all managed Windows hosts with rollback safety
Admin runs after Workflow 1
All Windows hosts
Workflow 3: Root CA Distribution to Windows Hosts
Push all root CAs to Windows hosts' trust stores
Admin runs when new CA added
All Windows hosts (idempotent)
Workflow 4: Execution Environment CA Trust Update
Update EE's CA trust e.g. via ConfigMap/Secret
Admin runs when new CA added
OpenShift ConfigMap/Secret

Initial Host Onboarding Workflow When adding a new Windows host to Ansible management, an administrator performs an initial RDP session to execute a PowerShell setup script. This is the "moment of physical presence" to establish trust relationships.

The onboarding script performs these steps:

1. Generate self-signed certificate and configure WinRM HTTPS listener on port 5986

1. Import the root CA certificate that signed the Ansible controller certificate into Trusted Root store

1. Read Ansible controller certificate thumbprint

1. Create local ansible_admin user account with strong random password (generated locally, never transmitted)

1. Create certificate mapping: "when this certificate is presented, authenticate as ansible_admin" After onboarding, Ansible can connect using certificate authentication without passwords transmitted over the network.

Red Hat Deliverable: Bootstrap Script Generation Red Hat shall develop a job template that generates the PowerShell onboarding script. The script is dynamically generated by reading the current controller certificate thumbprint, public certificate, and signing root CA from the ansible-platform-config Git repository, then rendering the templates/onboarding-workstation.ps1.j2 template. The generated script is committed to generated/onboarding-workstation.ps1 in the same repository. This generation is part of Workflow 1 (Certificate Preparation & Staging).

Workflow 1: Certificate Preparation and Staging This workflow extracts certificate materials from a Venafi export, stages them in Git, and generates the onboarding script. It is a "safe" operation that does not touch any Windows hosts.

Trigger:

Admin uploads venafi-export.pfx (or ZIP containing PFX) to controller-certificate/ folder in Git, then runs the job template with the decryption password.

Steps:

1. Pull latest from Git repository

1. Detect file format (PFX directly, or ZIP containing PEM files, etc.)

1. Decrypt/extract using provided password: Private key → Store in AAP credential storage; Public certificate → Write to controller-certificate/public-cert.pem; Thumbprint → Extract and write to controller-certificate/thumbprint.txt; Signing root CA → If not already in root-ca/ folder, add it

1. Render onboarding-workstation.ps1 from Jinja2 template, embedding the root CA, thumbprint, public certificate, and logic for local user creation and certificate mapping

1. Write rendered script to generated/onboarding-workstation.ps1

1. Commit and push all changes to Git

1. Job completes successfully Red Hat Deliverable: Certificate Preparation Job Template Red Hat shall implement Workflow 1 as a job template that extracts certificate materials from the Venafi export, handles various export formats, securely manages the decryption password (not persisted after job completion), and commits all extracted materials to Git.

Workflow 2: Certificate Rotation to Existing Hosts This workflow pushes the new certificate to all managed Windows hosts. It requires careful design for reliability and rollback safety.

Pre-Conditions:

1. Workflow 1 has completed — all materials staged in Git

1. The old certificate is still valid (running 45-60 days before expiration)

1. AAP currently authenticates to Windows hosts using the old certificate

1. Both old and new credentials are available to AAP Safety Mechanism: Scheduled Task Rollback Before changing anything on a host:

1. Create a Windows scheduled task that will restore the old certificate mapping

1. Task triggers either on system reboot OR after a timeout (e.g., 10-15 minutes) Then make the change:

1. Apply new certificate mapping (new thumbprint → ansible_admin) Validate immediately:

1. Attempt to connect using the NEW certificate Based on result:

1. Success: Delete the scheduled task (disarm the rollback), mark host as rotated

1. Failure: Leave scheduled task in place, mark host as failed — the host will auto-recover when the task fires or on next reboot Red Hat Deliverable: Certificate Rotation Workflow Red Hat shall implement Workflow 2 with robust status tracking, automatic rollback mechanisms, support for re-runs against failed hosts, and clear operational visibility. Red Hat shall devise the optimal approach for dual-certificate handling during the transition period.

Workflow 3: Root CA Distribution to Windows Hosts This workflow pushes all root CA certificates from the Git repository to all managed Windows hosts. It is idempotent — Windows handles "already present" gracefully.

Purpose:

Ensure Windows hosts trust all internal root CAs. This is needed for presigned URL downloads from Nutanix Objects S3 endpoint and any other internal HTTPS services Windows hosts may need to access.

Trigger:

Admin runs job template after adding new CA to root-ca/ folder in Git.

Steps:

1. Pull latest from Git repository

1. Read all .crt files from root-ca/ folder

1. For each managed Windows host: Push all CA certificates to Trusted Root Certification Authorities store (Windows handles idempotency natively)

1. Track per-host status: success, failed, unreachable

1. Persist state so re-runs can target only incomplete hosts

1. Report summary Red Hat Deliverable: Root CA Distribution Job Template Red Hat shall implement Workflow 3 with the same status tracking and re-run support as Workflow 2.

Workflow 4: Execution Environment CA Trust Update This workflow updates the AAP Execution Environment's CA trust store without requiring a container image rebuild.

Purpose:

Ensure the Execution Environment trusts all internal root CAs. This is required for amazon.aws.aws_s3 collection connecting to Nutanix Objects (presigned URL generation) and any other collections/modules that connect to internal TLS endpoints.

Desired Outcome:

1. Pull latest from Git repository

1. Read all .crt files from root-ca/ folder

1. Transform to format required by EE (Red Hat to recommend appropriate format)

1. Update the EE's CA trust source via the recommended mechanism

1. Result: Collections use system CA trust, which now includes new CAs — without rebuilding the EE image Red Hat Deliverable: EE CA Trust Update Automation Red Hat shall implement Workflow 4 to update the Execution Environment's CA trust without requiring image rebuilds, reading from the Git repository as the source of truth. Red Hat shall recommend and implement the optimal mechanism for the OpenShift environment.

Administrator Certificate Renewal Workflow The annual certificate renewal should be as simple as possible for the administrator.

Desired admin experience:

1. Admin requests new certificate from Venafi portal, downloads the export (typically a ZIP containing an encrypted PFX)

1. Admin uploads the raw Venafi export (unmodified) to the Git repository (controller-certificate/ folder) via the Git web interface

1. Admin triggers Workflow 1 through AAP

1. Admin provides the PFX decryption password when prompted (this password shall be treated as sensitive data and not persisted after the workflow completes)

1. Admin reviews the generated onboarding script and staged materials

1. Admin triggers Workflow 2 to rotate existing hosts

1. If a new root CA was added, admin triggers Workflow 3 and Workflow 4 (or these may be automatically chained) Key requirement:

The administrator should not need access to Linux command-line tools, OpenSSL, or manual file manipulation. The goal is that an administrator can complete annual certificate renewal by performing two (2) primary actions: uploading a file and providing a password.

CA Certificate Updates When a new internal root CA is issued (for example, when the certificate authority used to sign the Nutanix Objects TLS certificate rotates), the administrator commits the new CA certificate file to root-ca/ in the Git repository.

Required propagation:

1. Workflow 3: Push new CA to all existing managed Windows workstations

1. Workflow 4: Update Execution Environment CA trust via ConfigMap/Secret

1. Workflow 1 (if re-run): Regenerate onboarding script (though new hosts only need the controller certificate's signing CA) Operational Visibility and Recovery All certificate management workflows involve changes across hundreds of hosts or critical infrastructure. Red Hat shall design these workflows with operational visibility as a first-class concern.

Requirements:

1. Clear reporting on which hosts were updated successfully, which failed and why, and which were unreachable

1. Support for targeted re-execution against only the hosts that need attention, without re-processing hosts that already succeeded

1. A mechanism for administrators to verify the current state of any host (which certificate thumbprint is it trusting? which CAs are in its trust store?)

Presigned URL Workflow for Asset Retrieval Overview Abacus needs a two-job AAP Workflow that:

1. Job 1: Receives a list of S3 asset references, generates presigned download URLs for each, and stores them for the next job

1. Job 2: Executes the actual playbook, with presigned URLs available as variables for download tasks Assets are ZIP files that may contain a single file or a folder structure when extracted.

Job 1: Presigned URL Generation Trigger:

API request from Abacus to AAP.

Input (via extra vars):

Abacus provides a list of S3 asset references, each containing:

1. asset_id: Unique identifier for the asset

1. s3_bucket: Bucket name (e.g., "abacus-playbook-assets")

1. s3_key: Object key path (e.g., "assets/my-asset-key")

1. s3_version_id: Specific version of the object Processing:

1. Iterate over the assets_to_fetch list

1. Use amazon.aws.aws_s3 collection with mode: geturl

1. Generate presigned URL with twelve (12) hour TTL (if configurable)

1. Build output dictionary mapping asset_id to presigned URL Critical TLS Requirement Job 1 MUST use the FQDN TLS endpoint for Nutanix Objects (e.g., objects.abacus.yourdomain.com) and MUST validate the TLS certificate. Presigned URLs contain whatever hostname was used to generate them. Target workstations trust the CA-signed FQDN certificate, not a static IP.

Output (stored for Job 2):

A dictionary (presigned_url_download_array) where keys are asset identifiers and values are the presigned download URLs.

Job 2: Playbook Execution Input:

1. Git commit reference specifying which playbook to execute (Abacus determines this)

1. presigned_url_download_array from Job 1 Processing:

1. Pull playbook from Git at specified commit

1. Execute playbook where abacus.retrieve tasks reference URLs via: {{ presigned_url_download_array['asset_id'] }}

1. abacus.retrieve downloads the asset, saves to destination, handles unzipping The abacus.retrieve Collection abacus.retrieve is a custom Ansible module/collection that the Prime Contractor shall build. It handles downloading assets via presigned URL, saving to a destination path defined in Abacus, and extracting ZIP contents.

This collection is outside the scope of Red Hat's development responsibility.

Black Box Contract From AAP/Red Hat's perspective:

1. They receive: A dictionary containing all S3 asset identifiers plus versions needed

1. They produce: A structured map of asset identifiers to presigned download URLs, available to the subsequent playbook execution job From Abacus's perspective:

1. Abacus handles: Batching all required assets into the input dictionary, authoring playbooks that reference the output map, and determining which playbook (Git commit) to execute

Real-Time Job Output Streaming Abacus needs to display real-time job execution output to users as playbooks run. This requires integrating with AAP's job output streaming capabilities.

Requirements

1. Stream job output in real-time to Abacus UI as tasks execute

1. Display task status (running, success, failed, skipped) as it happens

1. Show playbook progress without requiring page refresh or polling

1. Support multiple concurrent users viewing different job outputs Proposed Approach: WebSocket Integration AAP provides WebSocket endpoints for streaming job output. Abacus (or its backend) needs to connect to these WebSockets to receive real-time updates.

Expected workflow:

1. Abacus launches a job via AAP API

1. AAP returns job ID

1. Abacus establishes WebSocket connection to AAP for that job ID

1. AAP streams job events (task start, task complete, output lines) via WebSocket

1. Abacus UI renders events in real-time

1. WebSocket closes when job completes WebSocket Integration Guidance Red Hat shall provide documentation and guidance sufficient for the Prime Contractor to implement real-time job output streaming from AAP to the Abacus application UI. This guidance shall enable an external development team to properly implement the streaming integration.

Requirements:

ID
Requirement
What It Enables
RH-WS1
Red Hat shall provide documentation of AAP mechanism(s) for real-time job output streaming (WebSocket endpoints, SSE endpoints, or equivalent).
Prime Contractor understands available streaming options
RH-WS2
Red Hat shall document authentication and authorization requirements for streaming connections, including token formats, session management, and credential refresh patterns.
Secure streaming implementation
RH-WS3
Red Hat shall document the event format and structure for job output messages, including field definitions, data types, and example payloads.
Prime Contractor can parse and display job events correctly
RH-WS4
Red Hat shall document best practices for connection establishment, keepalive mechanisms, and reconnection strategies on failure.
Robust streaming implementation that handles network interruptions
RH-WS5
Red Hat shall document rate limits, connection limits, or other constraints that affect streaming implementations.
Prime Contractor can design within system constraints
RH-WS6
Red Hat shall provide sample implementation or reference code for streaming integration (if available from Red Hat documentation or examples).
Accelerates implementation; reduces integration risk

This guidance supports requirement [K8] (real-time job output streaming with ≤5 second latency) and documentation deliverable [AD16] (streaming integration documentation).

Latency Requirement:

Per [K8], job output shall appear in the Abacus UI within five (5) seconds of AAP recording task completion status. Red Hat guidance shall enable the Prime Contractor to achieve this latency target.

Deliverable Format:

Red Hat may deliver this guidance as:

1. Standalone documentation package

1. References to existing Red Hat documentation with specific section citations

1. Combination of documentation and working sample code The Prime Contractor shall incorporate Red Hat's guidance into the [AD16] deliverable, documenting the specific implementation approach chosen.

Execution Environment CA Trust The Execution Environment used by AAP must trust the internal CA that signed the Nutanix Objects TLS certificate. This is required for:

1. Job 1 (presigned URL generation) to connect to Nutanix Objects S3 API with proper TLS validation

1. Any other playbooks that connect to internal TLS endpoints Requirement:

CA trust updates shall be possible without rebuilding the Execution Environment image. This is achieved via Workflow 4 (Section 4.9), which updates an OpenShift ConfigMap or Secret that the EE mounts as its CA trust source.

Source of Truth:

The root-ca/ folder in the ansible-platform-config Git repository contains all internal root CA certificates. Workflow 4 reads from this folder and updates the ConfigMap/Secret accordingly.

Red Hat Deliverable: EE CA Trust Configuration Red Hat shall configure the Execution Environment to mount CA certificates from an OpenShift ConfigMap or Secret, and implement Workflow 4 to update this ConfigMap/Secret from the Git repository without requiring container rebuilds.

AAP Infrastructure and Self-Management This section covers operational requirements for running and maintaining the AAP installation itself.

Operational Requirements Red Hat shall provide recommendations for standard operational practices. Areas of interest include:

1. Backup and Recovery: Automated backup of AAP configuration, credentials, and job history

1. Monitoring and Alerting: Health checks, performance monitoring, capacity planning

1. Log Management: Centralized logging, log retention policies, audit trails

1. High Availability: If applicable, HA configuration recommendations

1. Upgrades and Patching: Procedures for AAP upgrades with minimal disruption

1. Credential Rotation: Automated rotation of AAP service accounts and API tokens

1. Housekeeping: Job history cleanup, artifact management, database maintenance AAP Managing AAP Where possible, AAP should manage its own operational tasks via scheduled job templates:

1. Periodic health checks and status reports

1. Automated cleanup of old job records

1. Certificate expiration monitoring and alerting

1. Inventory synchronization verification

1. Git repository connectivity checks

Technical Environment Summary Windows Environment

Item
Details
Operating Systems
Windows 11, Windows Server 2019, 2022, 2025
Environment Mix
Domain-joined and workgroup machines
Scale
Hundreds of Windows hosts across eight (8) sites
WinRM Port
5986 (HTTPS)
Inventory Addressing
Static IP addresses (no DNS dependency)

Certificate Infrastructure

Item
Details
Certificate Authority
Venafi (manual portal-based generation)
Certificate Validity
One (1) year
Export Format
ZIP containing PFX (password-encrypted)
Rotation Timeline
45-60 days before expiration

Nutanix Environment

Item
Details
Clusters
Eight (8) Prism Element clusters (one per site) plus Prism Central
Object Storage
Nutanix Objects (S3-compatible)
Object Store Name
abacus-object-store
Objects FQDN
objects.abacus.yourdomain.com (TLS, CA-signed)
Ansible Collection
nutanix.ncp (recommended if using Ansible-based approach)

Nutanix Objects Buckets

Bucket
Purpose
Writers
Readers
abacus-playbook-assets
Asset storage for playbook execution
Abacus ingress uploader
AAP Job 1 (presigned URL generation)

Supporting Infrastructure

Item
Details
Source Control
On-premise Git (Gitea or GitLab)
Identity Provider
Active Directory via LDAPS
Container Platform
Red Hat OpenShift (deployed on Nutanix)
Platform Configuration
Git repository (ansible-platform-config)
Asset Storage
abacus-playbook-assets S3 bucket

Questions for Red Hat The following areas require Red Hat's expertise to validate approach and guide implementation decisions. These questions inform post-award technical planning.

Design Validation

1. Is the two-certificate model with self-signed server certs a sound approach for the stated constraints?

1. Are there better alternatives given the operational constraints?

1. What are the known limitations or gotchas with WinRM certificate authentication at scale?

Credential Management

1. What is the recommended credential storage approach? Options: AAP native encrypted storage, OpenShift Secrets with External Credential integration, HashiCorp Vault.

1. What are the trade-offs for each approach in terms of security, operational complexity, and credential rotation?

1. If starting with a simpler approach, what would the migration path look like to upgrade later?

AAP Integration

1. How does AAP handle certificate-based Windows authentication natively?

1. What is the recommended approach for storing the client certificate private key in AAP?

1. How should credentials be structured in AAP for multi-site Windows management?

API Design and Integration

1. Should Abacus use one (1) API token with access to all Organizations, or eight (8) separate tokens (one per site)?

1. Synchronous vs. asynchronous API response pattern? VM creation with sysprep can take 10-20 minutes.

1. How should errors be communicated to Abacus? HTTP status codes, webhooks, job polling?

Provisioning

1. At what point should AAP declare VM creation "successful"? Options: VM exists, VM powered on, VM responds to ping, Nutanix Guest Tools reports OS ready, WinRM connectivity established.

1. Can all disk configuration (additional drives, drive letters, NTFS allocation unit size) be done via sysprep? Or is post-boot PowerShell via WinRM required?

1. What is the recommended method for sysprep injection? Nutanix native guest customization, Nutanix Guest Tools, or another approach?

1. How should newly created VMs be automatically added to AAP inventory? API calls, inventory plugins, or another mechanism?

Constraints and Assumptions Firm Constraints

1. Single AAP installation (cannot deploy multiple AAP instances)

1. Static IP only — no DHCP or IP pool management

1. If using Ansible for Nutanix integration, the official Nutanix NCP collection (nutanix.ncp) is recommended

1. On-premise Git repository required (not GitHub/GitLab.com)

1. Single NIC per VM only

1. No VLAN deletion via AAP (prevents accidental outages)

1. Manual certificate generation via Venafi portal (no automated enrollment)

1. Cannot add software beyond base Windows installation

1. Presigned URL generation must use FQDN TLS endpoint with certificate validation Assumptions

1. AAP installation will be deployed as part of this project

1. Nutanix environment is operational and accessible

1. Active Directory is configured with appropriate site groups

1. Network connectivity exists between all components

1. Windows VM templates already exist in Nutanix (manual creation)

1. S3 bucket (abacus-playbook-assets) is available for asset storage

1. Git repository (ansible-platform-config) is available for platform configuration

1. Internal CA certificate is available for Execution Environment trust configuration Scope Clarification In Scope The following items are explicitly within Red Hat Professional Services scope for this engagement:

Item
Description
Reference
OpenShift Installation
Red Hat OpenShift Container Platform installation on Nutanix cluster per disconnected/air-gapped procedures
INF-OCP1 through INF-OCP6
AAP Installation
Ansible Automation Platform installation on OpenShift using AAP Operator
INF-AAP1 through INF-AAP6
AAP Configuration
Organizations, Projects, Inventories, Job Templates, Workflows, LDAPS integration, credential configuration
INF-AAP7 through INF-AAP12
Certificate Workflow 1
Certificate Preparation and Staging job template
CW1, CW2, CW3
Certificate Workflow 2
Certificate Rotation to Existing Hosts job template
CW4, CW5, CW6
Certificate Workflow 3
Root CA Distribution to Windows Hosts job template
CW7, CW8
Certificate Workflow 4
Execution Environment CA Trust Update
CW9
Presigned URL Workflow
Two-job workflow for S3 asset retrieval
J12, J18
WebSocket Integration Guidance
Documentation for external application integration with AAP job streaming
K8
VM Provisioning Playbooks
Technical guidance and AAP integration support (Contractor retains delivery accountability per L27, CR4)
L1 through L30
VLAN Configuration Support
Technical guidance for VLAN-related AAP integration if Contractor selects AAP-based implementation; no mandatory deliverables
LV1-LV22 (implementation at Contractor discretion)

Out of Scope The following items are explicitly outside Red Hat Professional Services scope:

Item
Responsible Party
Nutanix hardware procurement and physical installation
Nutanix Professional Services (Attachment B)
Nutanix Prism Element and Prism Central deployment
Nutanix Professional Services (Attachment B)
Nutanix Objects (S3) activation and configuration
Nutanix Professional Services (Attachment B)
Abacus application development
Primary Contractor
abacus.retrieve Ansible collection development
Primary Contractor
Asset Ingress Service development
Primary Contractor
Windows VM template image creation
Government
Active Directory configuration and service accounts
Government
TLS certificate issuance
Government PKI Team
DNS record creation
Government Network Team
Host inventory structure and automatic host registration
Primary Contractor (per U8)

Rationale: This clarifies that while Red Hat provides guidance on AAP inventory management (Section 3.6), the actual implementation of automatic host registration to the correct Organization/Inventory/groups is Prime Contractor responsibility. This prevents any ambiguity during acceptance testing.

Expected Deliverables Documentation

1. Architecture diagrams and design decisions

1. API specifications for Abacus integration

1. Admin runbooks (adding sites, rotating credentials, troubleshooting)

1. User guides for VLAN management workflows

1. WebSocket integration guide for external vendor

1. Certificate management workflow guide

1. Execution Environment CA trust update runbook

1. S3 credential rotation runbook AAP Installation and Configuration

1. AAP Operator deployment on OpenShift cluster (installation)

1. Nine (9) Organizations (eight (8) physical sites plus VISN23) with appropriate RBAC

1. Active Directory/LDAPS integration

1. Credential storage configuration (per recommended approach)

1. Projects (Git integration)

1. Job Templates for VM creation, VLAN management, template registration

1. Workflow Job Templates for presigned URL plus playbook execution pattern

1. Inventory structure supporting site/project/group hierarchy

1. Operational job templates for AAP self-management Internal AAP organization (specific Organization names, Project structure, Job Template naming) is at implementation team's discretion.

Ansible Playbooks and Roles

1. Git repository with all playbook code per [AD17] to aid in satisfying both source code and documentation according to contract requirements

1. Site-specific variable files (VLAN lists, cluster mappings)

1. Global template registry

1. VLAN-related playbooks/roles: Only if Contractor selects AAP-based VLAN management implementation per Category LV; scope determined post-award based on Contractor's architectural approach

1. Presigned URL generation playbook

1. Certificate preparation playbook (extracts from Venafi export, commits materials to Git, generates onboarding script)

1. Certificate rotation playbook for existing Windows hosts (with rollback mechanisms)

1. Root CA distribution playbook for Windows hosts (reads from Git root-ca/ folder)

1. EE CA trust update playbook (transforms CAs from Git, updates OpenShift ConfigMap/Secret)

1. VM onboarding playbooks with inventory registration Scripts and Workflows

1. PowerShell onboarding script generation (via Workflow 1, using Jinja2 template)

1. Certificate ingestion workflow (PFX extraction from Venafi export) Validation and Training

1. Test scenarios and validation checklist

1. Knowledge transfer sessions (format TBD with Red Hat)

1. Sample API payloads and integration examples

Scope Boundary Red Hat provides guidance documentation. The Prime Contractor implements the streaming integration and delivers [AD16] documentation. Per [CR4], the Prime Contractor is accountable for working capability regardless of Red Hat's guidance delivery.

— End of Attachment A —

File details come from the government source that posted it. Updated .