ATTACHMENT 7 - HUD Exchange System Description.pdf

Current HUD Exchange System

Hosting

Provider and region/s

AWS Commercial Cloud, us-east-1. (Some services are global, such as CloudFront and WAF.)

Bandwidth and storage

Total Storage: S3: 15 TB; EBS: 5,074 GB; RDS: 150 GB

Bandwidth: 1650 GB per month

Compliance and security

Physical security controls are inherited from AWS. Most compute is AWS-managed but there are several self-managed EC2s which are hardened according to CIS guidelines.

Networking segmentation best practices are employed (VPCs, subnets, security groups, NACLs, NAT/Internet Gateways, etc.).

Cloud-hosted resources are accessed by Sophos VPN (which runs on an EC2 AMI image).

Public endpoints are hardened by AWS’s WAF firewall.

Operational security controls include:

Access controls which limit users/and roles according to least privilege. These include application and cloud services.

Security scanners: AWS inspector is used to scan hosts (to CIS level 1.0 guidelines) and container images, Rapid7 for web application scanning, and dependency scanners for CI/CD pipelines.

EC2s are routinely patched, containers images are continuously upgraded, and dependencies are regularly updated.

Encryption in transit (TLS) as well as encryption at rest employed.

Scalability / Reliability

Depending on use case, vertical or horizontal scaling is used and high availability systems are utilized, following best practices.

Technology Stack CMS CMS consists of a stack running under self-hosted Docker Swarm including:

MariaDB RDBMS database on AWS RDS

Redis cache (containerized)

Mura CMS (containerized)

Lucee application server (containerized)

Tomcat servlet container (containerized)

Apache HTTPD web server (containerized)

AWS S3 and CloudFront

Adobe KMS for TLS endpoint encryption

The CMS hosts the following:

30,978 Production HTML pages

LMS The LMS has the following specifications:

Two FedRAMPed Adobe Learning Manager environments: 1) Production environment; and 2) STAGE environment

210,468 total users (peak monthly active users: 9,731)

Historical transcript data dating back to 2012:

o Total # of enrollment records: 617,634 o Total # of user progress records: 747,896

Discussion boards

Custom branded certificates

SSO is managed with Auth0, using the HUD Exchange CMS application as the

IdP.

API Connections - Communication between the HUD Exchange CMS application and ALM synchronizes user data and course information between the two systems using a series of API connections. User data is supplied by the CMS.

Integration with separate webinar platforms to support delivering virtual trainings, recording trainings, and providing for participant chat within trainings.

The course catalog, hosted in the CMS, is partially populated by LMS courses, with the addition of courses hosted by TA Provider outside the HUD

Exchange LMS.

HUD Exchange training catalog includes training pages for both trainings that utilize the HUD Exchange LMS and trainings where registration is managed by TA Providers outside of the LMS.

5,650 trainings are in the HUD Exchange Training Catalog.

Of those, 3,321 are active and 2,329 are archived.

1,856 trainings are hosted in the LMS.

Archived trainings have user transcript data attached to them.

A training may be composed of several files.

Types of trainings hosted in the HUD Exchange LMS include live webinars, live webinar series, in-person trainings, self-paced online trainings including SCORM, training videos, virtual, hybrid, and in-person conferences, blended training.

Webinar platform license management

Licenses for multiple webinar platforms utilized across Community Compass

TA Providers.

Webinar platform specifications:

Platform Number of licenses Scope

WebEx 7 1,000 seats, formal and informal webinars, breakout rooms

Zoom 1 3,000 seats, formal and informal webinars, breakout rooms

Sitewide Search Running under Kubernetes (EKS):

Elasticsearch (containerized)

Kibana (containerized)

FESS (containerized)

Custom site search service (containerized Python FastAPI microservice)

Quepid search relevancy tuning and regression testing tool (containerized Quepid microservice, containerized Redis, and RDS MySQL database).

Application Search Cores

Running under Kubernetes (EKS):

Solr (containerized)

Enterprise API Gateway

AWS API Gateway

Backend Ecosystem

EKS Kubernetes clusters

Legacy Docker Swarm EC2s (remaining workloads are being migrated to

EKS).

Many AWS services but most notably, Lambda & API Gateway.

PostgreSQL database instances running in AWS Aurora Serverless.

Observability:

CloudWatch

Elasticstack (containerized): Elasticsearch cluster, Kibana, various Beats

Infrastructure as Code (IaC)

Terraform is used for Cloud resources.

Ansible is used to provision legacy Docker Swarm EC2s.

Monolithic Application

The HUD Exchange CMS is a monolith application, with content pages primarily rendered with server-side MVC. It also hosts some of the embedded applications via custom CMS plugins.

COTS Products Adobe Learning Manager

MailChimp

ScreamingFrog

Google Analytics

FusionReactor

Development and Maintenance Environment

HUD Exchange has four permanent environments:

Development/Integration (private)

UAT (private)

Stage (password-protected; available to select stakeholders for UAT and demos)

Production (live environment)

Other environments are provisioned on an as-needed basis, for instance, for large efforts such as redesigns or overhauls.

CMS content is often scrubbed and propagated to lower environments, while application code and CMS templates are promoted to higher environments.

CI/CD Process Code is housed in Azure DevOps git repositories.

CI/CD pipelines are implemented in both Jenkins and Azure DevOps.

Pipelines differ based on the repository, but are generally used to build, test

(unit, integration, E2E, security), publish artifacts (Java libraries or docker images), and deploy.

Deployment Cadence

Most repositories are automatically built, tested, and deployed upon commit.

The CMS is deployed via Jenkins pipelines twice per week.

Out of cycle releases are done on demand.

Incident Management

Various monitors and alerts are set up and notify on-call responders, including:

Site availability monitors

Health check monitors

Resource usage monitors (such as low disk space)

Unexpected configuration changes

Scheduled E2E test failures

CI/CD pipeline failures

Documentation Tools

Confluence is used for project documentation. Markdown files are used for code repositories.

Number and Complexity of Templates

65+ custom base templates, each of which has customizable components, features, and design elements. In order to meet the needs of 90+ HUD programs, templates are designed to be highly flexible, and most require intermediate or advanced CSS/HTML knowledge.

Custom templates include, but are not limited to:

o Program Landing Page templates o Resource-related templates (i.e., online manuals, resource collections, video series, topical pages) o Resource Library-related templates (i.e., sitewide resource library, individual program resource libraries) o Tool-related templates (i.e., Funding Navigator, Equal Access

Assessment Tool, Income Calculator) o FAQ templates o News-related templates (i.e., sitewide news items, individual program newsletters) o Training-related templates (i.e., training calendars, class pages, training series) o Enhanced Interactive templates o Reports-related templates o Grantee-related templates (i.e., project profiles)

Hosted Applications The HUD Exchange currently contains 24 hosted applications.

Ask A Question (AAQ) AAQ is a system of help desks where HUD customers can ask questions and TA Providers and HUD staff answer questions.

Implemented as a CMS plugin to answer questions entered by the public.

It uses a subset of the CMS’s MariaDB database to store more than

229,000 questions (1,100 in process), answers, and related metadata.

Question/answer assets are stored in S3.

AAQ has 31 unique help desks, 20 of which are currently active, and a series of user permissions that are assigned by HUD Exchange to TA

Providers and HUD that allow 500+ users to coordinate answering the questions.

There is both full Admin access and a read only view into the questions.

Multiple workflows with dynamic form generation

Field pre-population for logged in users

Rich text editor

Attachments

Email notifications & automated replies

Issue and question linkages

Permission based access controls

Dashboards

Full field search (w/ SOLR)

Tabular extracts and ad hoc reporting

Collaboration features

FAQs FAQs is a searchable and filterable database of frequently asked questions across 30 HUD programs and requirements.

Implemented as custom CMS pages, with a custom admin panel, and a custom taxonomy system to support a finer level of categorization, and display related questions.

3,000+ unique FAQs on the HUD Exchange

Income Calculator Income Calculator is an interactive tool for determining income eligibility and assistance amounts for beneficiaries of 10 HUD programs.

Implemented as a custom CMS plugin and uses a subset of the CMS’s

MariaDB database.

https://www.hudexchange.info/program-support/my-question/ https://www.hudexchange.info/faqs/ https://www.hudexchange.info/incomecalculator/

It also uses a custom PDF generator to summarize calculations, with at least one PDF type per program. The PDF generator is built/published as a Jar by its own repository.

Income limit data is received annually, processed and loaded into the calculator.

Annually 57,000 calculations across 5,000 users

Annual Income Limits ETL Pipeline

Dashboards

Ad hoc reporting

Multiple dynamic multi-step workflows

Historical/archival of calculations

TA Request Portal TA Request Portal

Implementation serves 43,800+ HUD customer organizations.

Implemented as a custom CMS plugin and uses a subset of the CMS’s

MariaDB database to store 11,000+ TA requests, related comments, attachments, and other information. The assets are stored in S3.

Access to the user-facing TA Request form is provided to everyone with a

HUD Exchange account. Form includes ability to select individual or group of recipients, select from a list of relevant programs/topics (maintained via the CMS), enter request details.

A series of user permissions ensure that access to the “View TA

Requests” module that HUD uses to review and approve submitted requests is limited to select staff.

Maintain legacy TA Portal modules including Assignment, Work Plan, Management, Funding Sources.

The system auto-generates a weekly export for all approved TA Requests that is used by the HUD to import TA requests and related information.

Public registration and authentication

Multi-step dynamic workflow

Admin workflow with additional views for management

Email notifications & automated replies

Type-ahead

ETL from HUD source systems

Tabular reports

PDF exports

Funding Navigator Funding Navigator is an interactive tool for users to find billions of dollars in available funding to support climate resiliency, energy efficiency, renewable energy integration, healthy housing, workforce development, and environmental justice in HUD supported communities, programs, and properties.

React front-end with API Gateway, Lambda, and PostgreSQL database on serverless Aurora.

Data spreadsheet is supplied by a third-party on a weekly basis and is loaded via Data Export, Transform Load (ETL) process.

ETL process utilizes AWS Glue

Document Builder Document Builder is an interactive tool that provides a workspace for

HUD Exchange users to learn more about HUD requirements and draft organization documents, such as agency work plans.

React front-end application with API Gateway, Lambda, and PostgreSQL https://www.hudexchange.info/program-support/technical-assistance/ https://www.hudexchange.info/programs/build-for-the-future/funding-navigator/ https://stage.hudexchange.info/hudexchange-portal/work-plan-builder database on serverless Aurora.

Grantees Grantees is a searchable collection of HUD grantee contact information, awards, reports, and maps.

Implemented as a custom CMS plugin and uses a subset of the CMS’s

MariaDB database.

5,000 grantee pages containing contacts, awards, reports, and maps, which the Grantees app generates from the database.

Maintain three grantee search Solr cores for contacts, awards, and grantee/CoC pages.

Maintain 29 grantee reports pages with custom filters.

Receive and process over 350 contact update requests annually (CY 23)

(submitted via contact us/inbox/grantee admin)

Process and bulk upload CPD + CoC awards, including edits to original award amounts.

Over 9,000 awards mapped + loaded annually

Add new grantees to HUD Exchange organizational database

Maintain grantee/CoC report codes for publishing reports, including adding CoC report codes annually to align with current names

Process and post over 7,269 grantee-level reports annually

Process and post over 400 CoC jurisdiction map images to CoC pages annually

Environmental Review Calculators

Pipeline Potential Impact Radius Tool

Calculates the potential impact radius (PIR) of a pipeline that transports flammable or combustible gases or liquids with an operating pressure of above 200 psi.

React front-end with API Gateway, Lambda, and PostgreSQL database on serverless Aurora.

Lambda: Custom Python/SciPy for interpolating and extrapolating values from known 1d and 2d data sets.

Acceptable Separation Distance (ASD) Calculator

Calculates the Acceptable Separation Distance (ASD) from stationary hazards. The ASD is the distance from above ground stationary containerized hazards of an explosive or fire prone nature, to where a

HUD assisted project can be located.

Vanilla JavaScript Application. Form data is hosted in a CMS Content

Page. N.B. An AngularJS upgraded version that allows for construction of tanks for evaluation has been deployed but not enabled.

Barrier Performance Module (BPM) Calculator

Provides to the user a measure on the barrier's effectiveness on noise reduction

Vanilla JavaScript Application. Form data is hosted in a CMS Content Page

Day/Night Noise Level (DNL) Calculator

An electronic assessment tool that calculates the Day/Night Noise Level

(DNL) from roadway and railway traffic

Vanilla JavaScript Application. Form data is hosted in a CMS Content

Page.

STraCAT – Sound Transmission Classification Assessment Tool

Calculator to evaluate sound dampening of walls and materials

AngularJS with data from a JSON Data store from HUD sources https://www.hudexchange.info/grantees/

Housing Counseling Agency Eligibility Tool

(HCAET)

Housing Counseling Agency Eligibility Tool is a self-assessment tool that allows organizations to determine if they meet the basic requirements to apply to become a HUD-approved Housing Counseling Agency

Built as a custom CMS plugin and uses a subset of the CMS’s MariaDB database

CoC Data Maturity Assessment Tool

(DMAT)

DMAT is a self-assessment tool for organizations to review data practices for areas of improvement

Built as a custom CMS plugin and uses a subset of the CMS’s MariaDB database

Equal Access Assessment Tool

(EAAT)

EAAT is an interactive tool that provides tailored action steps to support users in meeting requirements of the Equal Access Rule

Built as a custom CMS plugin and uses a subset of the CMS’s MariaDB database

Trainings & Events Admin

Backend custom admin implemented as a CMS plugin.

Powers a searchable course catalog includes both LMS trainings and external trainings hosted outside of the LMS.

Ability to customize how trainings appear on the HUD Exchange based on training type (on demand, live webinar, in-person training), training status (coming soon/open/closed/waitlist), button text, time zone (for in-person trainings), and related topic tags.

Course data, including training status, are automatically imported from

LMS into the training catalog via an API, and courses are published to

HUD Exchange production site at appropriate time.

Site Resource Library A searchable, filterable collection of resources across the entire site that contains more than 10,000 Resource Abstracts, FAQs, Trainings, and

Reports.

Implemented as a custom CMS plugin with a custom admin for managing resources with custom tagging system.

Program-Specific Resource Libraries

Filterable collections of resources with a custom search.

Implemented as a custom CMS plugin with a custom admin for managing resources with custom tagging system.

Program-Specific Resource Libraries include CDBG-DR Resource Library;

Housing Counseling Resource Library; NFHTA Resource Library

Section 3 Resource Hub

Filterable collection of Section 3 resources with custom search and custom tagging system

React front-end with API Gateway, Lambda, and PostgreSQL database on serverless Aurora.

Export, Transform, Load (ETL) processes automated using AWS Glue

Data spreadsheet supplied by a third-party TA Provider on an on-demand basis and loaded via ETL.

Third Party-developed webpages and applications

Site content generated by TA Providers that is required to be hosted outside of the HUD Exchange CMS due to how it was developed and provided by third parties.

Hosted using S3 and CloudFront, some on sites.hudexchange.info, others integrated into www.hudexchange.info.

Making updates to these is handled outside of the CMS and requires developer involvement.

Reports Management Backend custom admin with built-in automation implemented as a custom CMS plugin to post reports to the HUD Exchange https://www.hudexchange.info/programs/housing-counseling/housing-counseling-agency-eligibility-tool/ https://www.hudexchange.info/programs/coc/coc-data-maturity-assessment-tool/ https://www.hudexchange.info/resources/equal-access-assessment-tool/ https://www.hudexchange.info/resources/ http://www.hudexchange.info/

219,000 reports hosted on the site, plus over 125,000 archived.

7,269 reports processed annually

Operations

User Load and Traffic Volume

1,793 GB user load

19,260,119 page views in CY2023 jMeter load testing tool used to assess the impact of planned features and configuration changes.

User Accounts 279,744 user accounts (all users)

38,609 new user accounts created in CY2023

Weekly User Issues / Technical Support Requests

9,100+ inquiries responded to annually, an average of more than

175/week.

Types of user inquiries: HUD Exchange content and tool, training, access, user accounts, HUD system/program/policy inquiries, general public including individuals experiencing homelessness or other housing needs, the press, researchers, and more

Listserv numbers and subscribers

80,668 listserv subscribers (HUD Exchange Mailing List)

47 unique active listserv groups (e.g., HOME, NFHTA)

750+ messages were sent via the HUD Exchange Mailing List in CY 2023 (over 19 million emails sent)

5 requests in CY2023 to create new listserv groups