1-Sources-Sought_Independent Validation Support Services .RFQ1704626.docx

Sources Sought Synopsis

The Department of Housing and Urban Development (HUD), Federal Housing Administration (FHA), is issuing this Sources Sought Notice as a means of conducting market research to identify parties having an interest in and the resources to support this requirement for providing Independent Validation Support Services for the financial/actuarial models. The result of this Market Research will contribute to determining the method of procurement. The applicable North American Industry Classification System (NAICS) code assigned to this procurement is 523920. The product service code is R411.

There is NO SOLICITATION at this time. This request for capability information does not constitute a request for proposals and submission of any information in response to this market survey is purely voluntary; the government assumes no financial responsibility for any costs incurred.

If your organization has the potential capacity to perform these contract services, please provide the following information: 1) organization name, address, email address, website address, telephone number, and size and type of ownership for the organization; and 2) TAILORED capability statement addressing the particulars of this effort, with appropriate documentation supporting claims of organizational and staff capability. If significant subcontracting or teaming is anticipated in order to deliver technical capability, organizations shall address the Administrative and Management structure of such arrangements.

The government will evaluate market information to ascertain potential market capacity to 1) provide services consistent in scope and scale with those described in this notice and otherwise anticipated; 2) secure and apply the full range of corporate financial, human capital, and technical resources required to successfully perform similar requirements; 3) implement a successful project management plan that includes: compliance with program schedules; cost containment; meeting and tracking performance; hiring and retention of personnel and risk mitigation; and 4) provide services under a performance based service acquisition contract.

Based on the responses to this Sources Sought Notice/Market Research, this requirement may be set-aside for small businesses or procured through full and open competition, and multiple awards may be made. Telephone inquiries will not be accepted or acknowledged, and no feedback or evaluations will be provided to companies regarding their submissions.

Submission Instructions: Interested parties who consider themselves qualified to perform the above-listed services are invited to submit a response to this Sources Sought Notice by Monday, July 8, 2023, at 2:00 pm EST. All responses under this Sources Sought Notice must be emailed to Diane Toledo-Gaskins at Diane.E.Toledo-Gaskins@hud.gov. Additional information on the required services is listed below.

If you have any questions concerning this opportunity, please contact: Diane Toledo-Gaskins at Diane.E.Toledo-Gaskins@hud.gov.

Background:

Task A – Single-Family (MMIF)

The MMIF is a group of accounts of the federal government in which transactions associated with FHA’s guarantee programs for single family mortgages are recorded. FHA currently insures roughly 8 million forward mortgages with insurance-in-force of $1.1 trillion and 600,000 reverse mortgages with $112 billion in insurance-in-force under the Home Equity Conversion Mortgage (HECM) program.

The intended uses of the Single-Family models to be validated are to:

A. Estimate FHA’s liability for loan guaranty for the forward and HECM programs for FHA’s financial statements.

B. Estimate selected elements of the MMIF capital ratio.

C. Develop budget estimates (credit subsidy rates and volume) and re-estimates; and, D. Estimate principal limit factors for HECM.

E. Enhance FHA’s ability to assess and manage risk, monitor trends, and evaluate policy with the capture of credit and loan application data.

Task B - Commercial (GI/SRI)

The GI/SRI is a group of accounts of the federal government in which transactions associated with FHA’s guarantee programs for Multifamily, Hospital, and Nursing Home/Assisted Living/Board & Care facilities’ mortgages are recorded. FHA currently insures roughly 14,000 commercial mortgages with insurance-in-force of $100 billion.

The intended uses of the commercial models to be validated are to:

A. Estimate FHA’s liability for loan guaranty for the commercial loans for FHA’s financial statements.

B. Develop budget estimates (credit subsidy rates and volume) and re-estimates.

Task C - Commercial Challenger Models (GI/SRI) FHA recognizes the significance of constantly refining and challenging existing models to ensure optimal decision-making. The developed challenger models will provide a systematic approach to validating whether the existing Commercial models can be improved.

The purpose of the developed Commercial Challenger models are as follows:

A. Add an additional layer of scrutiny beyond the standard validation techniques, offering a more thorough assessment of model performance and assumptions.

B. Promoting continuous improvement by encouraging regular reassessment and refinement of best practice modeling techniques.

General Description of Work:

Task A – Single-Family (MMIF)

(For each task (A1 through A5) the contractor shall perform the following 8 steps as needed)

Detailed descriptions of specific tasks are provided in Section 5 Specific Tasks and Deliverables.

Each validation report shall include findings and/or recommendations together with an assessment of the materiality of each weakness.

Task B - Commercial (GI/SRI)

(For each task (B1 through B18) the contractor shall perform the following 9 steps as needed)

Each validation report shall include findings and/or recommendations together with an assessment of the materiality of each weakness.

Task C - Commercial Challenger Models (GI/SRI)

(For each task the contractor shall perform the following steps as needed)

Step 1 Define objectives by clearing defining the objectives of the challenger model and its role in validating a commercial model. Determine the specific aspects of the existing model that need validation, such as accuracy, robustness, or predictive power.

Step 2 Gather relevant data needed for model development and validation. Ensure that the data is representative of the underlying processes and conditions that the models seek to capture. Clean, preprocess, and transform the data as necessary to ensure consistency and quality.

Step 3 Select appropriate modeling techniques and methodologies for developing the challenger model. Consider alternative approaches that may provide insights into different aspects of the problem or offer improvements over the existing model.

Step 4 Build the challenger model using the selected techniques and methodologies. Train the model on historical data, adjusting parameters and features as needed to optimize performance. Implement robust validation procedures, such as cross-validation, or out-of-sample testing, to assess the accuracy and generalization of the challenger model.

Step 5 Compare the performance of the challenger model with that of the existing model using relevant metrics and evaluation criteria. Identify areas of agreement between the two models, focusing on difference in predictions, biases, and error patterns. Conduct sensitivity analyses to understand the impact of key variables and assumptions on model outputs.

Step 6 Validate the challenger model against additional datasets or validation sets to ensure its reliability and generalizability. Assess the stability of model outputs over time and across different contexts to confirm its robustness. Address any discrepancies or inconsistencies between the challenger model and the existing model through further investigation and refinement.

Step 7 Document the development process, methodology, and findings of the challenger model in a comprehensive report. Provide clear explanations of the validation results, highlighting areas of improvement and implications for decision making.

Specific Tasks

Task A – Single Family (MMIF) For each year, all three Single Family models will be validated (Forward + HECM + TOTAL) concurrently. Thereafter, the Commercial models below, plus up to 2 model validations/year for Single Family validation scopes, to be determined at the beginning of each performance year (Commercial + Forward + HECM, Commercial + Forward + TOTAL, Commercial + HECM + TOTAL).

A1 - The HECM Annual Review Agenda. FHA uses a risk-based approach to review of its single-family models that provides full coverage over a three-year cycle. Higher risk model methods and components receive higher attention than lower risk elements. Consequently, the focus of model validation will change from year to year. FHA will work with the Contractor to develop an Annual Agenda by the beginning of each calendar year.

A2 - The FORWARD Mortgage Annual Review Agenda. FHA uses a risk-based approach to review of its single-family models that provides full coverage over a three-year cycle. Higher risk model methods and components receive higher attention than lower risk elements. Consequently, the focus of model validation will change from year to year. FHA will work with the Contractor to develop an Annual Agenda by the beginning of each calendar year.

A3 - The HECM Mortgage Insurance model is used to estimate HECM liability for loan guarantees, the HECM elements of the MMIF capital ratio, HECM budget re-estimates and the credit subsidy elements of budget development.

A4 - The FORWARD Mortgage Insurance model is used to estimate the forward liability for loan guarantees, the forward elements of the MMIF capital ratio, forward budget re-estimates and the credit subsidy elements of budget development.

A5 - The TOTAL-CHALLENGER Model (TCM) Review & Validation is used as a statistically derived algorithm developed by HUD to evaluate borrower credit history and application information. TCM is accessed through an Automated Underwriting System (AUS) and is not an AUS itself. TCM provides a rank-ordered score of the relative risk of borrower default during the first 4 years of the mortgage resulting in claim. TCM provides AUSs with the “Accept/Refer” decision based on the borrower’s score relative to a policy cut point and other policy rules. Validation of the TOTAL model will be done in increments as information is made available. The scope of the TOTAL model validation task will be determined at the beginning of each performance year and will be developed by working with the vendor to determine specific tasks to be performed that year.

Task B - Commercial (GI/SRI) For the Base Year, all three Single Family models will be validated (Forward + HECM + TOTAL). Thereafter, the Commercial models below, plus up to 2 model validations/year for Single Family validation scopes, to be determined at the beginning of each performance year (Commercial + Forward + HECM, Commercial + Forward + TOTAL, Commercial + HECM + TOTAL).

FHA applies 5 different types of modeling with 18 sub-categories, frameworks to the commercial (multifamily and healthcare) portfolios:

Within the logistic regression models, there are various sub-models. The differences between the sub-models are the variables and the types of loans or Risk Categories specified for each equation. But they are all structurally the same type of model, just with different variables.

The list of the model categories and the sub-categories are:

I. Logistic Regression Models

1. Multifamily Apartment Claim Model B1 - Unassisted New Construction/Substantial Rehabilitation Default Model B2 - Unassisted Refinance Default Model B3 - Assisted New Construction/Substantial Rehab. and Refinance Default Model B4 - Claim Transition Model

2. Multifamily Apartment Prepayment Models B5 - Unassisted New Construction/Substantial Rehabilitation Prepayment Model B6 - Unassisted Refinance Prepayment Model B7 - Assisted New Construction/Substantial Rehab. and Refi. Prepayment Model

3. Residential Care Facilities (Nursing Home/Assisted Living/Board & Care fac.) Models B8 - Section 232 New Construction/Substantial Rehabilitation Claim Model B9 - Section 232 Refinance Claim Model B10 - Section 232 New Construction/Substantial Rehabilitation Prepayment Model B11 - Section 232 Refinance Prepayment Model

4. Hospital Loan Prepayment Model B12 - Section 242 Prepayment Model II. Hospital Loan Financial Simulation Model B13 - Section 242 Claim Model III. Monte-Carlo Simulation Model B14 - Stochastic Simulation Model with data input from all the previous models.

IV. Cash Flow Models B15 - Cash Flow Models (CFM) for Liability for Loan Guarantee (LLG) B16 - Cash Flow Models (CFM) for Loan Loss Reserve (LLR) V. Recovery on Assets Models B17 – Recovery on Assets Models (ROA) for Liability for Loan Guarantee (LLG) B18 – Recovery on Assets Models (ROA) for Loan Loss Reserve (LLR) Task C – Commercial Challenger Models For each year, four Commercial Challenger models are delivered at different times of the year, not to coincide with Task A and Task B deliverables.

The purpose of the four developed Commercial Challenger models each year are as follows:

A. Add an additional layer of scrutiny beyond the standard validation techniques, offering a more thorough assessment of model performance and assumptions.

B. Promoting continuous improvement by encouraging regular reassessment and refinement of best practice modeling techniques.

The tasks require a set of skill categories based on the following approximate breakdown:

Subject Matter Experts The Contractor shall provide the minimum of two SMEs. SMEs shall have three years’ minimum experience in similar positions involving social science research and in one or more of the following fields: social science research, distressed asset disposition, servicing timelines, mortgage measurement, monitoring, mitigation and management, mortgage modeling, model management, mortgage valuation, mortgage insurance valuation, and other fields related to mortgage insurance contract servicing. SMEs must also have specific experience with validating financial and econometric models and validating statistical model coding. The SMEs should also have solid practical knowledge and experience with scientific research principles, methods and practices, as applied to social science research; including a variety of research approaches, tools, and techniques, such as qualitative and quantitative data, statistical analysis, experiments, field surveys, case research and so forth. SMEs are considered Key Personnel by the Government and shall be listed as such in accordance with HUDAR 2452.237-70, "Key Personnel”.

· Contemplated performance metrics:

Refer to section 7.0 (Performance Requirements Summary) of the Draft Performance Work Statement (PWS)for performance metrics.

· Contemplated report requirements:

Refer to section 6.0 (Deliverables) of the Draft PWS for report requirements.

· Contemplated dollar value of the project:

Estimated value: $ 5.5 Million.

· Contemplated place of performance:

The services to be performed under this contract shall be performed at the Contractor's facility.

Sources Sought Synopsis

APR 2019