Sources-Sought_Independent Validation Support Services .pdf

Sources Sought Synopsis Sample

APR 2019

Sources Sought Synopsis

The Department of Housing and Urban Development (HUD) is issuing this sources sought synopsis as a means of conducting market research to identify parties having an interest in and the resources to support this requirement for providing Independent Validation Support Services for the financial / actuarial models. The result of this market research will contribute to determining the method of procurement. The applicable North American Industry Classification

System (NAICS) code assigned to this procurement is 523920. The product service code is

R411.

There is no solicitation at this time. This request for capability information does not constitute a request for proposals; submission of any information in response to this market survey is purely voluntary; the government assumes no financial responsibility for any costs incurred.

If your organization has the potential capacity to perform these contract services, please provide the following information: 1) organization name, address, email address, website address, telephone number, and size and type of ownership for the organization; and 2) tailored capability statements addressing the particulars of this effort, with appropriate documentation supporting claims of organizational and staff capability. If significant subcontracting or teaming is anticipated in order to deliver technical capability, organizations should address the administrative and management structure of such arrangements.

The government will evaluate market information to ascertain potential market capacity to 1) provide services consistent in scope and scale with those described in this notice and otherwise anticipated; 2) secure and apply the full range of corporate financial, human capital, and technical resources required to successfully perform similar requirements; 3) implement a successful project management plan that includes: compliance with program schedules; cost containment; meeting and tracking performance; hiring and retention of personnel and risk mitigation; and 4) provide services under a performance based service acquisition contract.

Based on the responses to this sources sought notice/market research, this requirement may be set-aside for small businesses or procured through full and open competition, and multiple awards may be made. Telephone inquiries will not be accepted or acknowledged, and no feedback or evaluations will be provided to companies regarding their submissions.

Submission Instructions: Interested parties who consider themselves qualified to perform the above-listed services are invited to submit a response to this Sources Sought Notice by Friday, December 22, 2023, at 2:30 pm EST. All responses under this Sources Sought Notice must be emailed to Diane Toledo-Gaskins at Diane.E.Toledo-Gaskins@hud.gov and Dwayne Gist at mailto:Diane.E.Toledo-Gaskins@hud.gov

Dwayne.e.gist@hud.gov . Additional information on the required services is listed below.

If you have any questions concerning this opportunity, please contact: Diane Toledo-Gaskins at

Diane.E.Toledo-Gaskins@hud.gov and Dwayne Gist at Dwayne.e.gist@hud.gov .

Background:

Task A – Single-Family (MMIF)

The MMIF is a group of accounts of the federal government in which transactions associated with FHA’s guarantee programs for single family mortgages are recorded. FHA currently insures roughly 8 million forward mortgages with insurance-in-force of $1.1 trillion and 600,000 reverse mortgages with $112 billion in insurance-in-force under the Home Equity Conversion

Mortgage (HECM) program.

The intended uses of the Single-Family models to be validated are to:

A. Estimate FHA’s liability for loan guaranty for the forward and HECM programs for FHA’s financial statements.

B. Estimate selected elements of the MMIF capital ratio.

C. Develop budget estimates (credit subsidy rates and volume) and re-estimates; and, D. Estimate principal limit factors for HECM.

E. Enhance FHA’s ability to assess and manage risk, monitor trends, and evaluate policy with the capture of credit and loan application data.

Task B - Commercial (GI/SRI)

The GI/SRI is a group of accounts of the federal government in which transactions associated with FHA’s guarantee programs for Multifamily, Hospital, and Nursing Home/Assisted

Living/Board & Care facilities’ mortgages are recorded. FHA currently insures roughly 14,000 commercial mortgages with insurance-in-force of $100 billion.

The intended uses of the commercial models to be validated are to:

A. Estimate FHA’s liability for loan guaranty for the commercial loans for FHA’s financial statements.

B. Develop budget estimates (credit subsidy rates and volume) and re-estimates.

General Description of Work:

Task A – Single-Family (MMIF)

(For each task (A1 through A5) the contractor shall perform the following 8 steps as needed) mailto:Dwayne.e.gist@hud.gov mailto:Diane.E.Toledo-Gaskins@hud.gov mailto:Dwayne.e.gist@hud.gov

Step 1 Data and assumptions shall be evaluated to ensure that they are the most appropriate model inputs and that adequate controls are in place to ensure their accuracy and integrity.

Step 2 The actuarial, statistical, financial, and economic assumptions underpinning each model shall be evaluated from an empirical, analytical, and best-practice perspective to ensure they are acceptable and supportable considering the models intended business use.

Step 3 Validations shall evaluate the theory and mathematics of each model in terms of its accuracy and completeness with which the theory and mathematics replicate the actual phenomena being modeled and for logical flaws.

Step 4 Model code shall be evaluated to verify its accuracy according to industry best practices and standards, its fidelity to the corresponding theory and mathematics, its avoidance of unnecessary complexity and for processing efficiency. The Contractor will review all model code over a three-year cycle using formal verification techniques and software engineering standards.

Step 5 Performance tolerances shall be evaluated for reasonableness considering the adverse consequences of decisions based on incorrect or unreliable model outputs.

Step 6 All net present value and volume models shall be replicated for the upcoming yearly deliverables, starting with the LLG, using the most recently completed models. (In previous years, validation was done based on the model version aligned with the Budget.

This year and going forward, validation will be done on the current version of the models for the upcoming yearly deliverables, starting with the LLG.)

Step 7 Model outputs (reports) shall be evaluated for their strength at translating estimates produced by the model into clear, comprehensible, and useful business information of known accuracy and reliability.

Step 8 Such other evaluations as needed to identify all sources of and the full extent of model risk, identify model errors and other weaknesses and recommend appropriate action(s).

Detailed descriptions of specific tasks are provided in Section 5 Specific Tasks and Deliverables.

Each validation report shall include findings and/or recommendations together with an assessment of the materiality of each weakness.

(For each task (B1 through B18) the contractor shall perform the following 9 steps as needed)

Step 1 Data and assumptions shall be evaluated to ensure than they are the most appropriate model inputs and that adequate controls are in place to ensure their accuracy and integrity.

Step 2 The actuarial, statistical, financial, economic, and other theories and assumptions underpinning each model shall be evaluated from an empirical, analytical, and best-practice perspective to ensure they are acceptable and supportable considering the models intended business use.

Step 3 Validations shall evaluate the theory and mathematics of each model in terms of its accuracy and completeness with which the theory and mathematics replicate the actual phenomena being modeled and for logical flaws.

Step 4 Model code shall be evaluated to verify its accuracy according to industry best practices and standards, its fidelity to the corresponding theory and mathematics, its avoidance of unnecessary complexity and for processing efficiency.

Step 5 Internal controls on modeling processes shall be evaluated for their strength in ensuring reliable performance.

Step 6 Performance tolerances shall be evaluated for reasonableness considering the adverse consequences of decisions based on incorrect or unreliable model outputs.

Step 7 All net present value and volume models shall be replicated for the upcoming yearly deliverables, starting with the LLG, using the most recently completed models. (In previous years, validation was done based on the model version aligned with the Budget.

This year and going forward, validation will be done on the current version of the models for the upcoming yearly deliverables, starting with the LLG.)

Step 8 Model outputs (reports) shall be evaluated for their strength at translating estimates produced by the model into clear, comprehensible, and useful business information of known accuracy and reliability.

Step 9 Such other evaluations as needed to identify all sources of and the full extent of model risk, identify model errors and other weaknesses and recommend appropriate action(s).

Each validation report shall include findings and/or recommendations together with an assessment of the materiality of each weakness.

Specific Tasks

Task A – Single Family (MMIF)

For the Base Year, all three Single Family models will be validated (Forward + HECM +

TOTAL). Thereafter, the Commercial models below, plus up to 2 model validations/year for Single Family validation scopes, to be determined at the beginning of each performance year (Commercial + Forward + HECM, Commercial + Forward + TOTAL, Commercial + HECM + TOTAL).

A1 - The HECM Annual Review Agenda. FHA uses a risk-based approach to review of its single-family models that provides full coverage over a three-year cycle. Higher risk model methods and components receive higher attention than lower risk elements.

Consequently, the focus of model validation will change from year to year. FHA will work with the Contractor to develop an Annual Agenda by the beginning of each calendar year.

A2 - The FORWARD Mortgage Annual Review Agenda. FHA uses a risk-based approach to review of its single-family models that provides full coverage over a three-year cycle. Higher risk model methods and components receive higher attention than lower risk elements. Consequently, the focus of model validation will change from year to year. FHA will work with the Contractor to develop an Annual Agenda by the beginning of each calendar year.

A3 - The HECM Mortgage Insurance model is used to estimate HECM liability for loan guarantees, the HECM elements of the MMIF capital ratio, HECM budget re-estimates and the credit subsidy elements of budget development.

A4 - The FORWARD Mortgage Insurance model is used to estimate the forward liability for loan guarantees, the forward elements of the MMIF capital ratio, forward budget re-estimates and the credit subsidy elements of budget development.

A5 - The TOTAL-CHALLENGER Model (TCM) Review & Validation is used as a statistically derived algorithm developed by HUD to evaluate borrower credit history and application information. TCM is accessed through an Automated Underwriting System

(AUS) and is not an AUS itself. TCM provides a rank-ordered score of the relative risk of borrower default during the first 4 years of the mortgage resulting in claim. TCM provides AUSs with the “Accept/Refer” decision based on the borrower’s score relative to a policy cut point and other policy rules. Validation of the TOTAL model will be done in increments as information is made available. The scope of the TOTAL model validation task will be determined at the beginning of each performance year and will be developed by working with the vendor to determine specific tasks to be performed that year.

For the Base Year, all three Single Family models will be validated (Forward + HECM +

TOTAL). Thereafter, the Commercial models below, plus up to 2 model validations/year for Single Family validation scopes, to be determined at the beginning of each performance year (Commercial + Forward + HECM, Commercial + Forward + TOTAL, Commercial + HECM + TOTAL).

FHA applies 5 different types of modeling with 18 sub-categories, frameworks to the commercial (multifamily and healthcare) portfolios:

I Logistic Regression Models

II Financial Simulation Model

III Monte-Carlo Simulation Model

IV Cash Flow Model

V Recovery on Assets Model

Within the logistic regression models, there are various sub-models. The differences between the sub-models are the variables and the types of loans or Risk Categories specified for each equation. But they are all structurally the same type of model, just with different variables.

The list of the model categories and the sub-categories are:

I. Logistic Regression Models

1. Multifamily Apartment Claim Model

B1 - Unassisted New Construction/Substantial Rehabilitation Default Model

B2 - Unassisted Refinance Default Model

B3 - Assisted New Construction/Substantial Rehab. and Refinance Default Model

B4 - Claim Transition Model

2. Multifamily Apartment Prepayment Models

B5 - Unassisted New Construction/Substantial Rehabilitation Prepayment Model

B6 - Unassisted Refinance Prepayment Model

B7 - Assisted New Construction/Substantial Rehab. and Refi. Prepayment Model

3. Residential Care Facilities (Nursing Home/Assisted Living/Board & Care fac.)

Models

B8 - Section 232 New Construction/Substantial Rehabilitation Claim Model

B9 - Section 232 Refinance Claim Model

B10 - Section 232 New Construction/Substantial Rehabilitation Prepayment Model

B11 - Section 232 Refinance Prepayment Model

4. Hospital Loan Prepayment Model

B12 - Section 242 Prepayment Model

II. Hospital Loan Financial Simulation Model

B13 - Section 242 Claim Model

III. Monte-Carlo Simulation Model

B14 - Stochastic Simulation Model with data input from all the previous models.

IV. Cash Flow Models

B15 - Cash Flow Models (CFM) for Liability for Loan Guarantee (LLG)

B16 - Cash Flow Models (CFM) for Loan Loss Reserve (LLR)

V. Recovery on Assets Models

B17 – Recovery on Assets Models (ROA) for Liability for Loan Guarantee (LLG)

B18 – Recovery on Assets Models (ROA) for Loan Loss Reserve (LLR)

The tasks require a set of skill categories based on the following approximate breakdown:

Subject Matter Experts

The Contractor shall provide the minimum of two SMEs. SMEs shall have three years’ minimum experience in similar positions involving social science research and in one or more of the following fields: social science research, distressed asset disposition, servicing timelines, mortgage measurement, monitoring, mitigation and management, mortgage modeling, model management, mortgage valuation, mortgage insurance valuation, and other fields related to mortgage insurance contract servicing. SMEs must also have specific experience with validating financial and econometric models and validating statistical model coding. The SMEs should also have solid practical knowledge and experience with scientific research principles, methods and practices, as applied to social science research; including a variety of research approaches, tools, and techniques, such as qualitative and quantitative data, statistical analysis, experiments, field surveys, case research and so forth. SMEs are considered Key Personnel by the Government and shall be listed as such in accordance with HUDAR 2452.237-70, "Key Personnel”.

• Contemplated performance metrics:

Refer to section 7.0 (Performance Requirements Summary) of the Draft Performance

Work Statement (PWS)for performance metrics.

• Contemplated report requirements:

Refer to section 6.0 (Deliverables) of the Draft PWS for report requirements.

• Contemplated dollar value of the project:

Estimated value: $ 5.5 Million.

• Contemplated place of performance:

The services to be performed under this contract shall be performed at the Contractor's facility.