1-Draft PWS-Recompete Model Validation-20240604.pdf

Performance Work Statement (PWS)

Independent Model Validation

05/14/2024

Version 1.0

Vision Statement

1 General Information

1.1 Introduction

1.2 Background

1.3 Constraints

1.3.1 Project Planning and Management

1.4 Description of Services

1.5 Non-Personal Services

1.6 Period of Performance

1.7 Place of Performance

1.8 Hours of Operation

1.9 Special Qualifications

1.10 Post Award/Kickoff Conference

1.11 Status Meetings

1.12 Contractor Travel

1.13 Transition In

1.14 Transition Out

2 Definitions and Acronyms

2.1 Definitions

2.2 Acronyms

3 Government-Furnished Property and Services

3.1 Government Services

3.2 Facilities

3.3 Equipment

3.4 Materials

3.5 Quality Assurance (QA)

4 Contractor-Furnished Items and Services

4.1 Facilities

4.2 Equipment

4.3 Material

4.4 Contractor Responsibilities

4.5 Contractor Personnel

4.5.1 Special Qualifications

4.6 Identification of Contractor Employees

4.7 Quality Control

4.7.1 Quality Control Plan

5 Specific Tasks

5.1 Task A – Single Family (MMIF)

5.2 Task B – Commercial (GI/SRI)

5.3 Task C – Commercial Challenger Models

6 Deliverables

6.1 Quality Control Plan

6.2 Acceptance Criteria

7 Performance Requirements Summary

8 Related Documents

Performance Work Statement (PWS)

Independent Model Validation

Vision Statement

The vision of this requirement is to obtain contractor support for the Department of Housing and Urban

Development (HUD), Federal Housing Administration (FHA), to provide the following critical financial/actuarial models:

1. Annual validation of Single-Family Forward and HECM Models, and

2. Annual validation of Commercial Models to include Multifamily and Healthcare.

3. Annual challenger model development of a Commercial Model.

1 General Information

1.1 Introduction

The Department of Housing and Urban Development (HUD), Office of Risk Management and Regulatory

Affairs (ORMRA) is seeking contractor support to annually validate the financial / actuarial models.

Task A - Single Family Models (MMI Fund)

The purpose of Task A is to validate the Single Family Forward and Home Equity Conversion

Models.

Task B - Commercial Models (GI & SRI Fund)

The purpose of Task B is to validate the Commercial Mortgage Models

Task C - Commercial Challenger Models (GI & SRI Fund)

The purpose of Task C is to develop challenger models to the existing Commercial Mortgage Models

1.2 Background

Task A – Single-Family (MMIF)

The MMIF is a group of accounts of the federal government in which transactions associated with FHA’s guarantee programs for single family mortgages are recorded. FHA currently insures roughly 8 million forward mortgages with insurance-in-force of $1.1 trillion and 600,000 reverse mortgages with $112 billion in insurance-in-force under the Home Equity Conversion Mortgage (HECM) program.

The intended uses of the Single-Family models to be validated are to:

A. Estimate FHA’s liability for loan guaranty for the forward and HECM programs for

FHA’s financial statements.

B. Estimate selected elements of the MMIF capital ratio.

C. Develop budget estimates (credit subsidy rates and volume) and re-estimates; and

D. Estimate principal limit factors for HECM.

E. Enhance FHA’s ability to assess and manage risk, monitor trends, and evaluate policy with the capture of credit and loan application data.

Task B - Commercial (GI/SRI)

The GI/SRI is a group of accounts of the federal government in which transactions associated with

FHA’s guarantee programs for Multifamily, Hospital, and Nursing Home/Assisted Living/Board &

Care facilities’ mortgages are recorded. FHA currently insures roughly 15,000 commercial mortgages with insurance-in-force of over $100 billion.

The intended uses of the commercial models to be validated are to:

A. Estimate FHA’s liability for loan guaranty for the commercial loans for FHA’s financial statements.

B. Develop budget estimates (credit subsidy rates and volume) and re-estimates.

Task C - Commercial Challenger Models (GI/SRI) FHA recognizes the significance of constantly refining and challenging existing models to ensure optimal decision-making. The developed challenger models will provide a systematic approach to validating whether the existing Commercial models can be improved.

The purpose of the developed Commercial Challenger models are as follows:

A. Add an additional layer of scrutiny beyond the standard validation techniques, offering a more thorough assessment of model performance and assumptions.

B. Promoting continuous improvement by encouraging regular reassessment and refinement of best practice modeling techniques.

1.3 Constraints

The services identified in this PWS will adhere to the rules, regulations, laws, standards, and conventions identified by HUD as well as within the Federal Government. Constraints include the following:

Document Number Title

Released Mandatory/

Advisory 44 USC §3541 et seq. Federal Information Security Management Act (FISMA, supersedes the Computer Security Act of 1987) 2002 M

44 USC §3601 et seq. E-Government Act of 2002 2002 M Pub. L. 93-344, 2 USC §661 Federal Credit Reform Act of 1990 http://www.fms.treas.gov/ussgl/creditreform/fcratoc.html 1990 M

Pub. L. 104-106, 40 USC §1401 et seq.

Information Technology Management Reform Act of 1996 (Clinger-Cohen Act)

1996 M

Pub. L. 101-235, 12 USC

1709(f- 16) HUD Reform Act, amending the National Housing Act 1989 M

Pub. L. 105-277, 44 USC §3504 Government Paperwork Elimination Act http://www.whitehouse.gov/omb/fedreg/gpea2.html 1998 M

Pub. L. 105-220, 29 USC

701 et seq.

Section 508 of the Rehabilitation Act of 1998 1998 M

Pub. L. 97-177, 31 USC

§3901 et seq.

Prompt Payment Act 1982 M

Pub. L. 107-300, 31 USC §3321 Improper Payments Information Act of 2002 2002 M OMB Circular A-125 OMB Circular A-125, Prompt Pay http://www.whitehouse.gov/omb/circulars/index.html Latest version M

NIST Special Publication 800-

"Guide for Developing Security Plans for Federal Information Systems"

Feb 1996 A http://www.fms.treas.gov/ussgl/creditreform/fcratoc.html http://www.whitehouse.gov/omb/fedreg/gpea2.html http://www.whitehouse.gov/omb/circulars/a127/a127.html

NIST Special Publication 800-

"Security Self-Assessment Guide for Information Technology

Systems" Nov 2001 M

NIST Special Publication 800-

26, Rev. 1 "Guide for Information System Assessments and Program

Reporting " A

NIST Special Publication 800-

"Risk Management Guide for Information Technology Systems" Jul 2002 M

NIST Special Publication 800-

"Contingency Planning Guide for Information Technology

Systems" Jun 2002 M

Document Number Title Released Mandatory/

Advisory

NIST Special Publication 800-

"Guide to Information Technology Security Services" Oct 2003 M

NIST Special Publication 800-

"Guide for the Security Certification and Accreditation of Federal Information Systems"

Feb 1996 M

NIST Special Publication 800-

"Security Guide for Interconnecting Information

Technology Systems"

Aug 2002 A

NIST Special Publication 800-

"Recommended Security Controls for Federal Information

Systems"

Feb 2005 A

NIST Special Publication 800-

53, Rev. 1

"Recommended Security Controls for Federal Information

Systems"

Dec 2006 A

NIST Special Publication 800-

"Integrating Security into the Capital Planning and Investment Control Process"

Jan 2005 A

NIST Special Publication 800-

"Guide to Test, Training and Exercise Programs for IT

Plans and Capabilities"

Sep 2006 M

HUD Handbook 2400.25, Rev. 1

(or latest revision)

Information Technology Security Policy May 2005 M

HUD Handbook 1325.01, REV-

Privacy Act Handbook Sept 1995 M

HUD Enterprise Architecture Policy www.hud.gov/offices/cio/ea/index.cfm

Current version

M

HUD Cost/Benefit Analysis Methodology, Volume I –

Methodology

Current version

M

HUD Cost/Benefit Analysis Methodology, Volume II –

Workbook

Current version

A

‘Project Leader Help Guide’ http://hudatwork.hud.gov/po/i/it/sd/guide/guide.cfm

Current version

A

Constraints include the following:

• Compliance with all applicable aspects of OMB Circulars (A-11, A-123, A-127 and A-129 (OMB

Circulars).

• Compliance with all applicable Federal Accounting and Audit Standards (FASAB Handbook).

• Model risk management in accordance with principles and standards established by the Federal

Housing Finance Agency (AB-2013-07-model-Risk-Management-Guidance) et seq. and The

Federal Reserve Board and The Comptroller of the Currency (Supervisory Guidance on model

Risk Management) et seq.

Additional constraints include the following:

• Federal Managers Financial Integrity Act of 1982 (FMFIA) - https://www.whitehouse.gov/wp-content/uploads/2018/06/a11.pdf

• Digital Accountability and Transparency Act (DATA) Schema Model/Data Act Exchange

Standard website - http://fedspendingtransparency.github.io/

• Government Accountability Office (GAO) Standards for Internal Control in the Federal

Government (Green Book) - http://www.gao.gov/assets/670/665712.pdf http://www.hud.gov/offices/cio/ea/index.cfm http://hudatwork.hud.gov/po/i/it/sd/guide/guide.cfm https://www.whitehouse.gov/wp-content/uploads/2018/06/a11.pdf https://www.whitehouse.gov/wp-content/uploads/2018/06/a11.pdf http://fedspendingtransparency.github.io/ http://www.gao.gov/assets/670/665712.pdf

• Improper Payments Elimination and Recovery Improvement Act of 2012 (IPERIA) -https://www.gpo.gov/fdsys/pkg/PLAW-112publ248/content-detail.html

• Section 508 of the Rehabilitation Act of 1998 – http://www.gpo.gov/fdsys/pkg/PLAW-

105publ220/pdf/PLAW-105publ220.pdf

• Office of Management and Budget (OMB) Memoranda -https://www.whitehouse.gov/omb/information-for-agencies/memoranda

• Chief Financial Officer’s (CFO) Act of 1990 – https://www.gao.gov/products/afmd-12.19.4

Compliance with IT Security Requirements - The Contractor shall maintain compliance with all current and future Federal IT security requirements for all data, databases, artifacts, processes, and procedures that the contractor manages, develops, modifies, enhances, releases, and/or upgrades. In order to do this, the Contractor shall:

• Maintain IT security and comply with all terms and conditions of the contract throughout the term of the contract with respect to all data and information technology requirements.

• Use, maintain, enhance, develop, and upgrade all information technology software and system documentation under this Contract in accordance with Federal Laws, best practices, and regulations. This includes, but is not limited to:

1. Federal Information Security Management Act - https://www.dhs.gov/fisma

2. The Privacy Act - https://www.archives.gov/about/laws/privacy-act-1974.html

3. The E-Government Act - https://www.archives.gov/about/laws/egov-act-section-207.html

4. The Clinger-Cohen Act - https://www.fismacenter.com/Clinger%20Cohen.pdf

5. Government Paperwork Reduction Act - https://pra.digital.gov/

6. Office of Management and Budget Circulars A-130 and A-123 -https://www.whitehouse.gov/omb/information-for-agencies/circulars/

7. Office of Management and Budget Memorandum 17-12 - MEMORANDUM FOR HEADS OF

EXECUTIVE DEPARTMENTS AND AGENCIES (archives.gov)

8. Department of Housing and Urban Development regulations, Handbooks and Policies -http://www.HUD.gov

9. GAO directives et seq – https://www.gao.gov/agencies/department-housing-and-urban-development

10. Federal Financial Management Improvement Act (FFMIA) -https://obamawhitehouse.archives.gov/omb/financial_ffs_ffmia

11. Publications from National Institute of Standards and Technology (NIST) – Search | CSRC

(nist.gov)

12. Federal Records and Retention Acts and Regulations, and

13. Freedom of Information Act (FOIA) Laws and Regulations

• Follow HUDs Project Planning and Management (PPM) Life Cycle and industry best practices in the analysis, design, development, testing, and implementation of proposed new systems and/or the enhancement to existing systems. This includes prohibiting live data from being used in any environment other than Production and Disaster Recovery (DR) environments. Specifically, no live data should be used in development, testing, or staging environments.

• Ensure each financial or mixed financial system that the contractor manages, develops, modifies, enhances, releases and/or upgrades is in compliance with the Federal Information System

Controls Audit Manual (FISCAM) methodology that include control families for both General

Computer (Security Management, Access Controls, Configuration Management, Segregation of https://www.gpo.gov/fdsys/pkg/PLAW-112publ248/content-detail.html http://www.gpo.gov/fdsys/pkg/PLAW-105publ220/pdf/PLAW-105publ220.pdf http://www.gpo.gov/fdsys/pkg/PLAW-105publ220/pdf/PLAW-105publ220.pdf https://www.whitehouse.gov/omb/information-for-agencies/memoranda https://www.gao.gov/products/afmd-12.19.4 https://www.dhs.gov/fisma https://www.archives.gov/about/laws/privacy-act-1974.html https://www.archives.gov/about/laws/egov-act-section-207.html https://www.fismacenter.com/Clinger%20Cohen.pdf https://pra.digital.gov/ https://www.whitehouse.gov/omb/information-for-agencies/circulars/ https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2017/m-17-12_0.pdf https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2017/m-17-12_0.pdf http://www.hud.gov/ https://www.gao.gov/agencies/department-housing-and-urban-development https://www.gao.gov/agencies/department-housing-and-urban-development https://obamawhitehouse.archives.gov/omb/financial_ffs_ffmia https://csrc.nist.gov/publications/sp https://csrc.nist.gov/publications/sp

Duties, and Contingency Planning) and Business Process Application controls (Application

Security, Business Process Controls, Interfaces, and Data Management).

• Ensure compliance with identified OMB A-123 Appendix A "Management's Responsibility for

Internal Control" and Appendix D "Compliance with the Federal Financial Management

Improvement Act of l996 key controls.

• Maintain Security Assessment and Authorization (SA&A) and the Federal Information Security

Management Act of 2002 (FISMA) standards in accordance with guidance published by the

National Institute of Standards and Technology (NIST); to include the following Special

Publications National Institute of Standards and Technology (NIST) Special Publication (SP):

In addition, the Contractor shall:

• Ensure business services are able to function within the Federal government’s IT protocols and requirements.

• Ensure compliance with Federal Information Processing Standards (FIPS) 199/200 Security

Categorization Analysis: FIPS Requirements for Federal Info and Info Systems

• Ensure compliance with HUD OCIO Handbooks -https://www.hud.gov/program_offices/administration/hudc

• Ensure Model Validation business services meet Federal Government Single Sign On requirements -

1. https://playbooks.idmanagement.gov/playbooks/

2. https://playbooks.idmanagement.gov/docs/playbook-sso.pdf

• Employ Secured Socket Layer (SSL) transmission protocol.

• Comply with NIST and FISMA standards to safeguard CCAP data that is classified as having a

Moderate information security baseline.

• Support dynamic registration.

• Provide HUD with the attribute/values they need in the assertion to map to the users contained in their user store.

• Provide managed, secure, and controlled access to its application for interfacing with HUD systems and users.

• Make every reasonable effort to deliver information technology products to HUD free of known computer viruses by examining all products prior to delivery to HUD using tools and processes capable of virus detection.

Failure by the Contractor to adhere to the above NIST requirements could result in penalties, to include a contract performance stop-work order until compliance can be demonstrated. Disregard of these NIST requirements could also lead to other criminal, civil, administrative, or contract penalties, including, but not limited to:

• Breach of Contract damages

• False Claims Act damages

• Termination for Default

• Termination for Convenience

• Poor Past Performance https://www.hud.gov/program_offices/administration/hudc https://playbooks.idmanagement.gov/playbooks/ https://playbooks.idmanagement.gov/docs/playbook-sso.pdf

• Suspension/debarment

Compliance with Privacy Requirements - The Contractor shall maintain compliance with OMB

Memorandum 17-12 (Preparing for and Responding to a Breach of Personally Identifiable Information), which requires contractors and sub-contractors (at any tier) to:

• Cooperate with and exchange information with agency officials, as determined necessary by the agency, in order to effectively report and manage a suspected or confirmed breach

• Properly encrypt PII in accordance with OMB Circular A-130 and other applicable policies and to comply with any agency-specific policies for protecting PII

• Provide and participate in mandatory training on how to identify and report a breach

• Report a suspected or confirmed breach in any medium or form, including paper, oral, and electronic, as soon as possible and without unreasonable delay, consistent with the agency's incident management policy and US-CERT notification guidelines

• Maintain capabilities to determine what Federal information was or could have been accessed and by whom, construct a timeline of user activity, determine methods and techniques used to access

Federal information, and identify the initial attack vector

• Allow for an inspection, investigation, forensic analysis, and any other action necessary to ensure compliance with OMB M 17-12, the agency's breach response plan, and to assist with responding to a breach

• Identify roles and responsibilities, in accordance with OMB M 17-12 and the agency's breach response plan

• Explain that a report of a breach shall not, by itself, be interpreted as evidence that the contractor or its subcontractor (at any tier) failed to provide adequate safeguards for PII

• Data Handling - The Contractor shall follow these rules when handling all data related to this requirement:

• The Contractor and its employees shall not divulge, or release data or information developed or obtained during the performance of this contract without written approval of the Contracting

Officer.

• Information contained in all source documents and other media provided by HUD is the sole property of HUD.

• Include the following statement on deliveries of hardware, software, and all data media made set forth in this contract: "This product has been scanned for known viruses using [name of virus-screening product, including version number, if any] and is certified to be free of known viruses at the time of delivery.

• Adhere to HUDs policies for safeguarding sensitive electronically formatted information.

• Ownership: Ownership and data rights of all the elements, structures and processes feeding into or produced by the databases or other automated tools developed under this contract will be the property of the government. FHA must have effective access, through a custodian designated at

FHA’s discretion, to all versions of the CCAP processes, databases, reports, data, and related documentation at the end of each business day. The Contractor is required to maintain and update the models, tools, and data in a HUD supplied secure location. As expiration of the contract approaches, the contractor must prepare and deliver all CCAP data and code to the government’s designee in a machine-readable format as agreed between the parties. The CCAP data rights should include but not be limited to all versions of the CCAP processes, databases, reports, data, and related documentation.

1.3.1 Project Planning and Management

All HUD IT projects, including all information systems acquired, developed, enhanced, or maintained shall follow the policy, procedures, standards, and guidelines set forth within the IT Management

Framework using the Project Planning and Management (PPM) Life Cycle located at HUDs PPM Life

Cycle Website:

http://portal.hud.gov/hudportal/HUD?src=/program_offices/cio/ppm/PPMV20HOME

PPM was designed based on the best practices from the Capability Maturity Model Integration (CMMI) for systems development and PMBOK for project management. While at the foundation, PPM is designed as a Waterfall methodology, HUD encourages tailoring the PPM to use the many modern solution development methodologies available to enable the planning, development, and delivery of useable functionality within 6-9-month increments.

Contractors are encouraged to propose a preferred methodology whether their solution(s) are for custom development, includes prototypes or pilots, Commercial Off the Shelf (COTS)/ Government Off the Shelf

(GOTS) configuration, or Software as a Services. Working with the HUD Project Manager(s), the

Contractor will tailor PPM to take advantage of contractor proposed expertise in using other methodologies in a manner that will provide HUD the best value and address all of the work necessary for successful project completion, on time, within budget, and delivering intended functionality.

The decisions of PPM tailoring are captured in the Project Tailoring Agreement (PTA), which documents the specific agreement for creating, combining, referring, or omitting specific artifacts applicable to the project, as well as adjust project control gate reviews to be consistent with the tailoring. All tailoring of

PPM must be approved by HUDs Deputy Chief Information Officer (DCIO) for Business & IT

Modernization. As a default, any project without an approved project tailoring agreement is required to follow all PPM phases and artifacts as presented.

The services identified in this PWS will adhere to the rules, regulations, laws, standards, and conventions identified by HUD as well as within the Federal Government. Functional constraints include the following:

1. All assumptions, data, methods and techniques used in MMIF models must comply with 12 USC

1708(a) (4), 12 USC 1711(f) (4), 12 USC 1735(f) (16), the requirements of the Federal Credit

Reform Act as implemented through OMB circulars A-11 and A-129, FASAB Handbook of

Federal Accounting Standards and Other Pronouncements, as Amended including TRs 3 and 6;

FHA’s Model Risk Management Guidelines as well as model guidance including that issued by

Federal financial regulators and OMB.

2. Model Development should begin with the precise definition of the models intended use, a thorough description of the reality to be modeled and a statement of the model’s design objective.

The model should then be tailored to achieve its design objective. The resulting model should have three major components inputs, processes, and outputs.

http://portal.hud.gov/hudportal/HUD?src=/program_offices/cio/ppm/PPMV20HOME

3. Model inputs include data, assumptions, theory / mathematics, operational logic, and specific methodology. Model theory / mathematics describe abstractly the real-world process or system that produces the outcomes being modeled by substituting variables, parameters, constants, etc.

for the real-world actions, tendencies, decisions, and events they represent. Model mathematics are the basis for developing model code and include all calculations performed during data preparation which are external to the primary model.

4. Data and assumptions must be evaluated as part of model development to document that they are supportable as model inputs. Models must be based on sufficient relevant and reliable data, if sufficient relevant and reliable data is not available, each deficiency and the range of its potential impacts on the accuracy and reliability of model outputs must be incorporated into the information used to inform decisions. Informed opinion may be used to supplement or extend available data if the basis of the stated opinion is articulated and documented adequately. All data inputs and material assumptions must be identified explicitly along with the evidence and rationale for their use and evidence that they have been approved by the model’s owner.

5. Model design, theory and logic must be supported by documentary evidence of consistency with data inputs, industry practice or published research and an assessment of alternatives leading to the selected methodologies, risk quantification techniques and specifications. Models should then then go through an iterative and documented process of fitting, testing and adjustment which culminates with a final model for which the accuracy and reliability of outputs is known and acceptable. Testing must include significance, in-sample fit, sensitivity, out-of-sample fit to determine the accuracy and reliability of model outputs, and other assessments as may be appropriate. The sensitivity of model results to each assumption must be tested to determine materiality and reasonableness.

6. Each model must have a performance tolerance, which is supported by a documented rationale, and procedures for monitoring model outputs against the performance tolerance and reviewing out-of-tolerance outcomes to identify causes and the necessity for adjustments to the model.

7. Model outputs (reports) must translate the estimates produced by the model into useful business information. Model outputs must be clear and comprehensible to decision-makers and include indications of the accuracy and reliability of the information. Reports should provide a range of estimates (including statistical confidence intervals, sensitivity analyses and alternative econometric and financial model specifications) rather than placing decision-makers in a "take-it-or-leave-it situation.

8. Model documentation should be the only input needed for each independent validation. Model documentation should include a copy of the model’s annotated computer code, procedures for version control and controlling access to the model and its computer code and a list of personnel responsible for operating and maintaining the model.

9. Model methods and code should be evaluated using formal and quantitative techniques to a maximal extent.

Model Mechanics Integrity Evaluation

Model mechanics assessments are the evaluations of risk model practices results. These evaluations include rigorous tests to assess how well model practices are meeting quality standards. These including ensuring that models are:

1. Clearly documented and that such documentation, in combination with execution methods (to include statistical coding and use of spreadsheet templates), is sufficient to allow independent qualified staff to run current and maintain future model versions.

2. Free from errors in data extraction and transformation, arithmetic, logic, and coding.

3. In complete accordance with the model’s specifications.

4. Produced by efficient means, including optimization of data management and code as well as minimization of human and computing processing times.

5. From decision rules, algorithms and other formulations that can be easily altered for testing of alternative methodologies and program terms.

6. Not statistically vulnerable due to failure to pass significance tests, selection bias, and measurement error.

7. Using econometric specifications that have been tested against alternative data specifications and projections and are not vulnerable to multi-collinearity or fragility of variable choice.

8. Inclusive of diagnostic routines that provide as standard output, statistical summaries, and decision rules to identify material changes in model results and potential errors introduced by model changes.

Task B – Commercial (GI/SRI)

1. The resulting model should have three major components inputs, processes, and outputs.

2. Model inputs include data, assumptions, theory / mathematics, operational logic, and specific methodology. Model theory / mathematics describe abstractly the real-world process or system that produces the outcomes being modeled by substituting variables, parameters, constants, etc.

for the real-world actions, tendencies, decisions, and events they represent. Model mathematics are the basis for developing model code and include all calculations performed during data preparation which are external to the primary model.

3. Data and assumptions must be evaluated as part of model development to document that they are supportable as model inputs. Models must be based on sufficient relevant and reliable data. If sufficient relevant and reliable data is not available, each deficiency and the range of its potential impacts on the accuracy and reliability of model outputs must be incorporated into the information used to inform decisions. Informed opinion may be used to supplement or extend available data if the basis of the stated opinion is articulated and documented adequately. All data inputs and material assumptions must be identified explicitly along with the evidence and rationale for their use and evidence that they have been approved by the models’ owner.

4. Model design, theory and logic must be supported by documentary evidence of consistency with data inputs, industry practice or published research and an assessment of alternatives leading to the selected methodologies, risk quantification techniques and specifications. Models should then then go through an iterative and documented process of fitting, testing and adjustment which culminates with a final model for which the accuracy and reliability of outputs is known and acceptable. Testing must include significance, in-sample fit, sensitivity, out-of-sample fit to determine the accuracy and reliability of model outputs, and other assessments as may be appropriate. The sensitivity of model results to each assumption must be tested to determine materiality and reasonableness.

5. Model development must include internal controls on modeling processes to ensure reliable performance. These controls must include: adequate reconciliation of model inputs and outputs to source data systems; verification that computer code is accurate to the models’ theory, mathematics, methodology and logical flow; end-user computing controls including version control, change control, data integrity control, access control and availability control; procedures for operating, maintaining, and updating the model and verifying outputs.

6. Each model must have a performance tolerance, which is supported by a documented rationale, and procedures for monitoring model outputs against the performance tolerance and reviewing out-of-tolerance outcomes to identify causes and the necessity for adjustments to the model.

7. Model outputs (reports) must translate the estimates produced by the model into useful business information. Model outputs must be clear and comprehensible to decision-makers and include indications of the accuracy and reliability of the information. Reports should provide a range of estimates rather than placing decision-makers in a "take-it-or-leave-it” situation.

8. Model documentation should be the only input needed for each independent validation.

Model documentation should include a copy of the models annotated computer code, procedures for version control and controlling access to the model and its computer code and a list of personnel responsible for operating and maintaining the model.

Task C – Commercial Challenger Models (GI/SRI)

1. Perform requirements gathering such that key stakeholders are consulted including Office of

Risk Management or other program areas as needed. Define key performance indicators, risk metrics, and modelling objectives to guide the development process.

2. Identify and collect relevant data sources, including historical loan performance data, available third-party market data, economic variables, and borrower characteristics. Clean, preprocess and standardize the data to ensure consistency and accuracy in modeling.

3. Develop challenger models using advanced statistical and machine learning techniques.

Explore alternative modeling methodologies, including regression analysis, decision trees, neural networks, and ensemble methods, to capture complex relationships and uncertainties in loan performance.

4. Conduct scenario analysis to simulate various economic and market conditions, such as changes in interest rates, economic downturns, or sector-specific shocks. Use the challenger models to assess the resilience of the existing loan portfolio and evaluate the impact of different scenarios on loan performance.

5. Validate the challenger models against historical data and benchmark them against the existing suite of commercial models. Evaluate the accuracy, robustness, and predictive power of the challenger models through back testing and out-of-sample testing procedures.

6. Document the methodology, assumptions, and findings of the challenger models in a comprehensive report. Provide clear explanations and interpretations of model outputs, including key risk drivers, sensitivity analyses, and model limitations. Present the findings to the Office of Risk Management staff in a clear and concise manner.

1.4 Description of Services

HUD seeks a Contractor to provide Independent Validation support services for the financial / actuarial models. Each model validation and verification report will address the following elements at a minimum:

(For each task the contractor shall perform the following 8 steps as needed)

Step 1 Data and assumptions shall be evaluated to ensure that they are the most appropriate model inputs and that adequate controls are in place to ensure their accuracy and integrity.

Step 2 The actuarial, statistical, financial, and economic assumptions underpinning each model shall be evaluated from an empirical, analytical, and best-practice perspective to ensure they are acceptable and supportable considering the models intended business use.

Step 3 Validations shall evaluate the theory and mathematics of each model in terms of its accuracy and completeness with which the theory and mathematics replicate the actual phenomena being modeled and for logical flaws.

Step 4 Model code shall be evaluated to verify its accuracy according to industry best practices and standards, its fidelity to the corresponding theory and mathematics, its avoidance of unnecessary complexity and for processing efficiency. The Contractor will review all model code over a three-year cycle using formal verification techniques and software engineering standards.

Step 5 Performance tolerances shall be evaluated for reasonableness considering the adverse consequences of decisions based on incorrect or unreliable model outputs.

Step 6 All net present value and volume models shall be replicated for the upcoming yearly deliverables, starting with the LLG, using the most recently completed models. (In previous years, validation was done based on the model version aligned with the

Budget. This year and going forward, validation will be done on the current version of the models for the upcoming yearly deliverables, starting with the LLG.)

Step 7 Model outputs (reports) shall be evaluated for their strength at translating estimates produced by the model into clear, comprehensible, and useful business information of known accuracy and reliability.

Step 8 Such other evaluations as needed to identify all sources of and the full extent of model risk, identify model errors and other weaknesses and recommend appropriate actions.

Detailed descriptions of specific tasks are provided in Section 5 Specific Tasks and Deliverables.

Each validation report shall include findings and/or recommendations together with an assessment of the materiality of each weakness.

Task B - Commercial (GI/SRI)

(For each task the contractor shall perform the following 9 steps as needed)

Step 1 Data and assumptions shall be evaluated to ensure that they are the most appropriate model inputs and that adequate controls are in place to ensure their accuracy and integrity.

Step 2 The actuarial, statistical, financial, economic, and other theories and assumptions underpinning each model shall be evaluated from an empirical, analytical, and best-practice perspective to ensure they are acceptable and supportable considering the models intended business use.

Step 3 Validations shall evaluate the theory and mathematics of each model in terms of its accuracy and completeness with which the theory and mathematics replicate the actual phenomena being modeled and for logical flaws.

Step 4 Model code shall be evaluated to verify its accuracy according to industry best practices and standards, its fidelity to the corresponding theory and mathematics, its avoidance of unnecessary complexity and for processing efficiency.

Step 5 Internal controls on modeling processes shall be evaluated for their strength in ensuring reliable performance.

Step 6 Performance tolerances shall be evaluated for reasonableness considering the adverse consequences of decisions based on incorrect or unreliable model outputs.

Step 7 All net present value and volume models shall be replicated for the upcoming yearly deliverables, starting with the LLG, using the most recently completed models. (In previous years, validation was done based on the model version aligned with the Budget. This year and going forward, validation will be done on the current version of the models for the upcoming yearly deliverables, starting with the LLG.)

Step 8 Model outputs (reports) shall be evaluated for their strength at translating estimates produced by the model into clear, comprehensible, and useful business information of known accuracy and reliability.

Step 9 Such other evaluations as needed to identify all sources of and the full extent of model risk, identify model errors and other weaknesses and recommend appropriate action(s).

Step 10 A challenger model shall be developed and used as the basis for a complete commercial model validation in either Year 2 or Year 4.

Each validation report shall include findings and/or recommendations together with an assessment of the materiality of each weakness.

Task C - Commercial Challenger Models (GI/SRI)

(For each task the contractor shall perform the following steps as needed)

Step 1 Define objectives by clearing defining the objectives of the challenger model and its role in validating a commercial model. Determine the specific aspects of the existing model that need validation, such as accuracy, robustness, or predictive power.

Step 2 Gather relevant data needed for model development and validation. Ensure that the data is representative of the underlying processes and conditions that the models seek to capture.

Clean, preprocess, and transform the data as necessary to ensure consistency and quality.

Step 3 Select appropriate modeling techniques and methodologies for developing the challenger model. Consider alternative approaches that may provide insights into different aspects of the problem or offer improvements over the existing model.

Step 4 Build the challenger model using the selected techniques and methodologies. Train the model on historical data, adjusting parameters and features as needed to optimize performance.

Implement robust validation procedures, such as cross-validation, or out-of-sample testing, to assess the accuracy and generalization of the challenger model.

Step 5 Compare the performance of the challenger model with that of the existing model using relevant metrics and evaluation criteria. Identify areas of agreement between the two models, focusing on difference in predictions, biases, and error patterns. Conduct sensitivity analyses to understand the impact of key variables and assumptions on model outputs.

Step 6 Validate the challenger model against additional datasets or validation sets to ensure its reliability and generalizability. Assess the stability of model outputs over time and across different contexts to confirm its robustness. Address any discrepancies or inconsistencies between the challenger model and the existing model through further investigation and refinement.

Step 7 Document the development process, methodology, and findings of the challenger model in a comprehensive report. Provide clear explanations of the validation results, highlighting areas of improvement and implications for decision making.

1.5 Non-Personal Services

The Government will neither supervise Contractor employees nor control the method by which the

Contractor performs the required tasks. Under no circumstances shall the Government assign tasks to, or prepare work schedules for, individual Contractor employees. It shall be the responsibility of the

Contractor to manage its employees and to guard against any actions that are of the nature of personal services or give the perception of personal services.

If the Contractor believes that any actions constitute, or are perceived to constitute personal services, it shall be the Contractor's responsibility to notify the Contracting Officer (CO) immediately. These services shall not be used to perform work of a policy, decision making, or management nature, i.e. inherently

Government functions. All decisions relative to programs supported by the Contactor shall be the sole responsibility of the Government.

1.6 Period of Performance

The Period of Performance shall be for one 12-month base period and four 12-month option periods.

1.7 Place of Performance

The services shall be performed at the Contractor's facility.

1.8 Hours of Operation

The Contractor is responsible for providing services between the hours of 8:00 am to 5:00 pm EST, Monday thru Friday except for Federal holidays or when the Government facility is closed due to local or national emergencies, administrative closing, or similar Government directed facility closings. Weekly hours shall not exceed a forty (40) hour work week and a typical workday will be 8 hours each day

Monday through Friday. The Government reserves the right to change hours of operation or restrict contractor access. Work outside of these daily hours is prohibited without Contracting Officer approval. Government agencies will not be available during scheduled holidays, inclement weather, weekends, and after duty hours.

The Contractor shall always maintain an adequate workforce for the uninterrupted performance of all tasks defined within the contract when the Government facility is not closed for the above reasons. When hiring personnel, the Contractor shall keep in mind that the stability and continuity of the workforce is essential.

1.9 Special Qualifications

Subject Matter Experts

The Contractor shall provide the minimum of two SMEs. SMEs shall have three years’ minimum experience in similar positions involving social science research and in one or more of the following fields: social science research, distressed asset disposition, servicing timelines, mortgage measurement, monitoring, mitigation and management, mortgage modeling, model management, mortgage valuation, mortgage insurance valuation, and other fields related to mortgage insurance contract servicing. SMEs must also have specific experience with validating financial and econometric models and validating statistical model coding. The SMEs should also have solid practical knowledge and experience with scientific research principles, methods and practices, as applied to social science research; including a variety of research approaches, tools, and techniques, such as qualitative and quantitative data, statistical analysis, experiments, field surveys, case research and so forth. SMEs are considered Key Personnel by the Government and shall be listed as such in accordance with HUDAR 2452.237-70, "Key Personnel”.

1.10 Post Award/Kickoff Conference

The Contractor shall attend any post award conference convened by the contracting activity or contract administration office in accordance with FAR Subpart 42.5. The Government intends to convene a Post

Award Conference with the Contractor within ten business days after contract award. The Contracting

Officer will notify the Contractor of the specific date, location, and agenda within five business days after contract award.

1.11 Status Meetings

The Contracting Officer, Government Technical Representative (GTR) and other Government personnel, as appropriate may meet periodically with the Contractor to also review Contractor performance, requirement status, etc. At these meetings, the Contracting Officer will apprise the Contractor of how the

Government views the Contractor's performance or progress of the requirement. The Contractor will apprise the Government of problems, if any, being experienced. Appropriate action shall be taken to resolve outstanding issues. These meetings shall be at no additional cost to the Government. Post Award

Conference and subsequent meetings may be held via teleconference.

1.12 Contractor Travel

Prior to travel, the Contractor shall coordinate with and receive Government authorization from the GTR for all travel. Reimbursement of travel costs will be in accordance with the Federal Travel Regulation and in accordance with FAR 31.205-46. The Contractor shall travel using the lower cost mode transportation commensurate with the mission requirements. When necessary to use air travel, the

Contractor shall use the tourist class, economy class or similar lodging accommodations to the extent they are available and commensurate with the mission requirements. HUD will not reimburse Contractor's local travel. Local travel is defined as travel within fifty (50) miles of Washington, DC. All other travel will be reimbursed on a cost reimbursable basis; no profit or fee will be paid.

1.13 Transition In

The Contractor shall provide 30 days of transition-in time if required.

1.14 Transition Out

The Contractor shall provide 30 days of transition out time if required.

2 Definitions and Acronyms

2.1 Definitions

Business/Workdays - Every official workday of the week which are days between and including

Monday to Friday. This does not include public holidays and weekends.

Calendar Day - Any day of the week.

Contractor - A supplier or vendor awarded a contract to provide specific supplies or service to the

Government. The term used in this contract refers to the prime.

Contracting Officer (CO) - A person with authority to enter into, administer, and/or terminate contracts and make related determinations and findings on behalf of the Government. Note: the only individual who can legally bind the Government.

Defective Service - A service output that does not meet the standard of performance associated with the Performance Work Statement.

Deliverable Anything that can be physically delivered but may include non-manufactured things such as meeting minutes or reports.

Government Furnished Property (GFP) - Government-furnished property means property in the possession of, or directly acquired by, the Government and subsequently furnished to the Contractor for performance of a contract. Government-furnished property includes, but is not limited to, spares and property furnished for repair, maintenance, overhaul, or modification. Government-furnished property also includes contractor-acquired property if the contractor-acquired property is a deliverable under a cost contract when accepted by the Government for continued use under the contract.

Secondary-Contracting Officer’s Representative (S-COR) - An individual designated by the

Contracting Officer to assist in providing technical direction and monitoring performance under the contract.

Primary-Contracting Officer’s Representative (P-COR) - An employee of the U.S. Government appointed by the Contracting Officer to perform contract administration activities regarding technical issues. This individual has authority to provide technical direction to the Contractor if direction is within the scope of the contract, does not constitute a change and has no funding implications. This individual does NOT have authority to change the terms and conditions of the contract.

Performance Requirements Summary (PRS) - A listing of the performance requirements under the contract that are to be evaluated by the Government on a regular basis, performance indicators for these requirements, performance standards for these requirement and surveillance methods to be used to determine if performance standards are met.

Performance Standard - The Contractor's performance level required by the Government.

Performance Work Statement (PWS) - A statement of work for performance-based acquisitions that describe the required results in clear, specific, and objective terms with measurable outcomes.

Physical Security - Actions that prevent the loss or damage of Government property.

Quality Assurance (QA) - Policies and procedures adopted by the Government to ensure that supplies and services acquired under Government contracts conform to the contracts quality requirements.

Quality Assurance Surveillance Plan (QASP) - A plan describing how the agency will survey, observe, test, sample, evaluate and document the Contractor's performance in meeting critical performance standards identified in the contract.

Quality Control (QC) - All necessary measures taken by the Contractor to assure that the quality of an end product of service shall meet contract requirements.

Service Contract - A contract that directly engages the time and effort of a Contractor whose primary purpose is to perform an identifiable task rather than to furnish an end item of supply. Subcontractor -

Any person, other than the prime Contractor, who offers to furnish or furnishes any supplies, material, equipment, or services of any kind under a prime contract or a subcontract entered into in connection with such prime contract, and any person who offers to furnish or furnishes general supplies to the prime contractor or a higher tier subcontractor. The Government does not have privity of contract with a subcontractor.

Work Week - Monday through Friday, unless specified otherwise.

2.2 Acronyms

AQL - Acceptable Quality Level

BAA - Business Area Analysis Study

CFO or OCFO - Office of the Chief Financial Officer of HUD

CFR - Code of Federal Regulations

CIO or OCIO - Office of the Chief Information Officer of HUD

CO - Contracting Officer

COR - Contracting Officer Representative

CPO - Office of the Chief Procurement Officer of HUD

GAO - U.S. General Accounting Office

S-COR – Secondary-Contracting Officer’s Representative

P-COR – Primary- Contracting Officer’s Representative

HUD - U.S. Department of Housing and Urban Development

HUDAR - HUD Acquisition Regulation

HUD/ISG - Internet Services Group within the Telecom…