Attachment 2. ECMO_Attachments_A___D.pdf

Attachment A - Key Acronyms and Definitions

Acronyms

Acronym Acronym Definition AD Application Development API Application Program Interface BFD Business Focused Delivery BPA Blanket Purchase Agreement CFR Code of Federal Regulation CI Criminal Investigation CLIN Contract Line-Item Number CO Contracting Officer CI/CD Continuous Interaction/Continuous Deployment COR Contracting Officer’s Representative CPARS Contractor Performance Assessment Reporting System CUI Controlled Unclassified Information DUA Data Usage Agreement EA Enterprise Architecture ECM Enterprise Case Management ECMO Enterprise Case Management Office EIT Electronic & Information Technology ELC Enterprise Life Cycle EO Executive Order FBI Federal Bureau of Investigation FIPS Federal Information Processing Standards GFE Government-Furnished Equipment GFI Government-Furnished Information GSA General Services Administration HSPD-12 Homeland Security Presidential Directive 12 IRS Internal Revenue Service IRA Inflation Reduction Act IT Information Technology LB&I Large Business & International MBI Minimum Background Investigation MOU Memorandum of Understanding NIST National Institute of Standards and Technology NTE Not-To-Exceed OCI Organizational Conflict of Interest O&M Operations & Maintenance OMB Office of Management and Budget OneSDLC One Solution Delivery Life Cycle OSP Office of Security Programs Pii Personally Identifiable Information PIV Personal Identity Verification PM Program/Project Manager PMO Program Management Office QA Quality Assurance

Acronym Acronym Definition QCP Quality Control Plan SAT System Acceptance Testing SB/SE Small Business/Self Employed SBU Sensitive, but Unclassified SDLC Solution Delivery Life Cycle SIT System Integration Testing SME Subject Matter Expert SLA Service Level Agreement SOC Standard Occupational Classification SOO Statement of Objectives SOP Strategic Operating Plan T4 Tier 4 TAS Taxpayer Advocate Service TEGE Tax Exempt & Government Entities TD P 15-71 Treasury Security Manual UAT User Acceptance Testing UI/UX User Interface/User Experience W&I Wage & Investment

IRS acronyms in addition to those included in the SOO can be located at the below link:

Additional IRS Acronyms

Definitions

Low-Code/No-Code Platform: A software development environment that allows users to create applications without having to write code. This can be done by using a graphical user interface (GUI) to drag and drop components, or by using a visual programming language. This definition is platform agnostic. Some historical examples of low-code/no-code platforms utilized to provide capability to the IRS are Palantir, Pega Systems, Power Apps, Salesforce, ServiceNow, and UIPath.

Stakeholder: Unless otherwise stated at the task order level, Stakeholder refers to internal IRS Business and IT Stakeholders.

Attachment B – References

Listed below are the references that apply to the work to be performed. The Title and Date are current as of the date of award. The Contractor shall be responsible for complying with updates to these references and any changes these may have to the work performed on this contract.

Title/Date How to Obtain IRM 1.15, Records and Information Management March 7, 2024 https://www.irs.gov/irm/part1/ir m_01-015-001

IRM 2.17.1, Infrastructure Currency, Infrastructure Currency Policy for Software October 26, 2020 https://www.irs.gov/irm/part2/ir m_02-017-001

IRM 2.22.1 Unified Work Request (UWR) Process February 23, 2024 https://www.irs.gov/irm/part2/ir m_02-022-001

IRM 2.25.2, IRS Portal and Extranet Usage Standard February 26, 2018 https://www.irs.gov/irm/part2/ir m_02-025-002

IRM 2.31.1, One Solution Delivery Life Cycle Guidance January 23, 2023 https://www.irs.gov/irm/part2/ir m_02-031-001

IRM 2.142.2, Service Catalog Management Directive February 07, 2024 https://www.irs.gov/irm/part2/ir m_02-142-002

IRM 2.148.2, Incident Management Process July 26, 2023 https://www.irs.gov/irm/part2/ir m_02-148-002.html

IRM 2.150.2 Configuration and Change Management, Configuration Management (CM) Process April 2, 2024 https://www.irs.gov/irm/part2/ir m_02-150-002.html

IRM 10.8 Information Technology (IT) Security December 12, 2023 https://www.irs.gov/irm/part10/ irm_10-008-001r

IRM 10.8.1, IT Security Policy and Guidance December 12, 2023 https://www.irs.gov/irm/part10/ index.html

IRM 10.5.2, Privacy and Information Protection, Privacy Compliance and Assurance (PCA) Program January 24, 2020 https://www.irs.gov/irm/part10/ irm_10-005-002.html

IRM 10.8.21, Information Technology (IT) Security, Database Security Policy November 9, 2023 https://www.irs.gov/irm/part10/ index.html

IRM 10.8.22, Information Technology (IT) Security, Web Server Security Policy November 3, 2023 https://www.irs.gov/irm/part10/ index.html

IRM 10.8.24, Information Technology (IT) Security, Cloud Computing Security Policy February 2, 2024 https://www.irs.gov/irm/part10/ index.html

IRS IRM 10.8.50, Information Technology (IT) Security, Enterprise Incident, Vulnerability, and Security Patch Management March 11, 2024 https://www.irs.gov/irm/part10/ index.html

IRM 10.8.60 Information Technology (IT) Security, IT Service Continuity Management (ITSCM) Policy and Guidance July 16, 2021 https://www.irs.gov/irm/part10/ irm_10-008-060r

Title/Date How to Obtain IRM 10.8.27 Information Technology (IT) Security, Personal Use of Government Furnished Information Technology Equipment and Resources November 3. 2023 https://www.irs.gov/irm/part10/ irm_10-008-027

IRM 10.8.62 Information Technology (IT) Security, Information System Contingency Plan (ISCP) and Disaster Recovery (DR) Test, Training, and Exercise (TT&E) Process January 18, 2024 https://www.irs.gov/irm/part10/ irm_10-008-062

IRM 10.23.2 Personal Security, Contractor Investigations April 22, 2022 https://www.irs.gov/irm/part10/ irm_10-023-002

IRM 10.23.3 Personnel Security/Suitability for Employment and Personnel Security Operations May 09, 2019 https://www.irs.gov/irm/part10/ irm_10-023-003

IRM 21.1.3 Accounts Management and Compliance Services Operations, Operational Guidelines Overview September 5, 2023 https://www.irs.gov/irm/part21/ irm_21-001-003.html

IRS Directives and Memorandums:

Title How to Obtain IRS Publication 4812 Contractor Security & Privacy Controls https://www.irs.gov/pub/irs-pdf/p4812.pdf

Publication 1075Tax Information Security Guidelines For Federal, State and Local Agencies https://www.irs.gov/pub/irs-pdf/p1075.pdf

PMBoK Requirements:

PMI’s A Guide to the Project Management Body of Knowledge (PMBOK® Guide), Fifth Edition (ISBN 9781930699458) http://www.pmi.org/PMBOK- Guide-and-Standards.aspx

PMI’s The Standard for Program Management, Third Edition (ISBN-13: 9781935589686) http://marketplace.pmi.org/Pag es/ProductDetail.aspx?GMPro duct=00101388801

Project Management Institute’s (PMI’s) Government Extension to the PMBOK® Guide, Third Edition (ISBN 9781930699915) http://marketplace.pmi.org/Pag es/ProductDetail.aspx?GMPro duct=00100082501

Office of Management and Budget (OMB):

OMB Circular A-130, Managing Information as a Strategic Resource https://www.federalregister.go v/articles/2016/07/28/2016- 17872/revision-of-omb-circular-no-a-130-managing-information-as-a-strategic-resource

OMB M-07-16, Safeguarding Against and Responding to the Breach of Personally Identifiable Information https://www.whitehouse.gov/w p-content/uploads/legacy_drupa l_files/omb/memoranda/2007/ m07-16.pdf

OMB M-14-03, Enhancing the Security of Federal Information and Information Systems https://www.whitehouse.gov/w p-content/uploads/legacy_drupa l_files/omb/memoranda/2014/ m-14-03.pdf

OMB M-15-01, Fiscal Year 2014-2015 Guidance on Improving Federal Information Security and Privacy Management Practices https://www.whitehouse.gov/w p-content/uploads/legacy_drupa l_files/omb/memoranda/2015/ m-15-01.pdf

Federal Acts and Programs:

Affordable Care Act (ACA) Tax Provisions https://www.irs.gov/affordable

-care-act Digital Analytics Program (DAP) https://digital.gov/guides/dap/ E-Government Act of 2002 https://www.congress.gov/bill/

107th-congress/house-bill/2458

Federal Acquisition Regulation (FAR) 52.245.1 https://www.acquisition.gov/fa r/52.245-1

Federal Information Security Management Act (FISMA) https://www.dhs.gov/fisma Federal Risk and Authorization Management Program (FedRAMP) https://www.fedramp.gov/

Foreign Account Tax Compliance Act (FATCA) https://www.irs.gov/businesse s/corporations/foreign-account-tax-compliance-act-fatca

Health Insurance Portability and Accountability Act (HIPAA) of 1996 https://aspe.hhs.gov/reports/h ealth-insurance-portability-accountability-act-1996

National Archives and Records Administration (NARA) https://www.archives.gov/

National Institute of Standards and Technology (NIST) Special Publications and Manuals

NIST SP 500-267Br1, USGv6 Profile https://www.nist.gov/programs

-projects/usgv6-program NIST SP 500-281Br1, USGv6 Test Methods: General Description and Validation https://nvlpubs.nist.gov/nistpu bs/specialpublications/NIST.S P.500-281Br1.pdf

NIST SP 800-12 Rev 1, Introduction to Computer Security https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-18 Rev. 1, Guide for Developing Security Plans for Federal Information Systems https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-30 Rev. 1, Guide for Conducting Risk Assessments https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-34 Rev 1, Contingency Planning Guide for Federal Information Systems https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-37, Rev. 2, Risk Management Framework for Information Systems and Organization: A System Life Cycle Approach for Security and Privacy https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-39, Managing Information Security Risk Organization, Mission, and Information System View https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-40 Rev 4, Guide to Enterprise Patch Management Planning: Preventive Maintenance for Technology https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-53 Rev. 5, Security and Privacy Controls for Information Systems and Organizations https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-53A Rev.5, Assessing Security and Privacy Controls in Information Systems and Organizations:

Building Effective Assessment Plans https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-57 Part 2 Rev 1, Recommendation for Key Management- Part 2: Best Practices tor Key Management Organizations https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-92, Guide to Computer Security Log Management https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-115, Technical Guide to Information Security Testing and Assessment https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-122, Guide to Protecting the Confidentiality of Personally Identifiable Information (PII) https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-128, Guide for Security-Focused Configuration Management of Information Systems https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-137 Information Security Continuous Monitoring (ISCM) for Federal Information Systems and Organizations https://csrc.nist.gov/publicatio ns/sp800

NIST SP 800-144, Guidelines on Security and Privacy in Public Cloud Computing https://csrc.nist.gov/publicatio ns/sp800

NIST United States Government Configuration Baseline

(USGCB)

https://csrc.nist.gov/Projects/U nited-States-Government- Configuration- Baseline/USGCB-Content

Other

Information System Contingency Plan (ISCP) Tabletop Exercise https://www.irs.gov/irm/part10 /irm_10-008-062

Code of Federal Regulations (CFR), Title 36, Part 1194, Subpart D https://www.ecfr.gov/current/ti tle-36/chapter-XI/part- 1194/appendix- Appendix%20D%20to%20Par t%201194

Common Vulnerability and Exposure (CVE) https://www.cve.org/ Department of Homeland Security (DHS), Continuous Diagnostics and Mitigation (CDM) program https://www.dhs.gov/cdm

Earned Value Management Guide September 30, 2008 https://www.irs.gov/pub/irs-procure/tj-attachment13-evmguidev2.0.pdf

Federal Acquisition Regulations System, Subpart 34.2 - Earned Value Management System https://www.govinfo.gov/conte nt/pkg/CFR-2011-title48-vol5/xml/CFR-2011-title48-vol5-part1034-subpart34- 2.xml

Treasury Acquisition Bulletin 15-03 March 18, 2015 https://home.treasury.gov/syst em/files/281/Acquisition- Bulletin-%28AB%29-15- 03.pdf

Treasury Department Security Manual, TDP 15-71 https://home.treasury.gov/abo ut/general-information/orders-and-directives/numerical-index-of-treasury-directives

Treasury Directive 15-71 https://home.treasury.gov/abo ut/general-information/orders-and-directives/treasury-directive-15-71

Treasury Security Manual (TSM) 85-01 https://home.treasury.gov/abo ut/general-information/orders-and-directives/td85-01

Federal Information Processing Standards (FIPS) Publication (PUB)

FIPS PUB 199, Standards for Security Categorization of Federal Information and Information Systems http://csrc.nist.gov/publication s/fips/fips199/FIPS-PUB-199-final.pdf

FIPS PUB 200, Minimum Security Requirements for Federal Information and Information Systems http://csrc.nist.gov/publication s/fips/fips200/FIPS-200-final-march.pdf

Attachment C – High-level Reference to the Enterprise Case Management Office, Portfolio Management and Business Focused Delivery

The slides in this deck reflect recent approaches and plans which are subject to change and refinement.

NOTE: Reference in this deck to the number of initiatives are illustrative and are expected to exceed those shown. Team quantities are for representative illustration only and do not denote the actual number of teams required for execution.

Portfolio Overview

Delivery Team 1 (Program A)

Delivery Team

2 (Program B)

Initiative A Initiative B Initiative C Initiative D Initiative E

G overnance & oversight Re po rt in g pe rf or m an ce , r is k

The portfolio provides governance and oversight to ensure integration across initiatives while maintaining a view of overall performance.

Portfolio

Manage Cross-Program / Initiative Dependencies

Delivery Team

N... (Program

N...)

BFD

POD(s)

BFD

POD(s)

BFD

POD(s)

BFD

POD(s)

BFD

POD(s)

NOTE: This is a notional structure. The number of teams and initiatives is anticipated to be more than those shown.

Director

ECMO

Delivery Team 1

BFD

Pods

Delivery Team 2

Gov Lead Ctr Lead Gov/Ctr Delivery Team

• Staffed for Rapid Delivery

• Business Driven, Enterprise Supported

• Repeatable, Flexible, Collaborative

• “Decision Maker” Enabled

• Multi-Platform/Platform Agnostic

Technical

Advisor

Business Focused Delivery Model Overview

Imbedded w/BU

SMEs

Decision Maker

Enabled

Delivery

Focused

Imbedded w/BU SMEs

Decision Maker

Enabled

Delivery

Focused

Demand/Capacity

Reuse Visibility

Surge Flexibility

IT Coordination

Demand/Capacity

Reuse Visibility

Surge Flexibility

Portfolio

Manager

IT for

Integration

Delivery Team N…

ARCH-ENG TEAM

(ARCH, DESIGN, DATA MAPPING)

COMPONENT TEAM

(ENABLER DEVELOPMENT)

FRAMEWORK TEAM

(COMMON COMPONENTS)

SYSTEMS TEAM

(DEV/OPS, CI/CD)

CYBER TEAM

(RELRS [IT], SSP, GUARDRAILS)

DATA MIGRATION TEAM

(LEGACY DATA-ECM)

IT INTEGRATION with AD

LEADERSHIP

PLATFORM* ASSURANCE

ACQUISITION / BUDGET MGT INTAKE MANAGEMENT RELEASE MANAGEMENT

PLATFORM* DELIVERY SERVICES

PLATFORM DELIVERY SERVICES

DATA / FUNCTIONAL COMMON SERVICES PLATFORM INTEGRATION

INFRASTRUCTURE SERVICES CI/CD DEPLOYMENT

OPERATIONS & MAINTENANCE

ES CYBER EST Performance EOPS IRAP UNS

SUPPORT SERVICES COORDINATION

PORTFOLIO LAYER

OUTPUTS

PLATFORM* DELIVERY TEAMS

T ec hn ic al In te gr at io n ac ro ss

D el iv er y P ar tn er s

Platform Ancillary Support Architecture Overview (note: illustrative, not complete AD organizational structure)

PLATFORM ASSURANCE- AD Integration Services responsible for coordinating the triage and development IT services functions (e.g., acquisition/budget management, demand/delivery planning of service requests through intake, resource planning, infrastructure readiness, performance management, reporting, release management etc.)

PLATFORM DELIVERY TEAMS – Service development and deployment teams responsible for architecting and designing data / common services, developing APIs and data connectors to legacy systems, deliver the enterprise level focused framework common Service components, supporting development environment, including development and maintenance of the

Continuous Delivery Pipeline, developing security plans, and migrate legacy application data to appropriate platform.

SUPPORT SERVICES COORDINATION – Coordination support across IT organizations to provide services to enable application deployment.

PLATFORM DELIVERY SERVICES – Foundational services provided by the platform including data / common service, CICD pipeline (w/ DevOps), integration of new tools, Platform upgrades, infrastructure readiness/monitoring services, O&M support (post- application transition to AD), Data Migration. Note:

Integration to be done by the vendor owning each application, including the IRSFW & IRSUX layers and their respective components as well; not only DAS APIs.

SUPPORT TEAMS

Note: Team quantity is a representative illustration only; actual number of teams required for execution will be determined by Application Development (AD)

Note: * Platform encompasses Platform elements, including data, application, and infrastructure components.

ECM Transition Business Focused Delivery (BFD) - Lifecycle

The Implementation and Deploy phases continue for 1 or many release cycles allowing customers to address change requests and enhance their applications.

Portfolio/Program/Project Management, Stakeholder Management, Monitoring, Reporting, Governance, and Documentation

Planning

• Plan/Initiate

• Onboard

Transition

• Transition

• O&M

ECM’s BFD lifecycle is a tailored delivery approach with multiple cycles of planning, defining, building, validating, deploying, and releasing increments of software until the desired future-state functionality is fully implemented and adopted by ECM Business customers.

Attachment D – Service Level Agreements (SLAs)

Tables 1.A through 1.C categorize SLAs as follows:

1. Table 1.A - BPA Level: includes SLAs that apply at the BPA level

2. Table 1.B - Minimum SLAs For Each Order: includes SLAs that apply to all call orders

3. Table 1.C - Optional SLAs For Each Order: includes SLAs the government may include and refine as needed at the call order level

The government may also include SLAs at the call order level that are not included in Tables 1.B and 1.C. Such SLAs may reflect increased efficiency, increased processing time and associated cost savings for either or both.

Individual call orders will align services/deliverables AQLs, where applicable, with Performance Work Statement (PWS) tasks.

Table 2 provides examples of potential metrics that may be adapted to call order SLAs.

Table 1.A – BPA Level

Description Performance

Standards Acceptable

Quality Levels Methods of

Surveillance Performance SLA Impact

Contractor responds timely to call order requests for proposal

100% punctuality in submitting proposals by specific due dates.

100% punctuality in submitting proposals by specific due dates.

Date and time of email or other approved mechanism for submitting proposals is at or before the date and time the proposals are due.

CPARS

Contractor responds to requests for call order proposals

90% response rate in responding to requests for call order proposals (i.e., submit a proposal for 90% of the requests for call order proposals issued).

85% response rate. For the purposes of this SLA, response includes formal no-bid decisions submitted by BPA holders.

Calculating the percentage of call order responsiveness for BPA holders.

CPARS

Small Business

Subcontracting Goal

Attainment

100% adherence to Small Business Subcontracting

Plans set forth in call orders.

100% adherence to the Small Business

Subcontracting Plans.

SLA attainment is measured by routine status reporting and other Small Business

Subcontracting Plan artifacts.

CPARS

If the Contractor performs at a level lower than that specified in the respective Table 1.B Acceptable Quality Level column, the Contractor shall submit a mitigation plan to raise their performance level to meet the respective Acceptable Quality Level.

Table 1.B - Minimum SLAs For Each Call Order

Standards

Acceptable Quality Levels

Methods of Surveillance

Performan ce SLA Impact

Contractor meets established SLAs

100% SLAs meet acceptable quality levels.

98% SLAs meet acceptable quality levels.

100% Inspection through review of

Contractor performance reports.

Contractor must deliver required deliverables within established timeframe

100% punctuality in submitting deliverables by specific due dates.

100% punctuality in submitting deliverables by specific due dates.

100% Inspection through review of deliverables on the established date.

Develop high quality deliverables and reports

100% error free deliverables. Level of quality of deliverables and reports.

No more than 5% of written deliverables are rejected for quality issues or omission of information.

Corrections, if required, are made within five business days of government review comments.

100% Inspection through review of Deliverable content.

CPARS

Transition Execution

100% successfully execute call order

Transition-In/Transition-out in accordance with the Contractor proposed and

Government proved timeframes.

98% on-time delivery and target dates/timelines met IAW the Contractor’s respective

Transition Plan and call order requirements.

100% Inspection of deliverables and adherence to target dates

/ timelines.

Table 1.C – Optional SLAs for Call Order

If the Contractor performs at a level lower than that specified in the respective Table 1.C Acceptable Quality Level column, the Contractor shall submit a mitigation plan to raise their performance level to meet the respective Acceptable Quality Level.

Acceptable

Quality Levels Methods of

Surveillance Performance SLA Impact

Timely Incident Resolution

(P1, P2, P3)

See below for percentage attainment of the target resolution time following issue submission:

(P1) 100%

Percentage of incidents receiving a resolution within 30 minutes.

(P2) 99%

Percentage of incidents working toward a resolution within four hours.

(P3) 98%

Percentage of incidents receiving a response with estimate for resolution within three days.

98% SLAs meet acceptable quality levels.

100% Inspection through review of Contractor performance reports.

Security Incident Response and

Notification from Contractor to the

Government

100% on-time and accurate

Security Incident

Response and Notification from Contractor to the Government.

100% on-time and accurate

Security Incident

Response and

Notification from

Contractor to the

Government.

100% Inspection of logs, vulnerability scans, and other security relevant data sources.

Acceptable Quality Levels

Methods of Surveillance

Performance SLA Impact

Timely Patch Management

100% of the time the BPA components are patched in accordance with the Configuration Management

Plan.

95% of the time the BPA components are patched in accordance with the

Configuration Management

Plan.

100% Inspection utilizing monitoring dashboards.

Create automated scripts to validate services.

Event Notification 100% adherence to the Communicatio ns Management

Plan.

100% adherence to the Communicati ons Management

SLA

attainment is measured by inspection of email notifications, and relevant change management records.

CPARS

Backup/ Recovery

Restoration

100% of

BPA

components are restored in accordance with the Disaster

Recovery (DR) Plan.

95% of BPA components are restored in accordance with the DR

Restore content following an incident within the BPA

Environment will be assessed using manual reporting via incident management process.

DR Execution 100% of

BPA

components are restored in accordance with the DR

Plan.

100% at 72 hours.

100% of BPA components are restored in accordance with the DR

Plan.

100% inspection of deliverables as defined in the

DR plan.

Small Business Subcontracting Goal Attainment

100% adherence to the Small Business

Subcontracting Plan as

100% adherence to the Small Business

Subcontract-ing Plan.

SLA attainment is measured by routine status reporting and other Small Business

Acceptable Quality Levels

Methods of Surveillance

Performance SLA Impact specified in the call order.

Subcontracting Plan artifacts.

Continuous Monitoring

100% of supported system(s) under the call order are compliant during periodic audit or self-assessment.

100% of supported system(s) under the call order are compliant during periodic audit or self-assessment.

100% Inspection of assessment and audit reports.

CPARS

Tested Code 100% of code delivered under the order must have substantial test code coverage and a clean code base Version-controlled, public repository of code comprising the product, which will remain in the government domain.

100% test coverage of all code.

Combination of manual review and automated testing.

CPARS

Properly Styled Code

GSA 18F Front- End Guide

0 linting errors and 0 warnings.

Combination of manual review and automated testing.

CPARS

Accessibility Web Content Accessibility

Guidelines 2.1 AA standards.

0 errors reported using an automated scanner, and

0 errors reported in manual testing.

Pa11y CPARS

Acceptable Quality Levels

Methods of Surveillance

Performance SLA Impact

Deployed 100% of code must successfully build and deploy into each environment and all higher and lower environments.

100% successfully built, tested and deployed.

100% completion of code peer review testing (manual and automated and ultimately deployed to production).

CPARS

Documented 100% applies to all of the following:

All dependencies are listed and the licenses are documented.

Major functionality in the software/source code is documented.

Individual methods are documented inline using comments that permit the use of documentation-generation tools such as JSDoc.

A system diagram is provided.

100% combination of manual review and approval.

Manual review. CPARS

Security 100% Compliant:

OWASP

Application Security

Verification Standard 4.0, Level 2.

100% of code submitted must be free of medium-and high-level static and dynamic security vulnerabilities

100% clean tests from a static testing SaaS (such as npm audit) and from OWASP ZAP, along with documentation explaining any false positives.

Acceptable Quality Levels

Methods of Surveillance

Performance SLA Impact

User Testing, Research and

Resolution

100% completion of

User Acceptance Testing and other user research methods that are compliant with the call order requirements.

100% completion of artifacts from at the end of every applicable sprint, in accordance with the call order.

Manual testing and review.

Service Provisioning and

Orchestration

100% success in delivering services in accordance with established timelines as defined and requested at the call order level.

100% success in delivering services in accordance with established timelines as defined and requested.

SLA attainment is measured by inspection of the actual service delivery artifact (e.g., email or ticket) and the established parameters for the service being delivered.

Table 2 - Performance Metrics

Description Type Frequency Defect resolution rate Difference between number of total and resolved defects divided by number of total defects

Monthly

Number of outstanding defects Difference between number of reported defects and resolved defects in each iteration by severity level and age of defect

Sprint iteration, per testing cycle

Grouping of defect concentration by source

Source such as COTS product, custom code, capability, feature, operating system

Iteration

Requirements coverage against shall statements, epics or user stories

Number of requirements completed compared to total number of requirements Number of requirements tested against completed requirements

Once per sprint or iteration

Number of hangs or crashes reported by users after release

Count (Hangs or Crashes) Monthly, after release

Number of and severity of defects found after release

Distribution of defects across severity level

Monthly, after release

Mean time between failures Count (Hours) Monthly, after release Scheduled, but unplanned downtime for maintenance and upgrades

Count (Hours) Monthly, Quarterly, Semi-annually or per release cycle

Scheduled downtime for maintenance and upgrades

Count (Hours) Monthly, Quarterly, Semi-annually or per release cycle

Backlog status (for Agile development)

Program, release, and sprint backlogs

At the end of each sprint or iteration