Attachment 1 - Test Standards Area of Interest 13 June 2024.pdf

Attachment 1 – Test Standards Area of Interests DSO CSO 1 Step Call Test Standards

Air Force Life Cycle Management Center

Kessel Run (AFLCMC/C3C) Test Standards

Areas of Interest

Prepared by

AFLCMC/C3C

Version Date: 13 June 2024

1. Background Kessel Run (KR), a division of the U.S. Air Force Life Cycle Management Center (AFLCMC), leverages industry best practices and products to rapidly deliver capabilities while posturing for future and potentially disruptive information technology (IT). Over the past few years, KR has expanded its operational scope to include three major branches, each incorporating numerous portfolios and working teams, with over 1600 personnel. KR has revolutionized the way that the U.S. Air Force builds and acquires software by: utilizing balanced teams to continuously integrate and deliver software capabilities; embracing a culture of rapid feedback loops (build, measure, learn) and user-centered design to rapidly develop, produce, and deploy software applications to support all divisions in the Air Operations Center (AOC). To continue to rapidly deliver quality software to the warfighter, mitigate risk, and ensure changes deployed to production environments are appropriately tested pre-and post deployment, Kessel Run seeks expertise to support the identification of test standards, auditing those test standards, and reporting metrics on those test standards.

2. Common Solutions Attributes The following are solutions attributes that apply to all three (3) Areas of Interest and shall be addressed. All AoIs must be addressed in any proposed solutions.

2.1. What We Are Looking For:

2.1.1. A highly opinionated Test Standards document from a vendor experienced in agile software testing

2.1.2. Vendor must utilize SECRET-level cleared individuals for this project who can access test results, bugs, Controlled Unclassified Information (CUI) and International Traffic in Arms Regulations (ITAR) information.

2.1.3. Include expertise w/ Quality Assurance (QA)1 across software development continuous integration and continuous delivery/deployment (CI/CD) pipeline

2.1.4. Include subject matter experts who are familiar with GitLab and ServiceNow.

2.1.5. Include a roadmap/implementation plan tied to a test practice baseline, dates, effort, resources, and tools that, if followed, would enable the organization to accomplish a higher-level of quality assurance maturity.

2.1.6. The work would be conducted virtually via a Government-approved virtual meeting space such as Zoom Gov or Google Meet

2.1.7. A solution that does not require new applications or tools to accomplish.

2.1.8. While we are looking for innovative solutions they must be proven to be feasible.

1 Quality assurance (QA) in software testing ensures the quality and reliability of software products. QA involves monitoring all software engineering processes, methods, and work products to ensure they meet defined standards and models. QA also aims to improve the software development process and make it more efficient and effective.

3. Area of Interest #1: Test Standards

3.1. What We Are Looking For: Kessel Run seeks Test Standards documentation and implementation plan that must fulfill the following capabilities:

3.1.1. The vendor currently has the personnel and experience to appropriately research current test practices, recommend enhancements, document short-falls, and clearly lay out an implementation plan to adhere to all identified test standards

3.1.2. Provide documentation on test standards and an implementation plan by experienced Subject Matter Experts (SMEs) w/ QA oversight of agile, DevSecOps, CI/CD software development processes.

3.1.3. Baseline test standard requirements, meaning test type descriptions, a test standard definition, and a way to audit each test standard for applications operated by Kessel Run

3.1.4. Review existing research on current testing methodologies across KR products and teams, propose methods for improving test quality and filling any testing gaps

3.1.5. Develop remaining necessary standards: Smoke Testing, System Integration Testing, End-to-End Testing, Regression Testing, Functional Testing, Performance Stress Testing, Performance Scalability Testing, Performance Volume/Flood Testing, Endurance (Soak) Testing, and any others recommended by SME; SMEs could be from the private or public sectors

3.1.6. While the vendor should at a minimum address the test types stated, the Government is open to vendor suggestions for well-developed innovative and alternatives approaches to evolve with industry best practices and standards

3.1.7. Identify test standards required to reduce risk and produce high quality code (i.e. meet industry standard outage and performance metrics)

3.1.8. Optimize existing quality/test standards in accordance with multiple Test Strategies

3.1.9. Develop methodologies for our change review process in accordance with industry best practices

3.1.10. Provide industry knowledge to integrate IT Service Management methodologies, e.g.

problem management, root cause analysis, incident management, outages, etc.

3.1.11. Provide framework for establishing organic roles and responsibilities to sustain test standards effort

3.1.12. Provide recommendations for automation of tests

4. Area of Interest #2: Test Standards Auditing

4.1. What We Are Looking For: Kessel Run seeks Test Standards Auditing documentation and implementation plan that must fulfill the following capabilities:

4.1.1. Develop methods to enhance resiliency of Kessel Run’s applications and platforms, reduce incidents/outages, ensure changes deployed to prod (platform & app) are appropriately tested pre- and post-deployment, audit/measure adherence to standards, and report on that adherence

4.1.2. Recommend best practices, roles and responsibilities, etc. for collecting audit data and reporting on adherence data

4.1.3. Recommend tools if current tools are not sufficient

4.1.4. If recommending an audit tool, must include capabilities to handle CUI, ITAR, and

Secret data, be FedRAMP compliant, and single sign-on ready

4.1.5. Apps/services that should be included as part of the Test Standard Audit are currently:

● GitLab

● Elastic Observability

● Mattermost

● JIRA

● Naval Lift

● ServiceNow

● K6

5. Area of Interest #3: Test Standards Metrics to Report

5.1. What We Are Looking For: Kessel Run seeks identification and reporting of Test Standards Metrics documentation and an implementation plan that must fulfill the following capabilities:

5.1.1. Identify metrics to collect and report that will identify if development teams are being compliant with the established test standards

5.1.2. Establish a process to monitor the reported metrics to gauge the effectiveness of the test standards implementation

5. Area of Interest #3: Test Standards Metrics to Report