USAF_Weapon_System_Sofware_Management_Guide.pdf

United States Air Force

Weapon Systems

Software Management

Guidebook

Secretary of the Air Force for Acquisition

(SAF/AQ)

15 August 2008

Version 1 (Abridged)

DISTRIBUTION STATEMENT A. Approved for public release, distribution is unlimited.

NOTE: This abridged version is published with all links to secure (https) web sites removed.

USAF Weapon Systems Software Management Guidebook

Table of Contents

1 Introduction 2 Background

2.1 Why We Have ―Software‖ Problems

2.1.1 Planning Based on Unrealistic Expectations

2.1.2 Desired Software Performance is a Moving Target

2.1.3 Software Risks not Thoroughly Identified and Effectively Managed

2.1.4 Software Development Teams not Adequately Staffed, Stable, or Capable

2.1.5 Effective Software Processes not Established or Consistently Applied

2.1.6 Ineffective Systems Engineering Interface to the Software Development Process 9

2.1.7 Lack of Effective Management and Insight into Software Development

2.2 Air Force Software Policy

2.3 Mapping Policy to this Guidebook

3 Software Process Guidelines for Air Force Acquisition Organizations

3.1 Software Aspects of Acquisition Program Planning

3.1.1 Planning for Program Start

3.1.2 Planning for Acquisition/Development Program Execution

3.1.3 Planning for Post Deployment Support

3.1.4 Software Planning in the Systems Engineering Plan

3.1.5 Sources of Additional Guidance

3.2 Estimating Software Size, Effort and Schedule

3.2.1 High Confidence Program Criteria

3.2.2 Software Estimation Risks

3.2.3 Software Estimation Process Considerations

3.2.4 Estimating Reuse Feasibility

3.2.5 Realistic Program Baselines for Software

3.2.6 Sources of Additional Guidance

3.3 Management of Software Related Risks

3.3.1 Standard Software Risks

3.3.2 Other Typical Software Related Risks

3.3.3 Sources of additional guidance:

3.4 Source Selection Considerations

3.4.1 Software-Related Content in the Request for Proposal

3.4.2 Activities During Source Selection

3.4.3 Using Maturity Models to Evaluate Offeror Capability

3.4.4 Sources of Additional Guidance

3.5 Applying Earned Value Management to Software

3.5.1 Sources of Additional Guidance

3.6 Establishing and Managing Software Requirements

3.6.1 Requirements Basics

3.6.2 Requirements and Incremental Software Development

3.6.3 Software Size Control

3.7 Acquisition Insight and Involvement

3.7.1 Software Metrics

3.7.2 Qualitative Indicators

3.7.3 Involvement in Development Activities

3.7.4 Software Resources Data Report

3.7.5 Sources of Additional Guidance for SRDR

3.8 Safety Critical Systems

3.8.1 Sources of Additional Guidance

3.9 Non-Developmental Software

3.9.1 General Considerations for NDS

USAF Weapon Systems Software Management Guidebook TOC

3.9.2 Government Furnished Software

3.9.3 Open Source Software

3.10 Software Assurance and Anti-Tamper Protection

3.10.1 Software Assurance

3.10.2 Anti-Tamper Protection

3.11 Configuration Management

3.12 Life Cycle Support

3.12.1 Software Support Planning

3.12.2 Software Product Engineering Data

3.12.3 Establish System/Software Engineering Environments and Associated Facilities 71

3.12.4 Viable Life-Cycle Computer Systems and Software

3.12.5 Computer Programming Language Selection

3.13 Lessons Learned

Appendix A Software in the Integrated Master Plan Appendix B Software Content for the Statement of Objectives (SOO) and Statement of Work

(SOW)

Appendix C Example Software Content for RFP Section L Appendix D Example Software Content for RFP Section M Appendix E Software Contracting Considerations Appendix F Computer Systems and Software Criteria for Technical Reviews Appendix G Process Considerations for Safety Critical Systems Appendix H Air Force Core Software Metrics Appendix I Software Development Plan Appendix J Glossary of Supporting Information

FIGURE 1. System/Software Requirements Flowdown FIGURE 2. Software Cost Estimation Accuracy vs Phase FIGURE 3. Factors Affecting Software Estimate FIGURE 4. ASC Embedded Software Size Growth FIGURE 5. COTS/Reuse Assessment Attributes FIGURE 6. Reuse Attributes Risk Growth FIGURE 7. CMMI Levels

TABLE 1. Mapping Policy to Guidance TABLE 2. Software Focus Areas TABLE A- 1. Entry and Exit Criteria Sample TABLE E- 1. Commercial TD and CS Data Rights Assertion Categories TABLE E- 2. Non-Commercial TD and CS Data Rights Assertion Categories TABLE E- 3. Contract Type Comparison

1 Introduction

Lack of software management guidance has been identified as a major concern for Air Force organizations involved in the acquisition and sustainment of software-intensive weapon systems.

Over the past ten to fifteen years, virtually all previously existing software-related Air Force policy, guidance, and training has been rescinded or eliminated. This has happened even as the software contribution to overall weapon system capability continues to increase, and almost all software intensive system development efforts are challenged to satisfy their cost, schedule, and performance baselines. Results of numerous independent reviews have shown that problems encountered on past programs are being repeated on current programs.

Section 804, ―Improvement of Software Acquisition Processes,‖ of Public Law 107-314, Bob

Stump National Defense Authorization Act for Fiscal Year 2003, placed a new emphasis on software acquisition process improvement, focused on acquisition planning, requirements development and management, project management and oversight, and risk management.

Furthermore, the Act requires metrics for performance measurement and process improvement, a process to ensure acquisition personnel have appropriate experience or training, and a process to ensure adherence to acquisition processes and requirements.

This guidebook addresses these known issues and sets top level Air Force expectations for the development, acquisition, management and sustainment of weapon systems software and software embedded in Air Force systems, so that software-related problems that are now too typical can be understood and avoided in the future. The principles and techniques in this guidebook will generally apply to all software domains, but the targeted domains include aeronautical, electronics, weapons, and space systems. The intended audience includes Project

Officers, systems/software engineers, and other engineers that have a software acquisition element in their project. Software engineering is an integral part of system acquisition and systems engineering (SE), and the guidance offered herein is intended to fit within and support current management and systems engineering approaches in Air Force systems and acquisition programs.

This document, when combined with related Air Force policy, existing technical references, and training, is intended to help acquisition and sustainment organizations more rapidly and more predictably deliver capability by learning from the past, establishing realistic and executable plans, applying systems engineering processes in a disciplined manner, and engineering systems right the first time. The purpose of this guidebook is to provide concise guidance for organizations that acquire or sustain systems that involve significant development, integration or modifications to their embedded software. It should not be used as policy or referenced in contracts. Rather, it provides an overview of the activities necessary to have a successful system/software acquisition.

The term Software Intensive Systems (SIS) is used frequently in this guidebook, as well as throughout numerous Air Force and Department of Defense (DoD) acquisition policy and guidance documents and Joint Chief of Staff publications. A consensus definition of this term is provided for this guidebook:

a. A Software Intensive System (SIS) is:

(1) A system in which the majority of functionality is achieved through software, or where mission success depends directly on the correct operation of software; or http://www.dod.mil/dodgc/olc/docs/PL107-314.pdf http://www.dod.mil/dodgc/olc/docs/PL107-314.pdf

USAF Weapon Systems Software Management Guidebook Chapter 1

(2) A system in which software is a significant factor in system development cost, system development risk, or system development time.

This guidebook is specifically directed at Computer Systems and Software (CS&S) associated with weapon systems. Such CS&S applications are also known as embedded systems, defined as computer systems and software that are a part of a larger system and satisfies some of the requirements of that larger system. Examples include CS&S in an aircraft, missile, satellite, command and control center, training systems, or weapons. The guidance herein is also applicable to the design, development, coding, integration, and testing of firmware. Finally, while this guidebook is organized to align with new start programs, most of the guidance herein can be easily adapted and applied to legacy programs that are implementing new capability blocks/increments.

While this guidebook is applicable to all Air Force embedded systems, it is recognized that there are differences in acquisition policy and guidance between the space and non-space domains. A goal for the next version of this guidebook is to better harmonize the approaches between these two domains where possible, and provide more specific domain guidance where necessary.

Please forward recommendations for changes or improvements to this document to:

Mike Nicol, ASC/EN, (Michael.Nicol@wpafb.af.mil), Ernie Gonzalez, SAF/AQRE, (Ernesto.Gonzalez@pentagon.af.mil), or

Lt Col Scott Conner, SAF/USAP (scott.conner@pentagon.af.mil).

mailto:Michael.Nicol@wpafb.af.mil mailto:Ernesto.Gonzalez@pentagon.af.mil mailto:scott.conner@pentagon.af.mil

2 Background

2.1 Why We Have “Software” Problems

Many reasons have been documented to explain why software so often becomes an issue for weapon system acquisition programs. Here are a few of the most frequent based on Air Force experience.

2.1.1 Planning Based on Unrealistic Expectations

Weapon system acquisition programs routinely aim to develop and deliver unprecedented warfighting capability. This unprecedented capability is often realized by developing complex, SIS or integrating existing systems and subsystems with other equally complex systems in new ways. Since acquisition programs are planned and estimated when only top-level performance requirements are available, it is extremely difficult to develop high confidence estimates and align expectations early in the program life cycle. Such early estimates are relatively subjective, involve numerous assumptions, and are almost always optimistic since the engineering activities that result in a complete understanding of the work to be accomplished have not been completed.

This complete understanding typically does not mature until well into the design phase, and when it does, it usually confirms that initial estimates were optimistic, key assumptions (such as significant reuse) cannot be achieved, more work than planned needs to be done, and the amount of software that has to be developed and/or integrated is growing. Reuse is a particular problem that suffers not only from excessive optimism about reuse efficiencies, but also from the often unplanned effort required to integrate reused code, to accommodate the adverse impacts of unused or dead code in reused packages, and to address vulnerability assessments, platform selection, platform integration, user interface, and related information assurance concerns.

Furthermore, any change to requirements (such as increased scope) must result in a change to the software estimate. Experience shows that even these early optimistic estimates frequently exceed the available program budget and schedule. This can lead to pressure to adjust the estimates to fit the funding profile or the desired schedule. In the face of such pressure, it is very difficult for estimators, technical staff, or program managers (PM) to defend their estimates.

2.1.2 Desired Software Performance is a Moving Target

Most acquisition professionals would agree that a complete, stable, and detailed set of requirements is a pre-requisite for high confidence estimates and successful software development. However, for weapon systems, early requirements are often incomplete or abstract enough so as to require significant interpretation. It is difficult for a potential developer to bid realistically to such requirements, since the software-related work that must be accomplished is so difficult to accurately quantify. The impact of this situation is magnified when the subject program is large, complex, or unprecedented, and the situation worsens when requirements changes are introduced after work on a particular software build or increment has progressed into design or a later phase. There are no free changes for software, and this is true even when removing capability, and any change to requirements must be accompanied by appropriate adjustments to budget and schedule.

Frederick P. Brooks, author of The Mythical Man-Month, states in the article ―Essence and

Accidents of Software Engineering,‖ (IEEE Computer, April 1987): ―The hardest single part of building a software system is deciding precisely what to build. No other part of the conceptual work is as difficult as establishing the detailed technical requirements, including all the interfaces to people, to machines, and to other software systems. No other part of the work so cripples the resulting system if done wrong. No other part is more difficult to rectify later.‖

USAF Weapon Systems Software Management Guidebook Chapter 2

2.1.3 Software Risks not Thoroughly Identified and Effectively Managed

It is tempting to assume that by applying good management techniques, problems that plagued past embedded system software development efforts can be avoided. All too often this proves untrue. Certainly there is a spectrum of outcomes between failure and success, and legacy programs that have developed, integrated, and delivered some capabilities have climbed the learning curve. Such organizations typically become more predictable over time. But it is a mistake to assume away common risks, especially for new programs working toward the first delivery of a product. As the Air Force Software Technology Support Center (STSC) has noted, ―Risk factors are always present that can negatively impact the development process, and if neglected, can tumble you unwittingly into program failure. To counteract these forces, you must actively assess, control, and reduce software risk on a routine basis.‖

Certain software risks exist on every embedded system software development effort. One example is the basic growth in the amount of software that must be developed as the job is better understood. Another example is erosion of planned software reuse, which again typically results in growth in the amount of software to be developed, or rework to integrate some other solution.

Yet another frequent example is failure to produce software at the rate anticipated (e.g., lines of code developed per day) and used in program estimates. Such risks need to be addressed in every program‘s risk management approach, but there are many more software-related risks that must also be identified and managed. More information on risk is provided in section 3.3 of this guidebook ―Management of Software Related Risks.‖

As Tim Lister (well known software consultant and principal of the Atlantic Systems Guild) states:

―Risk management is not risk avoidance when it comes to software projects. All the software projects with high value and low risks were done years ago. Almost by the simple definition of your mission, software efforts at your organization are going to be high risk. Raise expectations and you raise risk. It is just that simple.‖

2.1.4 Software Development Teams not Adequately Staffed, Stable, or Capable

The developers of large, new start embedded systems routinely have problems meeting initial staffing requirements due to the sheer numbers of personnel that must be identified, brought on board, and cleared into the program. Once past that challenge, there is the problem of staff turnover. It‘s not uncommon for weapon system software development staff to turn over 10% or more per year, and this contributes to a constant learning curve that can have a serious impact to productivity. Finally, there is significant individual variation in productivity. Steve McConnell, Founder, CEO, and Chief Software Engineer, of Construx Software, notes that researchers have found variations ranging from 10x to 28x in coding speed, debugging speed, defect-finding speed, percentage of defects found, bad-fix injection rate, design quality, and amount of code generated from a design. Considering these factors, it is easy to see how even those programs fortunate enough to start out with the ―A-team‖ can experience significant variation in their overall productivity during the development life-cycle.

2.1.5 Effective Software Processes not Established or Consistently Applied

The ability to successfully develop large, complex, unprecedented software systems in part depends upon having effective processes institutionalized and implemented with discipline. This includes acquisition processes for the Government and development processes for the contractors or organic organizations that actually develop the software. For new programs, the adequacy of development processes is normally evaluated to some extent in source selection, and the proposed program is examined to see that utilization of the processes fits within the program budget and schedule. Acquisition organizations (those Air Force organizations that acquire and/or contract for sustainment of systems) must resist the temptation to gain false savings by negotiating out the systems and software engineering processes. Attention to detail is required throughout this complex, multi-step process to ensure the developer team commits http://www.stsc.hill.af.mil/

Chapter 2 USAF Weapon Systems Software Management Guidebook contractually to using the critical processes, and that the overall program budget and schedule is adequate to allow the processes to be continuously and effectively applied. Abandoning established, disciplined processes when difficulties arise usually leads to more problems.

Process discipline should be viewed as an enabler to predictability, since it reduces variation by individuals or groups.

2.1.6 Ineffective Systems Engineering Interface to the Software Development Process

Although many program failures are labeled as software failures, the root problem often traces to a lack of systems engineering at the system level. By definition, embedded software in weapon systems does not function alone; rather software functions within a complex system of hardware that is performing a complex mission. Disconnects between the systems engineering process and software can manifest in countless forms. Beyond the details of input/output (I/O) interfaces with hardware, problems often occur when inadequate requirements deconstruction from the system or subsystem result in unknown requirements to software developers.

It is important for acquirers to be aware that many hardware issues will have a ripple effect on software development and may have the potential to be solved through modified software. The iterative application of the systems engineering process provides an integrated framework within which refinements made in one area may impact other areas and change their results. This evolution continues until a balanced system is achieved. This can only work when all elements of the system, including software, are part of the process. Like the function of embedded software in a weapon system, systems engineering activities serve little purpose if they are performed in isolation. Continuous communication between systems engineers, domain experts and software developers is a must to ensure the thread of systems engineering passes through software.

2.1.7 Lack of Effective Management and Insight into Software Development

It is difficult to manage software with the same rigor as other products, since the relationship of a

―completed‖ requirements document, a top level design, etc. to a completed working software component is not obvious. It is typical for software status to appear normal or on-schedule for long periods of time and then change overnight as an important milestone approaches. The late notice may be due to inadequate progress metrics, misinterpretation of what the metrics portrayed, or an unanticipated failure. The bottom line is that the search for truly predictive software metrics has been elusive, and achieving dependable insight into software status requires continuous attention, even when the development appears to be going well.

Unfortunately, many of the issues noted above are all too familiar in weapon system programs.

Dr. Barry Boehm, developer of the Constructive Cost Model (COCOMO) for software estimating, and author of a number of software development textbooks and papers, states that ―Poor management can increase software costs more rapidly than any other factor.‖ The good news is, as weapon system acquirers, Air Force program office personnel can have a significant influence on how software for weapon systems is managed. The guidance that follows is intended to directly address these critical issues.

2.2 Air Force Software Policy

In order to meet the requirements of Section 804 and improve the efficiency and effectiveness of

Air Force acquisition processes and software management, SAF/AQ/US Memorandum 04A-003, ―Revitalizing the Software Aspects of Systems Engineering,‖ (20 September 2004), originally identified ten software focus areas that programs must address.

This policy has subsequently been incorporated into AFI 63-101, ―Operations of Capabilities

Based Acquisition System,‖ (July 2005), and AFI 63-1201, ―Life Cycle Systems Engineering,‖

Attachment 8, (July 2007).

The ten focus areas and their associated tasks mandate that software engineering practitioners and managers must:

a. Estimate software development and integration at a high level (80-90%) of confidence.

b. Ensure program baselines support the disciplined application of mature systems/software engineering processes, are compatible with the overall program‘s

Expectation Management Agreement (EMA), and are supported by the program‘s budget.

c. Manage computer systems and software specific risks as an integral part of the program risk management process.

d. Identify the software-related strengths, weaknesses, experience, process capability, development capacity, and past performance for all developer team members with significant software development responsibilities.

e. Ensure the developer team establishes and applies effective software development processes.

f. Ensure the program office establishes and applies effective acquisition processes, is adequately staffed, and supports application of effective processes by the developer team.

g. Collect and analyze Earned Value Management (EVM) data at the software level.

h. Employ a core set of basic software metrics.

i. Plan and develop life cycle software support capabilities and support operations.

j. Support the transfer of lessons learned to future programs by providing feedback to center level Acquisition Center of Excellence (ACE) and other affected organizations.

AFI 63-1201 requires programs to address these software focus areas, as a minimum, throughout the life cycle, beginning with pre-Milestone A/Key Decision Point A activities. AFI 63-101 requires programs to incorporate the focus areas as appropriate in the program System Engineering Plan

(SEP), Integrated Program Summary (IPS), or acquisition plans. Program Executive Officer

(PEO) may tailor the implementation of these focus areas as required, with appropriate supporting rationale, but the Service Acquisition Executive (SAE) should be notified of all tailoring. Tailoring includes developing and implementing additional procedures and/or reducing already stated procedures, where appropriate.

The focus areas described in AFI 63-1201 cover the entire acquisition life cycle. Additionally, this instruction discusses how other processes must include software as a consideration. For example, Human Systems Integration (HSI) must be addressed throughout the life cycle, and http://www.dod.mil/dodgc/olc/docs/PL107-314.pdf http://www.e-publishing.af.mil/shared/media/epubs/AFI63-101.pdf http://www.e-publishing.af.mil/shared/media/epubs/AFI63-101.pdf http://www.e-publishing.af.mil/shared/media/epubs/AFI63-1201.pdf must be consistently integrated into systems engineering implementation to balance total system performance (hardware, software, and human).

AFI 63-101 addresses the acquisition planning, requirements development and management, project management and oversight, and risk management process improvement requirements cited in Section 804. PEOs are required to determine how to implement the remaining Section

804 requirements (metrics for performance measurement and continual process improvement, appropriate experience or training for acquisition personnel, and implementing and adhering to established processes and requirements) within their portfolios.

SEP guidance (OSD Systems Engineering Plan (SEP) Preparation Guide, October 18, 2007, Version 2.0 and AFI 63-1201) requires software planning to be addressed in the systems engineering processes. The SEP must remain consistent with the Life Cycle Management Plan

(LCMP) which documents the integrated acquisition and sustainment strategy for the life of the system, which, as described in AFI 63-101, includes software considerations. AFI 63-101 also discusses software as an element of specific functions in the acquisition process.

http://www.acq.osd.mil/sse/guidance.html

2.3 Mapping Policy to this Guidebook

TABLE 1 maps the requirements of the policy memo and AFI 63-1201 to section 3, Software

Acquisition Process Guidance, and the appendices, of this guidebook. Primary focus area guidance is designated with a bold “X”; additional guidebook guidance related to the focus area is designated with a bold “o”.

TABLE 1. Mapping Policy to Guidance.

Guidebook Sections

Software Focus Area Reference # (See list below in TABLE 2)

1 2 3 4 5 6 7 8 9 10

3.1 Software Aspects of Acquisition Program Planning o X o o o X o o X

3.2 Estimating Software Size, Effort, and Schedule X X X X o X

3.3 Management of Software Related Risks o o X o X

3.4 Source Selection Considerations o X o X X X o

3.5 Applying Earned Value Management to Software o o X X

3.6 Establishing and Managing Software Requirements o X o o

3.7 Acquisition Insight and Involvement o X o X

3.8 Safety Critical Systems o o X o

3.9 Non-Developmental Software o o o

3.10 Software Assurance and Anti-Tamper Protection o o

3.11 Configuration Management o o X

3.12 Life Cycle Support o o o X

3.13 Lessons Learned o X

Appendix A: Software in the Integrated Master Plan X

Appendix B: Software Content for the SOO/SOW o X o

Appendix C: Software Content for RFP Section L o o o X o

Appendix D: Software Content for RFP Section M o o o X o

Appendix E: Software Contracting Considerations o X o

Appendix F: CS&S Criteria for Technical Reviews o X o

Appendix G: Process Considerations for

Safety Critical Systems X

Appendix H: Air Force Core Software Metrics X

Appendix I: Software Development Plan X

Appendix J: Glossary of Supporting Information

NOTE: Key: X – Primary Guidance o – Supplemental Guidance

TABLE 2. Software Focus Areas.

Focus Area

Ref. # Title

1 High Confidence Estimates

2 Realistic Program Baselines

3 Risk Management

4 Capable Developer

5 Developer Processes

6 Program Office Processes

7 Earned Value Management Applied to Software

8 Metrics

9 Life Cycle Support

10 Lessons Learned

3 Software Process Guidelines for

Air Force Acquisition Organizations

3.1 Software Aspects of Acquisition Program Planning

Planning for software must address the system life cycle from inception through development, operations, and support, until system disposal. Whether the anticipated system development effort is a new program start involving a source selection or a delivery of additional incremental capability for a legacy program, proper planning ensures an executable program is defined and all required resources are available. Planning also provides a yardstick by which progress can be measured.

The acquisition program office must address all the software focus areas identified in AFI 63-

1201. Three categories of software planning will be addressed here:

a. Planning for Program Start (section 3.1.1)

b. Planning for Acquisition/Development Program Execution (section 3.1.2)

c. Planning for Post-Deployment Support (section 3.1.3)

In addition, critical elements of planning should be incorporated as appropriate in the program

SEP or other program plans. Section 3.1.4 provides guidance for addressing CS&S related topics in the SEP.

See Appendix A, ―Software in the Integrated Master Plan,‖ for guidance on recommended software content in the IMP.

3.1.1 Planning for Program Start

Planning for Program Start/Source Selection should address the following activities:

a. Developing a CS&S acquisition strategy consistent with the system acquisition strategy, including program objectives and constraints; available and projected assets, resources, and technologies such as non-developmental items (NDI); acquisition methods; potential contract types and terms; end user considerations; risk identification; life cycle support approach; technology insertion; and architecture & interoperability.

b. Defining and establishing the membership and responsibilities for the Air Force organization that will be responsible for software (includes interaction with system developers, testers, supporters, and users; as well as the means of management and control of the CS&S development effort).

c. Identifying and obtaining sufficient trained and experienced personnel to plan and oversee the computer system and software development.

d. Identifying, capturing, documenting, and maturing end user/warfighter needs and requirements specific to CS&S.

e. Identifying any policies, standards, or other guidance applicable to the program.

Chapter 3 USAF Weapon Systems Software Management Guidebook

f. Identifying all software to be developed, reused (used-as-is), modified, integrated, tested, or delivered (includes operational software; tools for software development, integration, test, and data reduction; firmware; databases; software for mission planning, training, automated test, and other support equipment/functions).

g. Examine the range of potential architectural approaches and assess the risks and opportunities associated with each to arrive at initial system/software architecture.

h. Developing an early and independent program office estimate of the expected software size, effort (staff hours), cost, and schedule (prior to release of Request for Proposal

(RFP) or receipt of offeror proposals).

i. Ensuring the planned CS&S development is consistent with the program budget and schedule allocated to software.

j. Developing CS&S inputs to the RFP.

Note that one method for organizing the program office and other Government stakeholders for the CS&S role in acquisition was defined by what used to be called the Computer Resources

Working Group (CRWG). The CRWG typically functioned as an advisory body to the program manager. These functions would now be handled by a software IPT or perhaps a working group under the IPT. Typical responsibilities should include:

a. Advising the program or system manager in all areas relating to the acquisition and support of CS&S.

b. Developing and maintaining a Computer Resources Life Cycle Management Plan

(CRLCMP).

c. Planning and implementing a CS&S post deployment support capability.

d. Monitoring CS&S development progress, and monitoring program compliance with any applicable CS&S policy, plans, procedures, and standards.

e. Integrating software test activities with the overall test program.

3.1.2 Planning for Acquisition/Development Program Execution

Planning for the System Development and Demonstration (SD&D) phase should include the following activities:

a. Identifying and obtaining sufficient trained and experienced personnel to oversee the

CS&S development.

b. Ensuring the offeror/developer team possesses adequate capability and capacity

(including processes, tools, and people).

c. Developing/evaluating the proposed application of evolutionary strategies for software development and capability delivery to match approved time-phased delivery of capability with resources and available technology (definition of block/increment capability content and related EMA).

d. Addressing the program approach to system and software requirements definition and management, including the use of collaborative communications and other methods to control software size growth due to derived requirements and design evolution.

e. Refining the identification of all software to be developed, reused (used-as-is), modified, integrated, tested, or delivered (includes operational software; commercial off-the-shelf

USAF Weapon Systems Software Management Guidebook Chapter 3

(COTS) and other NDI software, tools for software development, integration, test, and data reduction; firmware; databases; software for mission planning, training, automated test, and other support equipment/functions).

f. Accommodating the acquisition and software development issues related to the use of

NDI, including COTS software, reuse software, and Government furnished software

(GOTS).

g. Formulating and managing CS&S-related budgets and schedules, including initial estimates, revised estimates, and actuals for software size, effort (cost), and schedule.

h. Ensuring the planned CS&S development is consistent with the program budget and schedule allocated to software and the disciplined application of the proposed software development processes.

i. Identifying and managing the CS&S-related program risks.

j. Managing the CS&S development with appropriate insight and control through:

(1) Integration of software engineering with the program‘s systems engineering process, including technical reviews, and engineering data and documentation.

(2) Software metrics.

(3) Government and contract requirements for Software Resources Data Reports

(SRDRs) in accordance with DoD 5000.04-M-1, Cost and Software Data Reporting.

(4) Application of EVM to software (measuring progress relative to planned discrete, measurable software events and work products with defined exit criteria).

(5) Software content in the Integrated Master Plan (IMP), Integrated Master Schedule

(IMS), System Engineering Plan (SEP), Test and Evaluation Master Plan (TEMP), and Work Breakdown Structure (WBS), to the lowest level of work managed.

(6) Establishing, obtaining commitment to, and executing the program through the disciplined application of effective systems and software development processes, including establishing and monitoring commitment to developer software processes as documented in the Software Development Plan (SDP).

(7) Maintaining appropriate insight into developer and subcontractor/supplier activities to ensure the processes being employed are effective in achieving the desired result.

k. Addressing contracting considerations for software, including:

(1) Use of COTS software, open source software (OSS), Government furnished software (GFS), and other previously developed software.

(2) Contract types for software development.

(3) Software related contract clauses.

(4) Software related contract deliverables.

(5) Software related performance incentives appropriate to the program's challenges and life cycle phase (through Award Fee and Incentive Fee structures).

(6) Intellectual property (IP) rights (including licenses).

http://www.dtic.mil/whs/directives/corres/html/500004m1.htm

(7) Coordination with Defense Contract Management Agency (DCMA).

l. Ensuring an adequate approach for software security assurance, including:

(1) Identification of critical software technologies and protection techniques including strategies to comply with the Anti-Tamper (AT) and Software Protection Initiatives

(SPI).

(2) Development of an information assurance strategy and requirements for

Certification & Accreditation (C&A).

m. Planning and developing the post-deployment support capability for the CS&S, including software product engineering data (documentation) and intellectual property rights.

3.1.3 Planning for Post Deployment Support

The details of planning for post deployment support are addressed later in section 3.12.1.

However, it is worth noting here that a CRLCMP is a good method for organizing and documenting such planning. The CRLCMP is no longer required by policy, but such a plan can still be developed to establish and document the buy-in of all stakeholders, including the end customer (e.g., Air Combat Command, Air Mobility Command), operational testers, and system sustainers. Items appropriate to be addressed in this plan include:

a. Identification of responsibilities of the acquisition, developer, support, test, and user organizations in planning and implementing the software sustainment capability.

b. Planned source of life cycle (post-deployment) software support and identification of the life cycle support infrastructure.

c. Identification of computer systems hardware, software, documentation, and software engineering environment (as applicable) that will be delivered.

d. Human resources required for software support with supporting assumptions and rationale.

e. Intellectual property rights (including licenses).

f. Software test and integration considerations, including responsibilities for various levels of test and integration of software, location and fidelity of test and integration laboratories, and required test assets.

g. Transition of applicable operational software and support tools from the developer to the post deployment support organization.

h. Interim support (if applicable) subsequent to the completion of system development but prior to the availability of the permanent post-deployment support capability.

i. Security classification, certification, and accreditation considerations.

j. Required facilities including buildings, integration labs, and test ranges.

3.1.4 Software Planning in the Systems Engineering Plan

The program SEP provides an opportunity to integrate CS&S planning with the overall systems engineering planning. The following paragraphs provide a mapping of software-related concerns to the SEP guidance, based on the OSD SEP Preparation Guide. Note: all comments below are references to Appendix B of the OSD SEP Guide, which contains the Sample Format for

Milestone B, System Development and Demonstration.

a. Section 1.1, 2nd paragraph: For software, this includes:

(1) Identification of computing system and software-related standards.

(2) Addressing sustainment planning (capability and capacity needs) in order to balance overall software acquisition and sustainment costs.

(3) Describing the program plan for determining the location of software sustainment activities, required personnel, software development and integration environments.

b. Section 2.3, last bullet: For computing systems and software, address such things as redundancy management and fault tolerance. Also address processor and computing system reserve capacity for throughput, memory, I/O.

c. Section 2.5, last bullet: Although not a perfect fit, for software this could include understanding the program approach to lessons learned, including how historical software size, growth, reuse, productivity rates, and developer capability have been used to ensure the current program is realistic and executable. All software (support software as well as prime mission equipment) must be addressed. The problem that repeatedly gets programs in trouble is expecting more performance than can be fit into the available program budget and schedule, or in other words, failing to ensure the program is realistic and executable. Address here the systems engineering role in defining a high confidence, low-to-moderate risk program that will allow us to deliver the required capability within the cost and schedule constraints.

d. Section 3.3, 1st bullet: For software, this means addressing:

(1) How the program office will organize and staff to manage the software acquisition, including the required staffing, training, and experience levels.

(2) The program plan to establish required support for specialized technology areas, such as information assurance or safety-critical systems.

e. Section 3.4, 2 nd bullet: For software, this means describing the processes the program office will employ, as well as how the program office will support the disciplined application of established development processes by the contractor/developer team.

f. Section 3.4, 4th bullet: Not just for ―systems integration,‖ but for software, this means describing:

(1) The program approach to identifying the mix of software to be developed, modified, reused, or purchased.

(2) How it will be assured that proposed reuse components are mature and suitable for the intended use; there is a full disclosure/understanding of any projected modification needed; baseline and configuration management is in place; and associated regression testing, ownership rights and licensing, product assurance, and other risks have been addressed.

g. Section 3.5, 6th bullet: For software, this means understanding:

(1) The approach the contractor/developer team and program office will use to establish and manage software development processes and to commit to their consistent use through the IMP, IMS, Statement of Work (SOW), and other documents such as the SDP.

(2) How the program office will ensure development processes are consistently employed.

(3) How supplier capability will be evaluated during source selection to determine strengths, weaknesses, and risks related to computing systems and software.

(4) How supplier capability will be evaluated when establishing software development/integration baselines and contracts.

h. Section 4.1: For software, this means describing the program approach, including computing system and software impacts, for incremental capability development, verification, and delivery/fielding.

i. Section 5: For software, this means identifying the critical software-related technical review content and associated entry and exit criteria, or identifying where this information can be located in other program documentation.

j. Section 6.1, 3 rd bullet: Although not a perfect fit, for software this could include explaining the program approach to applying EVM at the software level to provide an objective measure of software cost and schedule performance.

k. Section 6.1, 4 th bullet: For software, this means formally managing expectations for software-enabled capabilities by describing how the program will establish, with high confidence, that the cost, schedule, and technical (performance) baselines are compatible and achievable.

l. Section 6.1, 5th bullet: For software, this includes the overall program plan for software metrics, including:

(1) How metrics are established, collected, and employed to manage the program.

(2) How the Air Force core software metrics (size, effort, schedule, quality, requirements definition and stability, staffing, progress, and computer resources utilization) are used.

(3) How the program office will use metrics and other methods to maintain appropriate insight into the contractor/developer team software development status.

m. Section 6.3, 1 st paragraph: For software, this means addressing software-related risks as part of the overall program approach to risk management. Specific attention should be given to addressing software estimating risks (underestimating size or growth, reuse erosion, less than expected productivity (more hours than expected per line of code), and undefined requirements.) as well as other computing system and software risks.

3.1.5 Sources of Additional Guidance

a. SEI Software Acquisition Planning Guidelines: Provides guidance on software acquisition planning and strategy on 13 different topics, including open systems approach, COTS-based systems, software architecture, software sustainment, requirements development, requirements management, operational information assurance, information assurance for COTS sustainment, information asset protection, software testing, software risk management, software metrics, and software-based award fees.

http://www.sei.cmu.edu/pub/documents/05.reports/pdf/05hb006.pdf

3.2 Estimating Software Size, Effort and Schedule

Software estimation is an inexact science. The data input to the process is usually highly uncertain, and where commercial estimating models are concerned, Dr. Barry Boehm in his book

Software Engineering Economics states that ―a software cost estimation model is doing well if it can estimate software development costs within 20% of the actual costs, 70% of the time, and on its home turf (that is, within the class of projects to which it is calibrated).‖

The challenge then is to comply with Air Force policy that requires estimating and funding acquisition programs with high (80-90%) confidence. That is to say, programs are to be estimated and funded so that the total program costs for any given program would be within the budget 80-90% of the time. Also, program milestones and program completion should meet the planned schedule 80-90% of the time. Air Force programs that involve intensive software development have a poor record of satisfying cost, schedule, and performance expectations.

Independent Government software estimates must be developed to facilitate realistic program planning, set reasonable expectations, support evaluations of offeror proposed costs and schedules during source selection or the start of new development blocks for legacy programs, and facilitate monitoring contract execution. These estimates must be developed to the highest confidence achievable given the state of the known requirements and any other data that characterizes the program at the time the estimate is performed.

In addition to estimates at program start, an Estimate to Complete (ETC) or Estimate at

Completion (EAC) is periodically needed in order to support an independent program evaluation or to otherwise determine program financial condition. Much of the guidance provided herein can also be applied to determining the content remaining in a program in order to formulate an ETC or

EAC or support of an Integrated Baseline Review (IBR). How often a program performs such estimates is dependent upon program needs.

The high degree of uncertainty at program start-up and in early development program phases makes software estimation with high confidence extremely difficult. The following factors contribute to this difficulty:

a. Software size estimates and supporting information are often provided by developers/offerors in a competitive environment, which promotes overly optimistic estimates.

b. Acquisition organizations may be inclined to accept optimistic estimates when even those optimistic estimates strain the planned program budget and schedule.

c. Development organizations assume the best people, or new/improved processes or tools, will lead to unrealistic productivity improvements.

d. New systems usually involve development of unprecedented systems/capabilities.

e. System performance is typically stated in high-level terms that are not directly translatable to software requirements and associated software size.

f. It is common to make erroneous and optimistic assumptions about initial software size, software size growth during development, software reuse, and non developmental software (NDS) for large, complex, and unprecedented systems.

g. There is an unrealistic expectation of accuracy in the results from the chosen estimating models and methods.

h. The program being estimated may not fit the chosen estimating models and methods.

i. Programs that are divided into many increments may not allocate sufficient effort for integrating new development with the baseline.

j. Requirements that are not well understood may not be allocated to the software until later in the program.

k. Long development phases for large, complex programs provide ample opportunity for outside forces to impact the program (funding instability, requirements changes).

This section provides additional guidance on developing high confidence estimates, dealing with software estimation risks, software estimation process considerations, and establishing realistic program baselines for software. The objective is to develop estimates that are based on the best available information, while fully disclosing and accounting for uncertainties. Throughout the estimation process, the Government must demonstrate unwavering commitment, and must make clear to all potential offerors its intent to develop and apply realistic estimates in establishing executable program baselines.

3.2.1 High Confidence Program Criteria

The 2006 Defense Acquisition Performance Assessment (DAPA) report notes that ―…current acquisition strategies encourage a ‗Conspiracy of Hope‘ that introduces instability at the very beginning of acquisition programs. The ‗Conspiracy of Hope‘ occurs when industry is encouraged to propose unrealistic cost, optimistic performance and understate technical risk estimates during the acquisition solicitation process and the Department is encouraged to accept these proposals as the foundation for program baselines.‖

The DAPA report defines ―high confidence‖ as a program with an 80 percent chance of completing development at or below estimated cost. AFI 63-1201 specifically states in

Attachment 8, paragraph 8.1.1: ―Estimate software development and integration at high (80-90%) confidence levels.‖

At the program level, high confidence can be addressed in a number of areas. A very top-level summary of results from an Aeronautical Systems Center (ASC) Balanced Scorecard Integrated

Product Team (IPT) on High Confidence Program Criteria is provided here:

a.…