AF_SW_Guidebook_V0.9_DEC_04.doc

U.S. Air Force

Software Management Guidebook

Version 0.9, 17 December 2004

Table of Contents

11.0 Introduction

22.0 Roadmap for Software Process Improvement

32.1 Tracing Section 804 Requirements

42.2 PEO / Acquisition Center Expectations

53.0 Software Process Guidelines for Air Force Acquisition Organizations

63.1 Software Aspects of Acquisition Program Planning

93.2 High Confidence Software Estimates

123.3 Software Development and Realistic Program Baselines

133.4 Management of Software Related Risks

143.5 Source Selection Support

173.6 Applying Earned Value Management to Software

183.7 Establish and Manage Software Requirements

213.8 Acquisition Insight and Involvement

233.9 Safety Critical and High-Assurance Systems

243.10 Application and Sustainment of Non-Developmental Software

253.11 Software Security Assurance

273.12 Configuration Management

283.13 Life Cycle Support

323.14 Lessons Learned

33Appendices

34Appendix A:

Software in the Integrated Master Plan

37Appendix B:

Software-Related Content for the Statement of Objectives (SOO) / Statement of Work (SOW)

39Appendix C:

Suggested Software Content for Request for Proposal (RFP) Section L, “Instructions, Conditions, and Notices to Offerors”

49Appendix D:

Suggested Software Content for Request for Proposal (RFP) Section M, “Evaluation Factors for Award”

53Appendix E:

Contracting Considerations for Software

57Appendix F:

Computer Systems and Software Criteria for Technical Reviews

73Appendix G:

Process Considerations for Safety Critical / High Assurance Systems

1.0 Introduction

A Weapon System Software Sustainment Study completed by AFMC in 2001 identified lack of guidance as a major concern for Air Force organizations involved in the acquisition and sustainment of software-intensive weapon systems. As the software contribution to overall weapon system functionality continues to rise, almost all software intensive system development efforts are challenged to satisfy established cost, schedule, and performance baselines. We are expected to be responsive to Air Force Agile Acquisition objectives of decreasing acquisition cycle times and improving credibility. Managers are expected to accept and manage risk as they attempt to satisfy these objectives, and we are all expected to apply innovation, continuous improvement, and lessons learned in the acquisition process. The challenges are significant.

Additionally, Section 804 of the FY03 Defense Authorization Act (now Public Law 07-314) places a new emphasis on software acquisition process improvement. The focus of this improvement activity is on acquisition planning, requirements development and management, project management and oversight, and risk management. Furthermore, the Act requires metrics for performance measurement and process improvement, a process to ensure acquisition personnel have appropriate experience or training, and a process to ensure adherence to acquisition processes and requirements.

The development of software for weapon systems cannot be viewed as a stovepipe activity. Weapon system software acquisition engineering is an integral part of the system acquisition and systems engineering processes. The Air Force approach builds on this relationship, and while this guidebook is devoted to software, efforts will continue to more closely link software with revitalized systems engineering processes.

Dr. Marvin Sambur, Assistant Secretary of the Air Force (Acquisition) and Dr. Peter Teets, Undersecretary of the Air Force jointly released a policy memorandum to emphasize the importance of high confidence estimates, realistic program baselines, risk management, developer capability, developer processes, program office processes, earned value management, metrics, life cycle support, and lessons learned. Proper attention to the software aspects of these focus areas is critical to achieving well managed and technically sound Air Force programs. Our expectation is that we can improve and more rapidly acquire systems by learning from the past, establishing a realistic and executable plan, applying systems engineering processes in a disciplined manner, and engineering systems right the first time.

The processes described herein are intended for use within all Air Force system program offices (SPOs) within the aeronautical, electronics, weapons, and space domains.

Please forward recommendations for changes or improvements to this document to Mike Nicol, ASC/EN, (Michael.Nicol@wpafb.af.mil), Ernie Gonzalez, SAF/AQRE, (Ernesto.Gonzalez@pentagon.af.mil), or Maj Mark Davis, SAF/USAL, (Mark.Davis2@pentagon.af.mil).

2.0 Roadmap for Software Process Improvement

In response to Public Law 107-314 Section 804, Improvement of Software Acquisition Processes, OSD established a Software Acquisition Process Improvement Program Integrating Integrated Product Team (SAPIP IIPT). This IIPT assumed the responsibility for monitoring software process improvement activities across the services and several DoD agencies.

OSD Focus

The focus of the OSD IIPT has evolved to a focus of:

· Documenting progress the applicable DoD components (services and agencies) have made related to software acquisition process improvement

· Addressing compliance with the law, but in the context of the broader systems engineering revitalization effort across DoD

· Taking credit for activities already in place or in development, including

New OSD policy and guidance on systems engineering revitalization

Systems Engineering Plans (SEPs) as an implementation tool

Best practices clearinghouse

Previous/ongoing efforts by components and sub-components

It is expected that the OSD SAPIP IIPT will complete its work in early calendar year 2005, and the OSD oversight of software process improvement will be absorbed into the OSD Systems Engineering Forum. This forum is intended to ensure the proper senior-level focus on Systems Engineering concerns in the areas of policy, best practices, and education and training among the acquisition community, industry, and academia.

Air Force Approach

In January 2004, the Air Force established the Air Force Software Intensive Systems Strategic Improvement Program (AFSSIP). The purpose of this program is to evaluate the efficiency and effectiveness of Air Force software management and processes that are part of the systems engineering and capability acquisition, and to identify changes that may be needed to improve them. AFSSIP is intended to address existing concerns over software process improvement training and education, metrics, technology transition, and integration with systems engineering.

An AFSSIP working group was established, consisting of a core team responsible for accomplishing analyses and developing AFSSIP products, assisted by a large working group intended to integrate the concerns of acquisition practitioners and developers.

The first product of the AFSSIP working group was joint SAF/AQ and SAF/US Policy Memo 04A-003, “Revitalizing the Software Aspects of Systems Engineering”, dated September 20, 2004. The memo identifies ten software focus areas to be addressed by acquisition programs, and is available at http://www.safaq.hq.af.mil/acq_pol/documents/SofwareAspectsofSysEng.pdf.

This guidebook is the second product of the AFSSIP working group, and provides the next level of definition in setting Air Force acquisition expectations for systems that have significant software content or software-driven functionality. This guidebook will be followed by training intended to reach all acquisition personnel involved in planning and/or executing acquisition programs where those expectations must be satisfied.

Figure 2-1 shows the roadmap for software process improvement within the Air Force.

Figure 2-1: Roadmap for Air Force Software Process Improvement

2.1 Tracing Section 804 Requirements

AFSSIP products are responsive to Section 804 requirements, although the memo and guidebook content do not map one-to-one, nor in sequential order, to the specific requirements of Public Law 107-314. The following mapping may be helpful in tracing Section 804 requirements to the applicable sections of the AFSSIP policy memo and Air Force Software Guidebook:

Process Area – Acquisition Planning

Policy memo focus areas: 1, 2, 4, 5, 7, 8, 10

Guidebook section: 3.1

Process Area – Requirements Development and Management

Policy memo focus areas: 5, 6, 8

Guidebook section: 3.7 and all appendices

Process Area – Project Management and Oversight

Policy memo focus areas: All

Guidebook sections: All

Process Area – Risk Management

Policy memo focus area: 3

Guidebook section: 3.4

Appropriate metrics for performance measurement and continual process improvement:

Policy memo focus area: 8

Guidebook sections: 2.2, 3.8

Trained and Experienced Personnel:

Policy memo focus area: 6

Guidebook sections: 3.1, 3.8, Appendix F

Implementing and adhering to established processes and requirements:

Policy memo focus area: 6

Guidebook section: 2.2

2.2 PEO / Acquisition Center Expectations

With the publication of this guidebook, PEOs are expected to develop tailored approaches for use within the acquisition programs at their centers. The Section 804 requirements for improvement in the four listed process areas as well as training and experience are straightforward and are addressed adequately in the AFSSIP memo and this guidebook. The requirements for “metrics for performance measurement and continual process improvement” and “implementing and adhering to established processes and requirements” are more subjective, and are not addressed in detail in this version of the guidebook. The means of implementing these Section 804 requirements is left to PEO discretion.

For measurement and continuous process improvement, it should be noted that the policy memo and guidebook both address the typical metrics used to provide insight into development program status. Neither addresses the Section 804 focus on continuous improvement and measurement of the effectiveness and efficiency of processes employed by the acquisition organization. Acquisition organization process improvement activities can range from simply identifying, documenting, executing and measuring the effect of key processes (such as those documented in this guidebook), to applying a capability maturity model such as the Capability Maturity Model Integration Acquisition Module (CMMI-AM). More information is available on the CMMI-AM at http://www.sei.cmu.edu/publications/documents/04.reports/04tr001.html. Note that the CMMI-AM covers the complete range of acquisition processes, and is not limited to those that focus on software.

The means of ensuring established processes and requirements are implemented and adhered to is also left to the PEO. The intent of AFSSIP in this area is to encourage process improvement internal to the PEO portfolios and individual programs. The overall objective of acquisition process improvement is to ensure processes are documented, applied in a disciplined manner, and improved where necessary. A range of possible solutions is available, but as a minimum program offices should adopt or establish acquisition processes at the start of a program, and should review those processes periodically during development, implementing changes where warranted. The Air Force does not intend to collect or require reporting of this information in any centralized manner.

3.0 Software Process Guidelines for Air Force Acquisition Organizations

As described in the previous sections, the purpose of this document is to provide top-level guidance for organizations that are acquiring systems that require significant software development and/or integration, or that provide significant functionality through software. This section contains specific process requirements and guidance that focus on areas that have historically contributed to software-related problems.

Air Force product centers and program offices are expected to tailor the application of these processes to their specific needs, consistent with the responsibilities and capabilities of the acquirer/developer team. Tailoring includes developing and implementing additional guidance where appropriate.

Note that several of the focus areas addressed below are early yet critical steps in program planning. Once an acquisition program baseline has been established, it is extremely difficult, if not impossible, to recover when that baseline is not adequate to support the development of the volume of software needed, the typical growth in software size as the requirements and design evolve, and the disciplined application of effective development processes.

3.1 Software Aspects of Acquisition Program Planning

Since software plays a significant role in the majority of today’s acquisition programs and too often fails to satisfy performance, cost, and schedule baselines, it is prudent to carefully plan the software aspects of each acquisition program. The top-level objectives of software planning are the same as those for planning the program in general:

· Ensure an executable acquisition program is defined and required resources are available

· Establish a yardstick by which to measure progress

Program planning should accommodate the software focus areas identified in Memorandum 04A-003, Revitalizing the Software Aspects of Systems Engineering, Sep 20 2004 (Memo available at http://www.safaq.hq.af.mil/acq_pol/documents/SofwareAspectsofSysEng.pdf). The software focus areas are: High-Confidence Estimates, Realistic Program Baselines, Risk Management, Capable Developer, Developer Processes, Program Office Processes, Earned Value Management, Metrics, Life Cycle Support, and Lessons Learned. Specific considerations for addressing these focus areas are provided throughout this document.

In order to plan the software aspects of the program, all software to be developed, reused (used-as-is), modified, integrated or otherwise used in the system, must be identified. This includes operational software, and tools for software development, integration, test, and data reduction. It also includes firmware, databases, and software for mission planning, training, automated test and other support equipment/functions.

In general, the software element of acquisition planning ensures each component of the software identified in the previous step is incorporated in acquisition program planning and control tools such as the Work Breakdown Structure (WBS), Integrated Master Plan (IMP), and Integrated Master Schedule (IMS). In order to satisfy the objective of defining an executable acquisition program, this planning should consider all required software-related acquisition, development, and sustainment activities. It should also address performance and other related requirements; required resources in terms of trained and experienced personnel (acquisition and development), processes, and tools; associated effort (cost) and schedule; and software-related risks.

See Appendix A, “Software in the Integrated Master Plan”, for guidance for recommended software content in the IMP.

Planning for Project Start/Source Selection

Software planning during the project planning phase, including project start-up and/or source selection, should include the following:

Development of computer system and software inputs to the Request for Proposal (RFP)

Application of evolutionary acquisition strategies during software development

An estimate of expected software size, prior to receipt and independent of offeror proposals

During source selection, an evaluation of the offerors' software development capability and capacity

During source selection, a determination that the proposed software development effort and schedule is compatible with the disciplined application of the proposed software development processes

Planning for System Development and Demonstration

Software planning for the System Development and Demonstration phase should be documented and maintained over the life of the program, and should include:

Ensuring consistent treatment of project planning through the WBS, estimates of the work products, all defined life-cycle phases, and associated cost and schedule estimates

Formulating budgets and schedules; continuously identifying, analyzing and mitigating risks; and addressing data management, resource planning, and stakeholder involvement

Establishing commitment by documenting responsibilities, reconciling resources and planning, and obtaining stakeholder commitment

Identifying and obtaining human resources with the necessary skills, training, and experience to manage and execute the software acquisition

Addressing contracting considerations for software, including:

government furnished software commercial off the shelf (COTS) software contract types for software development software-related contract clauses software contract line items (CLINs) software-related performance incentives appropriate to the program's challenges and life cycle phase (addressed through Award Fee and Incentive Fee structures) coordination with Defense Contract Management Agency (DCMA)

A plan for managing and maintaining software project data, including estimates, changes, and actuals to software size, effort (cost), and schedule; software productivity; defects; etc. over the life of the project, to support both program execution and transfer of lessons learned

The program approach to address system and software requirements definition and management, including the use of collaborative communications and methods to control software size growth due to derived requirements and design evolution

The program approach to maintaining effective technical insight into and control over the software development effort, to include:

Establishing, obtaining commitment to, and executing the program through the disciplined application of effective systems and software development processes, including establishing and monitoring commitment to developer software processes as documented in the Software Development Plan (SDP)

Maintaining appropriate insight to ensure the processes being employed are effective in achieving the desired result

Measuring progress relative to a planned profile of discrete, measurable events and work products

Recognizing that there are cost and schedule consequences to changing requirements, development processes, or the size of the software to be developed and integrated The program approach to fully develop and integrate software within the program’s systems engineering process, including technical reviews, engineering data and documentation, etc.

The program approach for software security assurance (see section 3.11), including:

Identification of critical software technologies and protection techniques including strategies to comply with the Anti-Tamper and Software Protection Initiatives

Development of an information assurance strategy and requirements for Certification & Accreditation

The critical elements of planning for the software aspects of the acquisition program, including the software focus areas identified in memo 04A-003, should be incorporated as appropriate in the program Systems Engineering Plan (SEP), Integrated Program Summary, or other acquisition plans.

Planning for Operations and Support

Planning for the Operations and Support phase may be accomplished in a separate software or computer resources life cycle management plan. This plan can be developed to establish and document the buy-in of all stakeholders, including the end customer (e.g., Air Combat Command, Air Mobility Command), operational testers, and system sustainers. Items appropriate to be addressed in this plan include:

Identification of responsibilities of the acquisition, developer, support, test, and user organizations in planning and implementing the software sustainment capability

Planned source of life cycle (post-deployment) software support and identification of the life cycle support infrastructure

Identification of computer systems hardware, software, documentation, and software engineering environment (as applicable) that will be delivered

Human resources required for software support with supporting assumptions and rationale

Intellectual property rights

Software test and integration considerations, including responsibilities for various levels of test and integration of software, location and fidelity of test and integration laboratories, required test assets, etc.

Transition (if applicable) of all operational software and support tools from the developer to the post deployment support organization

Interim support (if applicable) subsequent to the completion of system development but prior to the availability of the permanent post-deployment support capability

Security classification, certification, and accreditation considerations

Required facilities including buildings, integration labs, and test ranges

Planning is essential in order to satisfy our agile acquisition objectives. Planning is the basis for successful execution, which is realized through an adequate level of insight, open communication, trust, and credibility.

“The common failing of programming groups is that there is too little management control, not too much.”

Frederick P. Brooks

“The Mythical Man-Month”

3.2 High Confidence Software Estimates

Air Force policy is to estimate and fund programs to a high (80-90%) confidence. That is to say, programs are to be estimated and funded so that the total program costs for any given program would be less than the budget 80-90% of the time. Also, program milestones and program completion should meet the planned schedule 80-90% of the time.

Accurate, dependable, and convincing estimates must be developed to facilitate realistic program planning and to set reasonable expectations. This is best accomplished as a joint effort between the government and industry at the earliest possible time, starting prior to RFP release through discussions at pre-solicitation meetings (e.g., industry days). The government must make clear to all potential offerors its intent to develop and apply realistic estimates. During source selection, every effort should be made (through discussions if necessary) to obtain the information required to support a comprehensive software development estimate.

The OSD Cost Analysis Improvement Group (CAIG) requires certain data collection and reporting in the form of a Software Resources Data Report (SRDR) for ACAT IA, ACAT IC or ACAT ID programs containing software effort with a projected value greater than $25M (FY 2002 dollars). The data collection and reporting applies to developments and upgrades whether performed under a commercial contract or internally by a government Central Design Activity (CDA) under the terms of a memorandum of understanding (MOU). The program office data collection effort should support both the CAIG SRDR and the program office comprehensive software development estimate.

In addition to estimates at program start, as a program matures, occasionally an estimate to complete (ETC) is needed in order to support an independent program evaluation. Much of the process described herein can also be applied to determining the content remaining in a program in order to formulate an ETC.

For this process, the software development period is defined as starting with a baseline (complete) set of software requirements in a formal specification format and ending with a fully integrated and tested subsystem / functional software product ready for software / hardware integration and test. This convention is used because most of the estimating models provide estimates based on empirical data for only these phases. Additional effort is required to develop, allocate, and analyze the subsystem and software requirements; perform software to hardware (subsystem) integration and test; and perform system integration and test.

Considerations For Establishing 80-90% High Confidence Software Estimates

The 20 Sept 04 SAF/AQ/US Memorandum addressing revitalization of the software aspects of systems engineering includes a focus area related to high confidence estimates. In the context of this policy, “high confidence” means estimates should be more conservative so that programs account for and plan to accommodate the various risks that routinely affect software development. These risks include but are not limited to:

· ability to properly size the software development/integration effort

· ability to account for growth due to developer derived requirements and design evolution

· ability/discipline to change effort/schedule estimates in response to software size growth

Estimating software development at high confidence is a multi-faceted process which includes assessment of the characteristics of the program, requirements maturity and stability, program schedule constraints, level of known historical performance data, and many other attributes of the development program. The development of a high confidence estimate considers the following factors:

The estimate is based on well defined, stable requirements. The estimate should reflect a level of requirements coordination and stability including spiral development strategies, acknowledging less firmness in requirements for follow-on spirals

The estimate incorporates the confidence in the program’s ability to accurately estimate software Source Lines of Code (SLOCS) given the known state of requirements during the estimate timeframe

The estimate does not apply optimistic (i.e., not yet realized) productivity rates resulting from software estimating model environment parameter settings that may not be sustained throughout the development period, such as developer capabilities (A-team), new productivity tools, or new, more effective processes

The estimate includes appropriate factors for historical program experience on code growth and ability to achieve planned levels of reuse

The estimate includes costs associated with modifying and integrating any planned COTS software

The estimate considers the ability to accurately characterize the developer’s capability and program environment which significantly affects cost and schedule estimates.

Actual cost/productivity/SLOC data is available on the same program or very analogous program at the same contractor facility

The estimating techniques used are appropriate to the program situation and comprehensiveness of available data (e.g., point estimates vs. Monte Carlo methods)

The cost estimate is developed within the framework of a comprehensive, detailed, realistic and well documented software development schedule. The program and the cost estimate must apply expectation management to balance cost (effort) with a realistic development schedule and set of program performance requirements)

The content of the software estimate (phases/activities included in estimate) is consistent with other program component estimate approaches and content. The estimate includes software support to system engineering, system and sub-system requirements definition, system integration and system test as appropriate)

Elements of the Software Estimation Process

The software development estimating process consists of a series of activities that are grouped into the following process steps:

a.

Identify and structure all software to be developed b.

Determine software size, itemized by source language and complexity and adjusted for requirements maturity, historical growth experience, growth potential and reuse degradation c.

Assess Commercial Off-The-Shelf (COTS) software and other reuse candidates for risk d.

Determine that software development has been appropriately and consistently addressed throughout the proposal and chart the software development plan e.

Estimate development effort and schedule for each software component using consistent input parameters and documented assumptions f.

Crosscheck estimate results with other methods g.

Estimate the front-end systems engineering effort & schedule and the back-end systems integration and test h.

Assess software schedule risk in the context of a well constructed Integrated Master Schedule (IMS) i.

Validate the estimate against the contractor’s productivity on similar efforts and review the estimate at the aggregate level with senior staff j.

Accomplish Monte Carlo based risk assessment at 90% cumulative probability value* considering size growth and other specific risks along with reasonable ranges of model parameters, and ensure the budget is allocated through the FYDP to support the estimate

* This however, does not equate to 90% confidence that the cost will be equal to or less than the predicted value. Typical limitations on the number and fidelity of data points supporting the analysis impacts the ability to state with certainty a 90% confidence.

“Another two-cultures problem is that customers and users have little feel for the difficulty of developing a given quantity of software within budget and on schedule. This leads to unrealistic expectations by customers that can be reinforced by developers’ desire-to-please or desire-to-sell tendencies, leading to frequent budget and schedule overruns.”

Barry Boehm

“The Art of Expectations Management” IEEE Computer, January 2000 “Accept only one independent variable. One clear message from software cost and schedule models is that larger amounts of developed software require larger amounts of budget and schedule. … It’s better to tell your customers that they can expect either a “fixed amount of software” or a “fixed cost or schedule”, but not both.”

Barry Boehm

“The Art of Expectations Management” IEEE Computer, January 2000 3.3 Software Development and Realistic Program Baselines

Nothing has done more to undermine our acquisition credibility and complicate effective management of the acquisition of software intensive systems than our inability to establish realistic software development cost and schedule baselines. Without a realistic estimate of the required effort and schedule, tied to specific requirements during program formulation, there is a high probability that the initial overall program estimate will not accommodate the software development. Likewise, if an estimate is not available when proposals are evaluated in source selection, there is no basis from which to judge the feasibility of the offerors’ proposed effort and schedule. Programs established with unexecutable cost and schedule baselines will inevitably be forced to cut corners along the way; and under these conditions successful outcomes are virtually impossible.

The challenge in building realistic cost and schedule baselines early comes from the fact that up-front estimates are based on numerous assumptions and uncertainties. Several iterations of cost and schedule estimation may be necessary to converge on a more accurate estimate as the software product matures. The engineering role is to provide the most accurate technical information (both assumptions and facts) available in order to create a realistic program software development baseline.

Steps to arrive at a realistic cost and schedule baseline include the following:

Develop and refine estimates of software size for all software to be developed and integrated. Include expected reuse and integration of legacy software, subcontractor efforts, COTS, and GFE. Base these estimates on similar past programs when possible. See section 3.2.

Use estimated size to establish the associated software development effort and schedule (this may include defining appropriate software estimating model input parameters). Collaborate with other government organizations in creating and maintaining program estimates

During source selection, ensure the proposed effort and schedule are compatible with disciplined application of the proposed software development processes. Also be cognizant of the effort (hours and dollars) and schedule (time) impacts of the proposed processes.

Ensure the proposed software development effort and schedule accommodates the required system performance (relates to software size), fits with the program cost and schedule baselines, and has the capacity to accommodate problems the program encounters

Identify and manage software-related risks (see Section 3.4).

3.4 Management of Software Related Risks

The program risk management approach should include Computer Systems & Software (CS&S) issues. It is recommended that programs use AFMC Pamphlet 63-101 (Risk Management), and DoD guidance found at http://www.dau.mil/pubs/gdbks/risk_management.asp in order to identify, track, manage, and resolve CS&S risks.

Risk management should include the following basic steps:

Risk planning

Risk assessment, i.e., identify, analyze and document risks to determine their relative importance by assessing impact and probability of occurrence. Define risk parameters, and establish and maintain a risk management strategy

Risk handling, i.e., identify methods to reduce adverse impacts on achieving objectives or the probabilities of risks occurring

Risk monitoring, i.e., review risks periodically for change in status and/or assessment of any new risks and implement risk mitigation plan(s)

Risk budgeting, i.e., fund to manage and mitigate risks

Specific CS&S risks that programs should consider and manage include:

Inability to accurately estimate software development size

Incompatible software development performance, effort, and schedule baselines

Inability to achieve planned levels of reuse

COTS/GOTS suitability, integration, and sustainment

Integration-heavy effort (significant integration effort for existing components)

- Concurrent hardware/software development where requirements allocation methodology is unclear

New and unprecedented requirements that drive the use of unproven technology, such as:

Domains and applications where the requirements are poorly defined or unstable

Extensive CS&S security requirements, such as multi-level security

Computer system / software architectures which are complex, highly integrated, or unproven; and which rely on shared use of computing elements in real-time critical applications

- Long duration system development timelines and technical obsolescence of underlying computing architectures and hardware

- Safety critical requirements

- Acquisition strategies and approaches which rely on:

Government furnished equipment (GFE) which has an unknown performance capability

Tools and technologies that are still under development

The use of tools, methods, and technologies with which the developer has no experience and capability, and therefore must go through a learning curve.

Multiple developers and subcontractors teaming to develop complex software intensive systems which must be tailored and integrated into a total system capability

Uncontrolled sources of software (foreign developers, open source, etc.)

3.5 Source Selection Support

Computer Systems and Software (CS&S) support for source selection includes instrumenting the Request for Proposal (RFP) with appropriate CS&S content and performing the actual Source Selection evaluation against the RFP requirements. RFP sections of interest include Instructions, Conditions, and Notices to Offerors (Section L) and Evaluation Factors for Award (Section M). Each offeror’s proposal is evaluated against Section M to identify and document software-related strengths, weaknesses, and risks. The evaluation process must ensure consistency across all proposal sections relevant to software development.

Accurate, dependable, and high confidence software effort and schedule estimates must be developed to evaluate the offeror’s proposed software development planning. During source selection, every effort should be made (through discussions if necessary) to obtain the information required to support a comprehensive software development (most probable) estimate for each offeror. Program office personnel faced with this need should contact their center ACE offices or their functional Engineering, Finance, and Contracting organizations for assistance.

3.5.1 Software-Related Content in the Request for Proposal

Software is addressed in the RFP in order to solicit proposals that provide the information to support an effective government evaluation and identification of strengths, weaknesses, and risks related to software. The following software-related content is suggested for the various sections of the RFP.

The Statement of Objectives (SOO) should include requirements to:

Develop, maintain, and comply with the offeror developed SDP

Support program office integrated product teams and working groups

The System Requirements Document, draft specification, or equivalent, should incorporate unique software requirements which are evidenced at the system level. This should include any user sourced requirements as well as typical computing system reserve capacity, growth, and architecture requirements.

RFP Section L should require:

Submittal of a draft Software Development Plan (SDP) which defines the offeror’s proposed software development processes for this program, including key processes which formed a basis for any assessed CMMI capability profile or maturity level

Software size information, software size definitions, and re-use estimates with rationale to support software estimation

Technical definition of the proposed computer hardware and software architecture (processors, buses, languages, object oriented design, open systems, etc), sizing, throughput, growth capacity, and technology update/refresh strategy

Data and information needed to conduct system/software capability appraisals during source selection

Section M should define criteria for evaluating the offeror proposals including:

Proposed software processes consistent with the proposal draft SDP, Integrated Master Plan (IMP), and Integrated Master Schedule (IMS)

- Software development process commitment (SOW or equivalent)

- A proposed CS&S technical solution that satisfies the system and software performance requirements (open systems architecture, spare and growth, for example)

- Proposed software development effort and schedule consistent with the proposed technical content as well as SPO estimates of the overall program effort and development schedule baseline

- System/software development capability

See the following Appendices for associated guidance

· Appendix B: Software-Related Content for the Statement of Objectives (SOO) / Statement of Work (SOW)

· Appendix C: Suggested Software Content for Request for Proposal (RFP) Section L, Instructions, Conditions, and Notices to Offerors”

· Appendix D: Suggested Software Content for Request for Proposal (RFP) Section M, Evaluation Factors for Award”

· Appendix E: Contracting Considerations for Software

3.5.2 Activities During Source Selection

During source selection, programs should:

Evaluate the proposed technical approach for computer systems and software

Determine that the software development effort has been consistently addressed throughout the proposal

Determine the software development size and expected growth during the proposed development period, and factor this into the evaluation of the offeror’s proposal

Develop most probable software effort (cost) and schedule estimates, and factor into the offeror’s risk ratings (estimates should be accomplished at the 80-90% confidence level)

- Assess proposed software development processes, including subcontractor oversight and management, and evaluate the compatibility of the proposed processes with the proposed program plan’s effort, schedule, and performance requirements

Account for planned concurrent development efforts by ensuring that adequate personnel, development stations, integration labs, etc. are available to support the plan

Evaluate software development capability for each offeror, and identify associated strengths, weaknesses, and risks

3.5.3 Due Diligence in Selecting a Capable Developer

Offerors have varying degrees of inherent development capability (strengths, weaknesses, and risks) by technical area, including software. Without specific effort to address known weaknesses and risks, the same weaknesses and risks will likely manifest themselves in new programs. Program offices need to identify each offeror’s specific capabilities and past performance and factor that into the source selection decision. During program execution, these identified strengths, weaknesses, and risks should be considered when defining the approach used to manage and oversee the offeror’s development efforts.

Suggested program office actions and considerations include:

- Incorporating software development past performance into the source selection past performance evaluation process

- Incorporating offeror system/software development capabilities evaluation into the source selection process

· If process maturity as defined by the Software Engineering Institute’s Capability Maturity Model Integration (CMMI) is used as a factor in the evaluation, take the following actions:

· Solicit in the RFP existing capability information for all members of the development team

· Include in Section L the proposal content instructions supporting submittal and evaluation of CMMI capability information

· Include a requirement to submit the Appraisal Disclosure Statement to identify and clarify relevant appraisal conditions and factors described below

· Include in Section M the evaluation criteria for evaluating CMMI capability information

· Accomplish the following in evaluating submitted CMMI capability information:

· Assess the scope of the submitted CMMI appraisal for:

· Organization coverage including location of proposed program execution within the organization scope of the submitted appraisal results

· Process Area coverage (inclusion) in the submitted appraisal results. Consider Process Areas excluded from the appraisal and the relevance to the program in source selection

· Relevance of the projects included in the submitted appraisal to the program in source selection

· Functional scope (e.g., System Engineering, Software, IPPD etc) of the submittal appraisal and the relevance to the program in source selection

· Consider the fidelity of the submitted appraisal (Standard CMMI Appraisal Method for Process Improvement (SCAMPI) Class A, B or C))

· Verify the SEI authorization status of the Appraisal Team Leader (refer to the SEI CMMI web site at http://www.sei.cmu.edu/managing/app.directory.html)

· Understand the level of independence of the Appraisal Team Leader from the proposed development team organization

· Consider the timeframe of the submitted appraisal information and relevance to current development capability

· Review the proposed SDP for inclusion of processes consistent with the submitted appraisal

· Review the proposed SOW for requirements to apply the proposed SDP processes consistent with the submitted appraisal

· Incorporate submitted CMMI capability information in the Source Selection evaluation results consistent with the Section M criteria and in the form of strengths, weaknesses, and risks

Note: Material in Section 3.5.3 is adapted from “Choosing a Supplier: Due Diligence and CMMI”, News@SEI 2004 Number 3, CMMI in Focus, David M. Phillips.

3.6 Applying Earned Value Management to Software

Earned value is an essential indicator of program health, and serves as a powerful tool to gain insight into program status. Software development earned value status can be monitored in the same manner as other program development activities.

In order to facilitate the implementation of Earned Value Management for software, engineering should:

- Collaborate with Program Management and Financial Management to ensure the WBS is defined with appropriate software visibility. The objective is to establish a program WBS which mirrors the actual work/product to be developed. Place software elements in the WBS at appropriate levels which are consistent with the developed product architecture and hierarchy. That is, do not create artificial “software-only” WBS elements for the purpose of facilitating easy collection and roll up of “software-only” data.

- Collaborate with Program Management and Financial Management to assess and understand the impacts or limitations of selected earned value reports (CPR vs. CSSR) and establish reporting requirements which facilitate the level of software insight desired. If the software elements, appropriately placed in the WBS to reflect the to-be built product design, do not facilitate the desired identification and reporting of software status (e.g. system reporting at level 3 only and software elements are at lower levels), then special reporting requirements should be implemented to achieve the desired software earned value reporting. This special reporting could, for example, require collection and amalgamation of software earned value data from across the numerous WBS elements containing software for reporting as a single software report, or alternately, reporting of software earned value status could be required at the subsystem level to provide more detailed software insight. It is important to establish reporting levels with enough granularity such that visibility into significant software development problems at the lower level components is not lost (washed out) in the reporting roll-up.

- Assure the program cost accounts and work packages reflect the WBS structure and are consistent with the software effort and schedule estimates for the program.

- Assure the work package schedules integrate into and are consistent with the program IMS.

- Collect and review earned value data to determined actual software effort and schedule status against the budgeted effort and schedule for the software components (Schedule and cost variances).

- Use the software earned value reporting to identify specific or systemic software development problem areas for focused monitoring or resolution.

- Consider use of on-line collaborative (real or near-real-time) access to earned value data to provide timely insight into software development status.

For more information on EVMS, refer to http://www.acq.osd.mil/pm/, or the Defense Acquisition Deskbook at http://acc.dau.mil/simplify/ev.php?ID=1435_201&ID2=DO_TOPIC.

3.7 Establish and Manage Software Requirements

The purpose of establishing and managing requirements is to ensure they are defined, complete, stable, and verifiable prior to designing and developing the software. Requirements, at any level, must be both consistent and traceable to applicable higher level (e.g. system and subsystem) requirements and lower level (e.g. subsystem and software) design and implementation, and must also be traceable to the verification methodology. Modern acquisition and evolutionary development approaches allow for the gradual maturation of requirements at the system level. However, once the development of a particular spiral or increment of capability is started, any change to the requirements will impact the near-term product. Requirements identification, allocation and verification is often a shared responsibility between the acquisition Program Office and the development contractor. The Program Office will normally focus on the higher system or subsystem level requirements identification and allocation and the contractor will focus on the subsystem tiers (down to hardware/software or hardware/software sub components) for which they are responsible.

3.7.1 Requirements Basics

Establishment and management of software requirements includes the following steps:

Gather top level CS&S requirements from identified users of the system. Assess each top-level requirement for feasibility of implementation and consistency within program constraints. If the requirement is impossible to implement within cost and schedule constraints, identify this as an expectation management issue.

Allocate all system, subsystem, and interface requirements to appropriate hardware and software configuration items. Ensure each requirement:

is written as a single definitive statement has a unique identification number for tracking purposes can be traced to a higher level source requirement or analysis (if a derived requirement) has been allocated to a specific computer software configuration item (CSCI)

Ensure that for each performance requirement there is a corresponding verification requirement. The specification section 4 verification requirement should define a means of objective measurement (Note: This goes beyond identifying the category and method of verification in the section 4 verification matrix. This is not a set of test procedures but rather a short statement of the method and conditions for demonstrating the requirement has been satisfied. This has two major benefits to the program. It establishes a set of test requirements upon which to build a test plan, and it forces ambiguity out of the section 3 performance requirements.)

Identify analyses, trade studies, prototyping, and demonstration efforts for risk reduction. Consider the following:

-- Completeness (Are all higher level requirements allocated?)

-- Consistency (Is the content, format, and notation from requirement to requirement, and from specification to specification, similar?)

-- Feasibility (Can the requirements be met?)

-- Verifiability/testability (Can the requirement be objectively verified?)

-- Human factors (Is the design consistent with sound human factors principles?)

Complete the definition of derived software requirements and examine them for consistency with system requirements, feasibility, and the effects of various implementation strategies (Note: derived requirements are often a significant source of software size growth.)

Apply early aggressive activities to verify that software intended for reuse satisfies requirements and will not lead to additional growth

Verify developers flow top level requirements down to lower level specifications and to lower-tier suppliers, including software configuration item specifications

Verify CSCI-to-CSCI and CSCI-to-HWCI interface requirements identification and definition

Verify that an adequate and compatible set of tools are in place to capture multi-level requirements (systems down through software) and…