Attachment 13 PRR Plan 4.5.21.doc

N00024-22-R-5324

ATTACHMENT 13

PRODUCTION READINESS REVIEW PLAN

1.0 GENERAL

1.1 Introduction Per the SOW, this attachment provides elaboration and further guidance for conducting Production Readiness Reviews (PRRs) for VLS canister production.

The PRRs are an optional CLIN under this contract, which is exercised solely at the Government’s discretion.

1.2 Purpose of PRRs

The PRRs are a risk mitigation tool that helps assure that the Contractor and its subcontractors have accomplished adequate production planning and preparations to produce Canisters and associated equipment that meet performance requirements in accordance with the contract schedule and without unacceptable risks. Preliminary and final PRRs may be conducted.

The preliminary PRRs shall be conducted 60 to 90 days after award of contract, and the final PRRs shall be conducted 120 days prior to first article delivery. PRRs of subcontractors will be conducted prior to the PRR of the Contractor, in order to examine during the Contractor PRR the Contractor’s responses to significant issues discovered during the PRRs of the subcontractors.

2.0 Entrance Criteria The following Entrance Criteria shall be accomplished to the Government’s satisfaction before either preliminary or final PRRs may start:

Contract Award.

Formal identification of subcontractors by Contractor.

For PRRs of subcontractors, 30 day formal notice of PRR from Contractor to subcontractor.

Mutually agreeable starting date & duration for PRR confirmed.

Confirmation of necessary hosting preparations by the company under review.

PRR planned in accordance with this attachment.

Final tailored set of PRR questions provided to company under review.

PRR team (Government and Contractors) identified and participation confirmed.

PRR funded.

3.0 Exit Criteria For a PRR to be evaluated as successful, the following Exit Criteria shall be accomplished to the Government’s satisfaction:

No critical risks are identified. (If critical risks are identified, a follow up assessment/mini-PRR is recommended.)

No major risks are identified of which the Prime Contractor was unaware. (If such major risks are identified, a follow up assessment/mini-PRR is recommended.)

The accumulation of major risks does not jeopardize the likelihood of successful production in the Government’s determination.

Manufacturing capability demonstrates a satisfactory basis for proceeding into production.

All spot checks of objective evidence completed.

Contractor’s Response Report submitted within 30 workings days of completion of the PRR.

All risks corrected or have a feasible corrective action plan acceptable to the Government, submitted within 30 workings days of completion of the PRR.

4.0 PRR Organized by Functional Areas Each PRR should examine key aspects of manufacturers’ operational and supporting functional areas that must be successfully accomplished for production. This examination is performed by asking questions to verify key aspects in each functional area and obtaining answers supported by objective evidence. The PRR questions are tailored to VLS canister manufacturing, and should be answered in the PRR. Additional questions should be added to this list to further tailor to the subcontractors’ role, relevant past information gained during the proposal process, such as information in Contractor Performance Assessment Reports (CPARs) submissions required by the Past Performance section, and other pertinent information that may be discovered during the solicitation process.

The Functional Areas are:

A. PRODUCIBILITY

B. MANUFACTURING PLANNING

C. MANUFACTURING SCHEDULES

D. CONFIGURATION MANAGEMENT

E. QUALITY ASSURANCE

F. MATERIAL PROCUREMENT & SUPPLIER CONTROL

G. FACILITY AND EQUIPMENT

H. PRODUCTION PROCESSING & FABRICATION

I. CALIBRATION, TOOL AND TEST EQUIPMENT

J. NON-CONFORMING MATERIAL CONTROL

K. COMPLETED ITEM INSPECTION AND TEST

L. HANDLING, STORAGE AND DELIVERY

The minimum PRR questions organized by Functional Areas are listed in Appendix A to Attachment 8 at the end of this document.

4.1 Presentation_of_Answers The company (contractor or subcontractor) that is the subject of the PRR shall present their answers to the PRR questions in a conference room setting. This presentation should be of the same general style as presentations given for the Semi-Annual reviews, with the exception that uploads of the presentations prior to the date of the PRR are neither required nor desired. The company should use their own standard conference room systems and processes to convey the answers in the most effective manner possible. Notwithstanding that, multiple presenters are appropriate, as subject matter experts should present their Functional Areas of expertise.

The audience for presentations of PRRs of Contractors will be Government and a representative cross-section of Contractor personnel responsible for the project, both management and technical subject matter experts. The audience for presentations of PRRs of Subcontractors will be Government, a representative cross-section of Contractor personnel responsible for the project, and a representative cross-section of subcontractor personnel responsible for the subcontractor’s portion of the project.

4.2 Objective_Evidence While a PRR is not an audit, answers should be supported by objective evidence as much as possible. Therefore, answers should not be simple Yes/No statements, and should contain sufficient detail to provide confidence that the process is understood and under control. A helpful and appropriate mindset to take when responding to the questions is to answer them as your company would want a critical supplier to you to answer the questions. Illustrations from actual plans, policies, procedures, working paper, and other documents under control and in use for the project should form part of many answers. Key documents should be readily accessible for consultation if needed in the conference room. The government reserves the right during the PRR to conduct spot checks on the factory floor and in support function spaces. Those spot checks shall be coordinated in a mutually agreeable manner to minimize disruption of operations, but need to be completed to successfully accomplish the PRR.

5.0 Team_Size_and PRR_Duration The entire PRR team should be in the conference room and absorbing the company’s presentation of answers . However, the PRR team can be divided into teams for detailed evaluation and processing of the company’s presentation and answers. The exact team structure can be tailored, but a typical set of teams is: Management Team; Manufacturing Team; and Quality Assurance and Configuration Management Team. (The company that is subject of the PRR should provide liaisons for each team.) Additionally, the PRR team should have a PRR Chairperson.

Therefore, with 1 or 2 people per team, and a Chairperson, the PRR team size should be 4 to 7 people, depending on the tailoring to the size of the company and the complexity of their product. The anticipated duration of a PRR is 2 full days, with ½ day latitude either way to accommodate a smaller scale PRR of a subcontractor or a more challenging PRR of a Prime Integrator Contractor.

6.0 PRR_Findings There are two categories of PRR findings. A “deficiency” is a violation of contractual obligations, including internal company policies or procedures and adopted standards relevant to the product. An “observation” is a non-compliance with Industry Best Practice. This “deficiency”/”observation” dual categorization is separate from the severity of the finding as a risk, of critical, major, minor. As noted in footnotes above, a Critical Risk is one where the probability of unsuccessful contract performance is high. There is likelihood of significant disruption of schedule, increase in cost, or degradation of performance, even with special contractor management attention and emphasis. A Major Risk is one where one where the probability of unsuccessful contract performance is significant. There is significant potential to cause disruption of schedule, increase in cost, or degradation of performance. Considerable contractor management attention and emphasis will be required to mitigate the risk. A minor risk still has potential to impact schedule, cost, and performance if not addressed, but does not rise to the level of critical or major.

Just because a finding is an observation does not mean it is not important. Observations can be more significant than deficiencies because they may be wider in scope or more forward looking in highlighting not taking advantage of industry best practice Findings may be generated throughout the PRR on-site visit. The PRR team should document in-writing the presentation spot-checks of the facility and manufacturing. If desired by the company under review, the PRR team should conduct an informal out-brief of the findings at the end of each day. To allow the PRR team group consultation and sub-team preparation for that, the day’s presentation should end approximately 1 hour early and prior to the informal out brief. The informal out brief is best conducted verbally, without diverting time to making viewgraphs or other elaborations; and final categorization of the findings as “deficiency”/”observation or risk severity is not necessary at this point. The intent of the informal out brief is to pass the key finding concerns to the company so there are no surprises, and potential misunderstandings can be corrected.

On the last day of the PRR there is a formal out-brief. The PRR team should present all findings to the company and rest of audience in viewgraph format. Elaborate viewgraphs are not desired, but the presentation should be professional and complete. This presentation should in effect be a complete package of findings so that the company can proceed on preparing corrective action. The contractor shall only receive information during the out-brief; there is no opportunity for debate or discussion between the parties regarding the findings. Disagreements with findings by the Company should be instead included in the written response of corrective actions forwarded through contractual channels Appendix A to Attachment 13 A. PRODUCIBILITY Functional Area

A.1 Are there any new required manufacturing methods, or any that have not been proven by the company in production quantity rates in the past? If yes, elaborate.

A.2 Has the Manufacturing Group completed review of the TDP and translated the TDP requirements into specific manufacturing/production/operations that can be release to the work force? If not, what is the percentage of completion and final completion date?

A.3 Have written shop instructions been developed and released for manufacturing personnel?

A.4 Does the Manufacturing Group review and sign-off on all assembly documentation releases? If so, does the sign-off signify producibility review/approval?

A.5 To what extent will shop instructions be developed for manufacturing personnel? Please describe graphical or pictorial aids used during production.

A.6 How are vendor or subcontractor producibility problems identified and managed, including critical dependencies between any projects’ producibility issues and suppliers? How do you perceive current vendor/subcontractor producibility risk areas?

B. MANUFACTURING PLANNING Functional Area B.1 What is the estimated production cycle time per unit? Can the manufacturing line support the planned production schedules? Please provide a summary of the analysis used to design the floor plan and calculate the production cycle time.

B.2 Are there qualified people available to support the Project’s manufacturing Program and schedule? Is there a manpower planning plan? Please discuss how shop loading was determined.

B.3 Please identify the critical outside vendors required to support the manufacturing schedule. Can they meet the Statement of Work criteria? Do they have the capability to support the program? Please explain your back-up plan for your critical outside vendors? (The Statement of Work here means both the SOW from government to your company, and SOWs or Purchase Orders or any other equivalent imposed by your company on vendors.)

B.4 Please provide a manufacturing organizational chart with the identified areas of responsibility.

B.5 What is the role of Quality Assurance in the manufacturing planning activity?

B.6 What Engineering resources are available to support the Program? What engineering support is available when production problems occurred on the factory floor?

B.7 Is there a risk management plan for manufacturing? Please explain how the risk management plan for manufacturing is implemented. How does it fit in the overall Risk Management Program?

B.8 Are there any new start or existing programs competing for resources during the period of performance defined for this program?

C. MANUFACTURING SCHEDULES Functional Area C.1 Does an Integrated Master Schedule or a master manufacturing schedule or exist?

Do lower level subassembly schedules exist that support end-item delivery requirements? Are these schedules in conflict with requirements of other programs?

C.2 Do key manufacturing milestone dates meet program schedule while being realistic in terms of your manufacturing operations and capabilities? Please elaborate.

C.3 Has a critical path for planning and hardware deliveries been established? Please demonstrate.

C.4 What information is provided to upper management and also to the government to give them schedule performance vs. plan visibility? How is this data validate and updated?

C.5 Based upon your review of the TDP, other than as identified in the TDP, are there any other long lead or critical items? What are your criteria for designation as critical or long lead items? How will your company assure that long lead items and critical items from suppliers meet VLS canister deliveries dates? Do you have second sources for critical and long lead time items?

C.6 What percentage of your labor resources is committed to support this program? Will more than one 40 hour shift be necessary? What support manpower has been assigned?

C.7 How will production priorities and resource allocation between this program and other customers’ programs be resolved?

D. CONFIGURATION MANAGEMENT Functional Area D.1 What standard is your configuration management system based on?

D.2 Please explain the engineering change control process you will use for this project. Describe how a change will be received and processed. Show how you accomplish review, concurrence, and communication to manufacturing and method to implement/break in change in the manufacturing process. Where does the customer fit in?

D.3 What are the controls for the incorporation of engineering changes into production paper (such a work instructions and production processes, as opposed to engineering drawings)?

D.4 How will configuration control of production paper be maintained?

D.5 Please explain in moderate detail how do you will control the production baseline.

D.6 What and how are Configuration control requirements flowed down to subcontractors?

D.7 Please explain the process you will use to meet contract CDRL requirements for this project. Who is responsible?

E. QUALITY ASSURANCE Functional Area E.1 What standard is your quality assurance system based on?

E.2 Please provide an organizational chart that identifies the organizational elements responsible for performing the various quality tasks. How does the Quality Assurance organization relate with other departments? Is the chain of command effectively defined?

E.3 Please explain how the QA group has the authority and independence to identify and evaluate quality problems and initiate corrective action.

E.4 Does QA participate in program planning, process development and approval? Please elaborate.

E.5 Does QA monitor both the issuance of work instructions and compliance to them? Please elaborate.

E.6 How many QA FTEs are assigned to this project? Please briefly discuss QA personnel responsibilities, assignments (including degree of commitment to other concurrent projects) and experience.

E.7 Please explain and illustrate the receiving inspection program for materials and supplies. Please provide a process flow chart for material.

E.8. How is discrepant material handled at receiving?

F. MATERIAL PROCUREMENT & SUPPLIER CONTROL Functional Area F.1 Have all suppliers been identified and orders placed? How do you ensure that parts are purchased from only approved suppliers? ("Supplier" includes subcontractors and those vendors purchased from with a purchase order.)

F.2 How are supplier problems identified and managed? Are there any risk areas identified with the suppliers? Please include any potential risks due to delivery and schedule. Who are your suppliers of most concern, and why?

F.3 How do you rate your suppliers? How is that rating used in supplier source selection? Do you allow Research & Development, emergency, provisional, or other non‑standard status for suppliers; if yes please explain that sub-process.

F.4 Are there procedures for providing suppliers with appropriate data regarding unsatisfactory quality and for assuring that suppliers correct all nonconformance?

F.5 What quality requirement is invoked at suppliers? Are Quality Notes or equivalent included in purchase orders issued to suppliers?

F.6 Based upon your review of the TDP, what Diminishing Manufacturing Sources (DMS) items have you identified? What is your DMS strategy for these items?

F.7 Is your company prepare to conduct process certification or review for special process such as plating, bonding, and painting, if these process are performed at suppliers? Please elaborate.

G. FACILITY AND EQUIPMENT Functional Area G.1. How will this program affect the capacity of this plant? Is any facility improvement needed or anticipated?

G.2 Are all tools, equipment, and test equipment including GFE available to support production, and are they under the appropriate level of configuration management control? Please elaborate.

G.3 Will the test equipment meet the production rate requirement? What is the basis of your confidence? What analysis was done?

G.4 Is any special tooling needed at suppliers?

G.5 Are there adequate procedures for assuring that production tooling and test equipment are properly maintained?

G.6. What are the risks associated with equipment downtime, back-up machines and tools, and maintenance contracts?

H. MANUFACTURING PROCESSING & FABRICATION Functional Area H.1 Do the work instructions and inspection sheets provide criteria for determining whether production, processing and fabrication work is acceptable or unacceptable? Please explain in moderate detail.

H.2 What percentage of planned work and inspection instructions are complete?

H.3 Is approved inspection criteria provided for all required inspections and monitoring actions? Please elaborate, including traceability to the TDP, CIPS, and Parameters Document.

H.4 How is training correlated to project/TDP needs to assure no gaps exist? What training programs are in place? How will personnel be trained on the manufacturing, assembly and test requirements for this program?

H.5 What are the critical manufacturing processes? How will these processes be controlled?

H.6 What mechanism will be used to control critical manufacturing processes at outside suppliers?

H.7 Are appropriate manufacturing environments provided for this program? Please elaborate.

I. CALIBRATION, TOOL AND TEST EQUIPMENT Functional Area I.1. Are the required tool, gauges, testing, and measuring equipment necessary to assure that products meet technical requirements available for use? Please list any potential choke points.

I.2 . Do procedures exist for the proper calibration of tooling, test and measuring equipment?

I.3. Do procedures require that these devices be inspected on a regular basis to determine that they are of the required accuracy? Please define the recall system.

I.4. Who does calibration? What calibration standard do they comply with?

I.5 Does the company require its suppliers to have a system that assures the accuracy of their test and measuring equipment?

I.6. Please explain the process to follow-up on any released products that were built with out-of-calibration tooling or test equipment. What is the timeline for notification to the customer in this situation?

I.7. Is any production tooling used as inspection equipment? If yes, is such production tooling proved for accuracy prior to use? Is such tooling re-inspected at intervals established in a manner which assures the adjustment replacement or repair of the tooling before it becomes inaccurate?

J. NON-CONFORMING MATERIAL CONTROL Functional Area J.1 Please explain your process for identification, segregation, and disposition of non‑conforming material. What role does QA have in the process?

J.2 Are holding areas adequate for the segregation, security and storage of non‑conforming material? Please elaborate.

J.3 Please explain the Material Review Board (MRB) process you will use for this project? Please identify membership (by role/title) for this board. Is the customer involved? If so, how?

J.4. How are rework and repair procedures generated and who has the approval authority?

J.5 Based upon your review of the TDP, have any documented repair procedures for non‑conforming material been generated? If so, have they been submitted and accepted by the government?

K. COMPLETED ITEM INSPECTION AND TEST Functional Area K.1 Do instructions provide for a final inspection and test of completed items for overall quality?

K.2 Do procedures adequately provide for the timely correction of final inspection test problems or deficiencies, and prompt reporting to appropriate management for follow-on action? Please elaborate.

K.3 Is there an integrated test plan? Does it identify all tests, screens, burn-in, etc. done by your company and suppliers (as appropriate)? Has a requirements verification matrix been developed and distributed showing which tests verify which requirements, and which company is responsible?

K.4 Is there a requirement for re-inspection and retest for all items that are reworked, repaired or modified for defects found during initial testing?

K.5 Has time been allocated in your schedule for rework and retest? Please illustrate using your scheduling tools.

L. HANDLING, STORAGE AND DELIVERY Functional Area L.1 How are the VLS canister’s specific Packaging, Handling, Storage, and Transportation (PHS&T) requirements integrated into your PHS&T system and process?

L.2 Is there a procedure to assure the proper cleaning/preservation for items that could degrade during fabrication or interim storage?

L.3 Do procedures provide for the proper identification and labeling of all material to be stored or shipped?

L.4 Are there any “limited shelf-life” materials? If yes, what are your procedures that provide for clear identification of "limited shelf-life" materials and for their disposal when shelf life has expired?

� A Critical Risk is one where the probability of unsuccessful contract performance is high. There is likelihood of significant disruption of schedule, increase in cost, or degradation of performance, even with special contractor management attention and emphasis.

� A Major Risk is one where one where the probability of unsuccessful contract performance is significant. There is significant potential to cause disruption of schedule, increase in cost, or degradation of performance. Considerable contractor management attention and emphasis will be required to mitigate the risk.

� It is recognized that it is not practical for all corrective actions to be completed within 30 days. For example, if the corrective action is to obtain additional specialized equipment to meet surge capability, that may take more than 30 days, but the VLS canister program may find acceptable the corrective action proposed if it demonstrates firm commitments to obtain the equipment within the needs of the overall schedule. On the other hand, if a risk was identified of some crucial long lead item not yet ordered, the controlling impact of long lead items on delivery schedule may cause a critical risk that a plan alone cannot correct.

�? do you mean submitting the PRR answers in advance of the meeting?

�Absorbing the answers? Recommend “… in the conference room prepared to interactively engage with the government regarding the presentation.”

�Are you saying that some observations are more significant than SOME deficiencies?

�I don’t understand the last part of this sentence.

�I’m confused by this last sentence. Doesn’t the government make the findings?