RRL_FY24_CC_PSC_Brief_Final.pdf

Ready, Relevant Learning FY24 Content Conversion Aviation Pre-Solicitation Conference

12 June 2023

Presented to: Industry Partners

Presented by: RRL FY24 CC Aviation Acquisition Team

Virtual Brief Rules of Engagement

• In-Person: Please sign the sign-in sheet in the room

• Virtual: Please sign-in using the MS Teams chat feature provide Name, Title, and Company Name

• Mute audio unless speaking

• One speaker at a time

• Utilize chat feature:

Ask questions Provide comments

Chat

Government Team

Name Role

CAPT Robert Betts TSPM

Mary Driskel Deputy TSPM

Kevin Jones Acquisition Lead

LCDR Denver White Acquisition Lead

Kelsey Denove IPT Lead

David Kors Procuring Contracting Officer

Austin Brookshire FY24 CC IPT Contract Specialist

Antonio Fraser Lead Engineer

Brooke Davis FY24 CC Lead ISD, Pre-Contract Award

Jackie Reynolds FY24 CC Lead ISD, Execution

Industry Day Agenda

Introduction RRL Program Overview Ground Rules

Pre-Solicitation Activities Solicitation Overview

Requirements Explanation

TSPM

CAPT Betts Austin Brookshire

Kelsey Denove Brooke Davis Antonio Fraser

Introductions

Presented by Kelsey Denove

RRL Program Overview

Presented by CAPT Betts

Ground Rules

Presented by Austin Brookshire

Ground Rules

• Primary Purpose – understanding the content of the draft solicitation

• All information released to date and during this conference is to be considered draft and for informational purposes only.

• Internal source selection plans and procedures will not be discussed.

• Today’s remarks by Government officials involved in the RRL FY24 Content Conversion Aviation procurement should not be considered a guarantee of the Government’s course of action in proceeding with the project.

• The information shared today reflects current Government intentions and is subject to change based on a variety of circumstances, including internal and external comments. The formal (final) solicitation is the only document that should be relied upon in determining the Government’s requirements.

Ground Rules (cont.)

• The Government may answer questions as the presentation progresses.

– All questions may not be answered today due to time constraints.

– After today, questions shall be submitted in writing to the primary and alternate POCs:

• Primary POC: Austin Brookshire, austin.e.brookshire.civ@us.navy.mil

• Alternate POC: David Kors, david.a.kors.civ@us.navy.mil

• Do not wait for Final RFP release to submit questions and concerns.

– Review the Draft RFP and subsequent Final RFP in their entirety.

– Submit questions/comments to give the Government an opportunity to clarify our requirements and address concerns with the RFP.

• It is the Government’s intention to formally release answers to Draft RFP questions. If necessary, changes will be made via the Final RFP release.

Pre-Solicitation Activities

• Completed

• Sources Sought Notice and one-on-one meetings (Nov - Dec 2022)

• Draft SOW posted to SAM.gov (28 February 2023)

• Acquisition Strategy shared with industry via SAM.gov (20 March 2023)

• Full-and-open competitive procurement

• Contract Type anticipated to be Firm-Fixed Price (FFP)

• Potential Incentive Clause for Early Delivery

• Contract Award by the end of December 2023

• Notional Pre-Final RFP Schedule updates (Apr, May 2023)

• NAWCTSD PALTs and today's Pre-Solicitation Conference

• Released Draft RFP, with Sections L&M (6 June 2023)

• Pre-solicitation/Industry Day Conference (12 June 2023)

Pre-Solicitation Activities (cont.)

• Planned

• Draft RFP Questions DUE by COB 16 June 2023

• Release of Final RFP planned for 30 June 2023

• Contract Award by the end of December 2023

Solicitation/Contract

• Contract Type: Single Award Indefinite Delivery Indefinite Quantity (IDIQ), predominately Firm Fixed Price (FFP) CLINs

• Unrestricted Competition under FAR Part 15

• Evaluation Factors

• Past Performance

• Price

• Small Business

Solicitation Highlights Section B

• Estimated CLIN quantities will be provided at Final RFP release

• Overall ceiling amount of the IDIQ contract is currently estimated at $161M

Section F

• The basic IDIQ contract has a five (5) year ordering period from the time of award. The delivery dates for items provided under this IDIQ contract will be established in the individual delivery orders.

• While the IDIQ contract has a five (5) year ordering period, each CLIN has a specific ordering period tied to its use on this IDIQ contract. Below is an overview of the ordering period availability for each CLIN.

– CLIN 0001 – Date of Contract award – 12 months After contract Award*

– CLIN 1010 – Date of Contract award – 24 months After contract Award

– CLIN 1011 – Date of Contract award – 24 months After contract Award

– CLIN 1012 – Date of Contract award – 24 months After contract Award

– CLIN 1013 – Date of Contract award – 24 months After contract Award

– CLIN 1014 – Date of Contract award – 24 months After contract Award

– CLIN 1015 – Date of Contract award – 24 months After contract Award

– CLIN 1016 – Date of Contract award – 24 months After contract Award

– CLIN 1017 – Date of Contract award – 60 months After contract Award

– CLIN 1018 – Date of Contract award – 60 months After contract Award

– CLIN 1019 – Date of Contract award – 60 months After contract Award

– CLIN 1020 – Date of Contract award – 60 months After contract Award

– CLIN 1021 – Date of Contract award – 60 months After contract Award

• *CLIN 0001 award of the Post Award Conference (PAC) will occur only once and on the first Delivery Order (DO) awarded.

• The timeframes reflected above identify the Ordering Periods for each CLIN. A DO may be issued at the corresponding Section B CLIN Unit Prices within the applicable Ordering Period dates reflected above, although the delivery date could extend beyond the contract ordering period.

Section F (cont.)

• The Government's required Content Conversion durations once awarded on a delivery order during IDIQ contract execution is presented in Section F.4.

– Durations are months after award of a delivery order.

– At delivery order award, the PCO may lengthen a Content Conversion

Duration unilaterally at no cost impact.

– In the event a shorter Content Conversion Duration is required at delivery order award, the PCO will coordinate any potential impacts with the contractor.

Section H

• Ordering Procedures

• When issuing a delivery order, the Government will identify the number and type of courseware training hours required for each rating/path/course in SOW Appendix A for each delivery order. CLINs 1010 through 1016 in the IDIQ provide the unit pricing for those hours of rating/path/course training, by immersion level. The aggregate of that unit pricing is captured in the delivery order as a single CLIN for the courseware of that rating/path/course.

• At time of delivery order award or modification, the Government will provide the RRL FY24 CC contractor with a delivery order SOW to include the quantity of hours of IMI, by level/type in Attachment (1) Appendix A that will be provided at the delivery order level for the rating/path/course.

• The Appendix A will also specify the quantity of VSIM media by type (A through D) when ordering VSIM Media.

• An estimated travel cost in support of IMI development will also be provided, which will be invoiced at Cost via CLIN 1019 during performance (no fee).

• Applicable CDRLs for the delivery order will also be provided.

• Train the Trainer and Pilot CLINs will be negotiated separately after DO award via a modification.

Section H (cont.)

• Incentive for Early Content Conversion CLIN Deliveries

• The Government will pay an incentive fee for each applicable Content Conversion CLIN that is delivered early in accordance with all contract requirements.

• 10% of Content Conversion CLIN value to be paid for incentive if delivery and unconditional acceptance occurs at least 60 calendar days prior to CLIN delivery date in Section F of delivery order.

• 5% of Content Conversion CLIN value to be paid for incentive if delivery and unconditional acceptance occurs at least 30 calendar days prior to CLIN delivery date in Section F of delivery order.

S

Section H (cont.)

• Incentive for Early Content Conversion CLIN Deliveries (cont.)

• Meeting the Early Delivery Goal includes achieving all contract requirements for the applicable CLIN.

• Unconditional acceptance indicates that all contract requirements (other than perhaps the delivery date) are met.

• Conditional acceptance indicates that some requirement(s) is not met.

• The contractor is not owed an incentive fee for CLINs accepted conditionally.

• The contractor may be eligible for either the 10% incentive fee or the 5% incentive fee, but not both.

• Meeting the Early Delivery Goal for each applicable CLIN-level end-item courseware is solely the responsibility of the contractor.

Section H (cont.)

• Incentive for Early Content Conversion CLIN Deliveries – Example

CLIN 000X (Example Courseware Rating (ECR) Content Conversion) is priced at $1,000,000 with a required delivery date in Section F of the delivery order that is 365 days after award of the delivery order. If the contractor delivers the CLIN-level end-item courseware in accordance with all contract requirements 335 days after award of the delivery order (30 calendar days prior to CLIN delivery date in Section F of the delivery order), then the contractor will be eligible to collect an incentive in the amount of 5% of the value of CLIN 000X, which is

$50,000.

Notional Ordering Schedule

Evaluation Criteria

Factor 1: PAST PERFORMANCE

Factor 2: PRICE

Factor 3: SMALL BUSINESS

• The Past Performance Factor is significantly more important than the Price Factor.

• The Small Business Factor will be evaluated on an “acceptable/unacceptable” basis. The Small Business Factor must be acceptable to be eligible for award but is not a component of the tradeoff.

• The Administrative Volume is not a component of the tradeoff but some of the information may be necessary for an offeror's proposal to be acceptable for contract award.

Evaluation Criteria – Past Performance

• Assign a Performance Confidence Assessment Rating that will address the Government's level of confidence in the offeror's ability to successfully perform the required effort based on the offeror's (including principal subcontractors) relevant past performance and systemic improvement.

• There are three (3) aspects to the past performance evaluation:

• Recency

• Relevancy

• Demonstrated Past Performance

Evaluation Criteria – Past Performance

• The Government will evaluate a maximum of four (4) total Government or commercial past performance contract references of the prime offeror, JV team members, and principal subcontractors' recent and relevant past performance.

• Past performance references performed within five (5) years of date of issuance of the final solicitation.

• Principal Subcontractor = performing at least 20% of total contract price.

• Single award IDIQs and orders under multiple award IDIQs, BOAs, and BPAs will each count as one reference.

• Offeror may identify up to a maximum of four (4) orders under a single award IDIQ. Those orders in the aggregate will count as one reference.

• Orders under a Multiple Award Contract (MAC) will each count as a single reference. Similarly, orders awarded under BOAs or BPAs (or commercial equivalents) will each count as a single reference.

Evaluation Criteria – Past Performance

• Recency is defined as performance occurring within five (5) years of the final solicitation release date identified on Block 5 of the SF-33 of the solicitation.

• If there is at least one day of performance within the five (5) year timeframe, the performance of the entire contract will be considered.

• Each order submitted under a single award IDIQ must independently meet this recency requirement.

• Relevancy

• Scope - Supporting SOW paragraphs

• Interactive Multimedia Instruction (IMI) Media Types (SOW 3.5.2, does not include subordinate paragraphs)

• Complex Media Type requirements (SOW 3.5.3, does not include subordinate paragraphs)

• Requirements similar to the utilization of an agile or hybrid agile methodology approach in development of IMI and/or Complex Media (SOW 3.5.3.2.1, 3.5.3.2.3, 3.4.6.4, and 3.5 (does not include subordinate paragraphs)).

Evaluation Criteria – Past Performance

• Relevancy

• All past performance references must include the CAGE code in the cited past performance reference, and the cited past performance reference cannot include a higher tier contract.

• For past performance references where the contract is not directly with a Government agency and no CAGE code is identified, the past performance reference must include the company name and address that corresponds to the offeror's CAGE code that is identified in Attachment L-6.

Evaluation Criteria – Past Performance

• Relevancy

• The paragraphs cited in Attachment L-3 must be pinpoint citations to the lowest level paragraph that contains exact scope the offeror believes to be similar to the solicitation requirements. Do not provide high-level, broad citations.

• The offeror must highlight (in the supporting documents it provides) the exact content in those cited paragraph that it believes is similar to the solicitation requirements.

• The offeror shall not provide any self-explanation or description of the alleged past performance effort or what the offeror allegedly performed under a past performance reference.

Evaluation Criteria – Past Performance

• Past Performance Relevancy Ratings

• Very Relevant

• Relevant

• Somewhat Relevant

• Not Relevant

Evaluation Criteria – Past Performance

• Demonstrated Past Performance (DPP)

• The Government will evaluate the offeror's DPP of all past performance references found to be Recent and either Very Relevant, Relevant or Somewhat Relevant in the following areas:

• Quality of Product or Service.

• Schedule.

• Program Management and Subcontract Management.

• Utilization of Small Business.

• CPARS – Contractor Performance Assessment Reports

• CPAQs – Contractor Performance Assessment Questionnaires

Evaluation Criteria – Past Performance

• The Government will assign one of the performance confidence assessment ratings to the offeror's Past Performance Factor based upon the Government's confidence that the offeror will successfully perform the solicitation's requirements based upon the offeror's Recent and Relevant past performance record.

• In the case of an offeror without a record of past performance or for whom information on past performance is so sparse or not available, the offeror will receive a "Neutral" rating.

• Only past performance references assessed a Very Relevant, Relevant, or Somewhat Relevant rating will be considered in assigning a Performance Confidence Assessment rating.

• Performance Confidence Assessment Ratings:

• Substantial Confidence

• Satisfactory Confidence

• Neutral Confidence

• Limited Confidence

• No Confidence

Evaluation Criteria – Past Performance

• Required Section L Attachments for Past Performance

• Attachment (L-2) Contractor Performance Assessment Questionnaire

(CPAQ)

• Attachment (L-3) Relevancy Information

• Attachment (L-6) Offeror Summary Table (also required for Price & Administrative Volumes)

Evaluation Criteria – Price/Admin

Attachment (L-6) Offeror Summary Table

Evaluation Criteria – Price

• Provide FFP Unit Pricing by IMI and Complex Media type

• Complete Attachment (L-4) Pricing Form (yellow shaded cells only)

• No other entries within Attachment (L-4) are permitted.

• The sum prices for each CLIN as outlined in Attachment (L-4) CLIN Pricing Form will represent the Total Evaluated Price for evaluation purposes.

Evaluation Criteria – Price

• Price Reasonableness

• Anticipated to result based on adequate price competition

• Balanced Pricing

• Proposals will be evaluated to determine if CLINs are materially unbalanced

IAW with solicitation

• Total Evaluated Price (TEP)

• The TEP will be the sum of the estimated prices provided by the Government for CLINs 1017, 1018, and 1019 combined with the sum of the extended prices for CLINs 0001, 1010, 1011, 1012, 1013, 1014, 1015, 1016, and 1021.

• The extended price for a CLIN (except CLINs 0001 and 1021) is the unit price multiplied by the unit quantity. Unit quantity is provided by the Government.

• The price for CLIN 1021 is included in the TEP and is automatically calculated in Attachment L-4 as 10% of the sum of CLINs 1010 - 1016

Evaluation Criteria – Small Business

• Each offeror (Large and Small Businesses) shall complete the Small Business Participation Commitment Document (SBPCD) provided as Attachment L-5.

• The SBPCD shall identify a minimum small business participation goal of 25% of the total evaluated price and list the planned small business participants.

• Completeness

• Evaluation to ensure all information requested in Section L Part B 3.0 is provided.

Evaluation Criteria – Small Business

• Acceptable

• Unacceptable

• The offeror’s Small Business Evaluation rating must be “Acceptable” to be eligible for award.

Evaluation Criteria – Administrative

• Administrative Requirements

• The Government will evaluate the offeror's proposal to determine whether the offeror completed the information requested in Section L Part B, paragraph 4 of the solicitation.

• Failure to submit completed and signed SF33 for the basic solicitation and SF30(s) for all amendments is a deficiency.

• Exceptions and Deviations are deficiencies.

• A proposal that contains an exception or deviation is unawardable.

Evaluation Criteria – Administrative

• Notice of Organizational Conflict of Interest (OCI)

• Refer to provision ORGANIZATIONAL CONFLICT OF INTEREST INSTRUCTIONS (NAVAIR) (MAY 2023) in Section L.

• All offerors are required to affirm whether an Actual OCI (AOCI) or Potential OCI (POCI) exists among themselves or any proposed subcontractors.

• If the prime offeror or subcontractor has an identified AOCI or POCI, the offeror shall notify the PCO of this solicitation in writing and include an OCI Mitigation Plan of the contractor who has the identified AOCI or POCI in accordance with the provision in Section L.

• If an OCI Mitigation Plan is submitted, it will be reviewed to ensure it addresses the provision in Section L and that as a result of the implementation of the OCI Mitigation Plan, there is no unfair competitive advantage identified.

Typical Proposal Issues

• Proposal Instructions are not followed

• Information not provided as requested

• Too much information that wasn’t requested or desired

• Statements in the proposal are not well supported

• Proposals are not well organized

• Does not follow Proposal Instructions structure and makes the evaluator hunt for the information

• Past Performance POCs are not current

• Not signing RFP or acknowledging all amendments

• Not providing the appropriate Exceptions and Deviations Statements as outlined in the solicitation

Requirements Explanation

Presented by Kelsey Denove, Brooke Davis, Antonio Fraser

IMI Development

RRL Content Conversion Goal

Content Conversion Considerations

• Content Conversion is more than content migration.

• Program misconception Technology insertion = automagical increase in training effectiveness

• Truth

Appropriate Media

Technology Insertion

Modern Design

Training Effectiveness

Modernization is the infusion of proven learning strategies with technology to bolster the Sailor’s experience. See RRLGM #22-0001 - Practical Applications of Leaming Science (PALS)

Prototype Requirement

• Previous Content Conversion Contracts required model products

• To align to the RRL Process Manual, FY24 CC will requires a Prototype

– Requires all AIM documents for one lesson

– Requires full functionality, error free, and pass GCAT

– Requires demonstration of all strategies of the integrated curriculum

– Requires compliance with the NETC Style Guide

• IAW the SOW, Prototype delivery requirement follows:

– 1 course per rating

– 1 Prototype Lesson IFIT IMI 2 (w/TCSP)

– 1 Prototype Lesson SDIT IMI 3 (w/TCSP)

Design Requirements

• Deliverables

– Government does not require design CDRLs per the SOW

• Collaborative Design Sprints

– LDS

– SSBs

– OSLs

• GFI will include

– IPRD

– IMRD

– Appendix A

– Initial Design Documents (TCCD/TPP)

Agile Methods

• Process that encourages an incremental approach to design and delivery

– Promotes improving how development teams (Government and Contractor) work together to deliver training materials

– SOW para: 3.4.6.3, 3.4.6.4, & 3.4.6.5

– For FY24, Agile will foster:

• Incremental design and development releases

• Continuous collaboration and elaboration on design

• Daily communications between cross functional teams

• Mutually agreed upon sprint/release cycles

• Demonstration of Minimum Viable Products

Design

Build

Test

ReleaseOperate

Design

Build Test Release Operate

Functional Training Materials

Agile Methods- Management

Waterfall Approach

Agile Methods

FY24 CC Hybrid Approach

*IMI CDRLs are only required for future sustainment

Agile Methods- IMI Development

The contractor shall present functional training materials in increments during sprints to allow the government to review and provide concurrence to move to the next sprint

Functional Training Materials

SOW Requirements & CDRL Requirements equate to a waterfall process.

The requirements must retrofit into an Agile concept to meet the Air Boss Request for modernized courseware delivered to the Fleet.

Functional Training Materials

Requirements Explanation Agile Methods- IMI Development

The contractor shall present functional training materials in increments during sprints to allow the government to review and provide concurrence to move to the next sprint

Agile Method SOW Requirements

Sprint Planning Delivery of IMS

Sprint Backlog Training Material Design, TCSD, TPSD, IMI

Sprint Cycle Training Material Development

Sprint Review IPT Meeting, CPSMR

Functional Training Materials Minimum Viable Product

Agile Methods- Complex Media Development

The contractor shall present functional training materials in increments during sprints to allow the government to review and provide concurrence to move to the next sprint

Functional Training Materials

Agile Method SOW Requirements

Sprint Planning Delivery of IMS

Sprint Backlog Agile Software Development

Sprint Cycle Agile Software Development

Sprint Review Release Backlog In-Process Review

Functional Training Materials Complex Media Design TIM

Complex Media Construct

Media CLINs Simulated

Environment Task

Interactions per scenario

VSIM A

Low Low 0 to 50

VSIM B Low Medium 51 to 120

VSIM C Medium Medium 51 to 120

VSIM D Medium High 121+

1. For cost estimating, assume each VSIM will be developed to teach, practice, and assess a single simulation.

2. Anything above medium in the simulated environment category will be priced as a separate CLIN.

3. The “task” column will be evaluated at the simulation level.

4. The “interaction per scenario” column will not be evaluated at the simulation level but per scenario.

NAWCTSD

Austin Brookshire FY24 CC (Aviation), Contract Specialist austin.e.brookshire.civ@us.navy.mil

David Kors Procuring Contracting Officer david.a.kors.civ@us.navy.mil

Point of Contact

51CUI

Questions

Back Up

Agile Methods- Engineering

Waterfall Elements: Complex Media

• IMS:

• IMS deliveries tracking program status, sprint durations, SETR events, IPT meetings and CDRL deliveries.

• IPT Meetings

• Release Backlog In-Process Review

(RBIPR)

• Complex Media Design Technical

Interchange Meeting (TIM)

• Complex Media In-Progress Review

• CDRLs

• Software Product Design (SPD)

• Test Procedures (TP)

• Test Inspection Report (T/IR)

• Software Product Package (SPP)

Hybrid Approach: Complex Media

• IMS

• Sprint durations, SETR, IPT meetings and CDRLs deliveries.

• Daily Stand-ups:

• Team communication during sprints.

• Biweekly Sprints

• Working sessions as established by IPT.

• Release Backlog In-Process Review

• Conducted after multiple sprints that result in software version release.

• Coded product, sprint backlog, derived requirement, software design and portion of design CDRLs.

• Work performed & path forward for next sprint cycle.

• Complex Media Design TIM

• Present proof of concept simulation

• Concurrence to proceed into final development.

• Complex Media In-Progress Review

• Demonstrate completion of working software.

• Testing Events