Attachment 8 - Additional Questions and Answers_15 September 2023.xlsx
XLSX spreadsheet 28 KB Posted
- Solicitation number
- W911S023RMCTU
About this file
This document contains questions and answers regarding a federal solicitation for the Army Virtual Learning Environment (AVLE) program. The solicitation seeks proposals for courseware development across five levels of interactivity, including courseware, checks on learning, practical exercises, pre-tests and post-tests. The pricing matrix was revised to correct quantities for each interactivity level and clarify pricing of test packages and validation plans. Questions addressed past performance documentation requirements, definitions of deliverables such as simulations and games, and ensuring accurate formulas in the pricing matrix.
View the file
Other files
Show all 36
On GovTribe
Work with this file
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
Questions
| AVLE 2.0 | ||||
| Solicitation Document | Reference/ | Questions/Comments | Answers | |
| Attachment_8_-_Questions_and_Answers_10_August_2023, line 90 request that the Volume page count be increased to 50 pages and the answer states Past Performance was revised. The Solicitation Section L did increase the page count from 25 to 30, but this is still under the page count limits outlined on page 55. If a company submits 8 PP citations at a limit of 5 pages each, that would be 40 pages just for the Specific Relevant Contract Reference Sheets. Then we would add 8 pages to cover the Performance Summary Page and Organizational Structure Change History sections. This would outline at least 48 counted pages in this volume. | Page limitations have been adjusted accordingly. | |||
| If an offeror is/was a subcontractor on a referenced past performance, and the Government Contracting Officer responds that they will not complete a PPQ since the offeror was a subcontractor and not the prime contractor, who should complete the PPQ? Can the PPQ POC requirement include the prime contractor as the POC and have this person complete the PPQ since the Government Contracting Officer has refused? The requirement to have the Government POC sign a PPQ for a company that is a subcontractor under the project is unrealistic. The Government would have no contractual obligation to the subcontractor and would therefore refuse to fill out the PPQ. A specific example of a response we have received from a Government POC is : “The CORs cannot complete a past performance questionnaire on a subcontract agency. From the definition of the performance, the prime is responsible for managing the task order execution and all facets. Communication from the Government is directly to the prime, therefore the lack of legal interaction, communication, and transparency with the subcontractors renders it impossible to complete the requested past performance questionnaire.” | ||||
| If the Government is not going to evaluate those PPQs from a prime contractor, and the PP Government POC refuses to provide input, how will the Government evaluate the past performance? If the Government POC is called, after they have refused to answer PPQs, does the evaluation team expect a different answer or enhanced cooperation? By not allowing a prime contractor to evaluate and submit a PPQ, the evaluation team will not be able to conduct a first-hand assessment of the offeror's past performance. The PP, and in particular the PPQ requirements, are extremely limiting to competition – especially for a small business. | Yes, the prime KTR can complete the PPQ with regards to the refenced situation. | |||
| First sentence of section 1.7 states "The Contractor shall develop, implement and maintain an effective Quality Control System which includes a written Quality Control Plan (QCP) provided to the government within 15 days of award". Further down in the paragraph it states "The Contractor shall provide to the Government the QCP with their proposal submittal." Will the USG please clarify whether the QCP is required with the proposal submission or within 15 days ARO? | Within 15 days ARO. PWS language updated accordingly. | |||
| Should the Contigency Plan be included within Volume II of the submission or may it be included as a separate attachment? If it is required as part of Volume II, will it be included in page count? | The language was revised in Amendment 4 to state that the plan will be provided after award if requested. | |||
| With the exception of the Travel CLINs (xxx8), for which the Gov’t has already provided a figure, we request the Gov’t tell offerors specifically which cell (or sum of multiple cells) from the Pricing Matrix should be used for each CLIN on the SF 1449 (0001–40007; again, excepting for the Travel CLINs). As a notional example only: “For SF 1449 CLIN 1002 (OY1 Design), offerors should use the sum of cells A17, B15, and ZZ24 from the Pricing Matrix.” | The Pricing Matrix CLINs are highlighted in green and have been updated. | |||
| Given that this is an 8a award, will the USG consider accepting a minimum of 2 Specific Relevant Contract Reference Sheets for the primary offeror in the Past Performance Volume? | The minimum number has been lowered to three. | |||
| Are the Past Performance citations required for the Specific Relevant Contract Reference Sheets the same as the references to be submitted for Section 1 – Contract Descriptions? | Yes | |||
| The RFP, PWS, and Pricing Matrix do not provide sufficient information for offerors to complete the Pricing Matrix and SF 1449. For instance, there are too many variables involved in the production of any one deliverable or single hour of IMI at any level for offeror’s to price them accurately. Without further information, offerors cannot price any line item in the Pricing Matrix with confidence, and any prices submitted would be based solely on individual offerors’ assumptions, making it impossible for the USG to fairly evaluate and compare offers. Further, this leaves the USG open for post-award protest. | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. | |||
| We request the government definitize the contents of IMI Levels 3 and 4, 2D Models and 3D Animations, 3D Models and 3D Animations, Simulations, Educational Gaming (AR), and Serious Gaming (VR) Re. Pricing Matrix and Factor 4: | ||||
| Issue: The current pricing matrix is un-definitized. Without a more detailed definition of what is contained in the specific hour of Level 4 IMI, or specific 3D model, or simulation, etc., the offeror is either forced to provide a price for very complex content that will be disadvantageous to the government for less complex task orders, or price less complex content that will be financially unsustainable for the offeror. The RFP, PWS and Pricing Matrix include requirements to price Level 3 and 4 IMI containing complex simulation, artificial intelligence controls, complex 3D graphics and animations, gaming environments and virtual reality. RFP B.1.4 states, “The final proposed pricing for the SAIDIQ will become the maximum price allowable (ceiling rate) for each deliverable under the resultant contract when pricing TOs and TO modifications”. The pricing matrix suggest that it is possible to price a single hour of Level 4 IMI that can be used to address the gambit of possible combinations of interactivity addressed in task orders over the next 5 years, including vehicle simulations, 3D models and animations, AI algorithms dynamically assessing student performance, and virtual reality environments potentially spanning the range of rotary wing aircraft simulation to small arms maintenance. The “3D Models and 3D Animations” called out in the pricing matrix as a single unit can cover a very wide range of complexity and corresponding cost. | Simulations, including AR and VR, should be no more complicated to produce than the 3D model definitons per IMI listed in the answer below. These products are made up of the 3D models defined in the answer below, which will drive how complicated they should be. AI dynamically assessing student performance should be about as complicated as, for example, multiples of branching trees that change a learner's path of experiences and expectations based on their performance in the course. | |||
| Requirements under AVLE 1 have included 3D models and animations for everything from avatars in an office setting to modeling every part in a very complex aircraft, to the interior of an armored vehicle. Each of these examples has an extremely wide range of prices. | ||||
| Question: How will pricing developed for 3D models to demonstrate the functionality of all parts in a vehicle transmission be applied to the pricing of 3D models for connecting hoses to a series of water tanks, as well as to avatars in an office setting? | Level 0 & 1 description example: 3D modeling of ink pen - creating virtual 3D Model without break apart capability. |
Level 2 description example: 3D modeling of Animals/Soldiers/satchels/first aid kits - creating virtual 3D Model without break apart capability.
Level 3 description example: 3D modeling of ordnance or small firearms - creating virtual 3D Model with break apart capability and/or animations of effects (i.e. rockets exploding to show blast radius, etc.).
| Alternatively, will the government allow the inclusion of labor categories and rates that will allow these tasks that involve widely disparate levels of effort to be separately priced on individual task orders? | The contract is firm-fixed price, not a level of effort. | ||
| If an offeror was a subcontractor on a referenced past performance, and the government did not interact with or was/is not knowledgeable of the subcontractor’s work performance, can an offeror include the prime contractor POC and have this person complete the PPQ instead of a government person? | The PPQ can be completed by the prime that the subcontractor was working for. | ||
| If an offeror is/was a subcontractor on a referenced past performance, and the Government Contracting Officer responds that they will not complete a PPQ since the offeror was a subcontractor and not the prime contractor, who should complete the PPQ? Can the PPQ POC requirement include the prime contractor as the POC and have this person complete the PPQ since the Government Contracting Officer has refused? | The PPQ can be completed by the prime that the subcontractor was working for. | ||
| Under CLIN 0003 Development, are the total estimated quantity of courseware hours meant to represent the size of the audience/user base (i.e., “e-learner hours”)? For example, is the IMI Package Level 0 Unit Price meant to be one hour of learning material that will be accessed by 2,638 users? | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. | ||
| The Government's answers in the referenced lines in the Q&A document seem to be conflicting. Please clarify the contradiction. | |||
| Will the Government revise the RFP section to allow for the prime contractor to complete the PPQ for the subcontractor's experience. | The PPQ can be completed by the prime that the subcontractor was working for. | ||
| In response to a PPQ request, we have had a Contracting Officer refuse to complete it. Specifically the Contracting Officer stated in her email, "ACC-APG will not be provide any response to this request for PPQ. The Contracts provided are base contracts on "MATOC" neither of these contracts, are Prime contract efforts for "offeror" effort to "offeror". As such, ACC APG will not be providing a response as "offeror" request for PPQ data as "offeror" does not hold a prime contract on "MATOC"..."Our privity and information release is, with the primes and not the subcontractors". | |||
| Based on this response, and a PPQ is not returned in support of an offeror's relevant past performance for the AVLE 2.0 proposal because said offeror was a subcontractor and not the prime, will the corresponding past performance reference still be evaluated? | The PPQ can be completed by the prime that the subcontractor was working for. | ||
| Will past performance references for subcontracted work without corresponding PPQs from a Government POC be evaluated equally as the past performance references that include corresponding PPQs? | The PPQ can be completed by the prime that the subcontractor was working for. | ||
| Throughout Attachment 8, the government annotated a good number of answers that the solicitation documents were “Revised”. However, in many cases, we are unable to see that the provided Amendment 0004 RFP documents reflect those annotated revised updates from the previous Amendment 0003. As a result, we are unable to determine if the solicitation Section L and/or M have been updated. We have used the compare feature of Adobe to further aid our ability to determine changes and do not see very many substantial changes in the amendment documents. Would the government be able to provide a track changes or highlighted amendment to enable us to validate the Q&A revised statements have been made in the Amendment 0004, since that document and not the Q&A will be used to respond in our proposal. | Any revisions to the solicitation will be anotated in the question and answer attachments. We will not be able to include a track changes version. Amendment 6 will have all the revisions and updates, and will allow ample time to adjust accordingly. | ||
| Are offerors required to submit the PPQ to two (2) points of contact for each submitted reference (up to 8 allowed), which means a total of 16 PPQs may be required. | Please follow the Instructions to Offerors section of the solicitation. | ||
| Attachment 3-AVLE 2.0 Pricing Matrix v4.xlsx, there might be an overestimation of courseware hours required for IMI Packages in the development section (rows 46-75). For example, courseware hours to be developed for IMI Level 4 is 10,417 hours or roughly 5.5 years of full-time (based on an 1880 work year) IMI Level 4 for a student to go through. If we were to reference an accepted industry model such as the Chapman Alliance Model for best practice in developing IMI Level 0 through 4, Level 4 IMI would take between 400 and 1,000 man-hours for every hour of courseware development. For example, we could estimate the level of effort for each hour of Level 4 IMI as 500 hours within this model. Assuming an average fully burdened rate of $100/hour, the 10,417 requested on the spreadsheet would total over $.5 BILLION annually. Additionally, in rows 77-91, only 6 hours of 2D, 3D, simulations, and content support are required. This is not commensurate with the requested 10,417 hours of IMI Level 4. |
| If the spreadsheet is filled out with the requested IMI courseware hours indicated in the development section, total contract values would be well into the Billions. Please advise. | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. | |
| The quanity of courseware hours that was previously listed in cells (E46, E52, E58, E64, E70) of the Pricing Matrix has been removed from the IMI Package Level (0-4) pricing section. Will the Government revise the courseware hours for each IMI Package Level to match to the previous version? For your reference, the courseware hours listed in the prior version of the Pricing Matrix are as follows: |
Level 0 = 77 Level 1 = 96 Level 2 = 15 Level 3 = 20
| Level 4 = 10 | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. | ||
| Based on the number of courseware hours listed in Column D, Rows 47-75 of the Pricing Matrix v4 and using the table provided on the Army Distributed Learning Program Contract’s website, which details the range of development hours required to produce a single hour of courseware, the number of labor hours calculated on an annual basis will be in excess of 44,621,000 man hours which equates to approximately 23,240 FTEs. We believe this to be a serious error that if left uncorrected would result in a Total Evaluated price in excess of 15 BILLION dollars. Will the government please revise the number of courseware hours listed in the Pricing Matrix to accurately reflect the program requirements? (To illustrate the problem, please see below for a snapshot of the TADLP based calculation using the quanties from Pricing Matrix v4.) | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. | ||
| Regarding the quantities in CLIN X003, IMI Package Levels 0 – 4: It seems the quantities are not realistic on a per year basis. Perhaps the quantities for the IMI Packages are intended for the entire contract effort. We believe the current quantities are driving the price to a point that is unrealistic and more than the Government is expecting to see. Right now our team is looking at a total proposed value of several billion dollars, based on the quantities depicted in Attachment 3, with most of that contained in IMI Package Levels 3 and 4 in CLIN X003. |
Question 1: Did the Government intend for us to multiply our unit price by the total quantities for every year in the RFP?
| Question 2: Would the Government consider updating the quantities to a number that is realistic on an annual basis? | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. | ||
| Will the Government segregate Test Package pricing by IMI level to reflect the scope and complexity of each IMI level? | No, the Test Package will be priced as each. | ||
| Will the Government confirm these numbers are minutes and not hours and modify the Price Matrix to reflect same? | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. | ||
| Will the Government segregate the Validation Plans and Validation Reports by IMI level to reflect the scope and complexity of each IMI level? | The Validation Plan and Validation Reports will be priced as each. | ||
| Will the Government provide a definition for "Casual Games", "Mid Core Games", or "Interactive Web-Based Simulations"? | Casual games: The development cost of basic games similar to Pong, Pac-Man, and Snake with attractive game features and designs. |
Mid Core games: These games are most valuable as they have different levels, game characters, and a storyline that make them more interesting than any other game.
Interactive web-based simulations: Simulations via the web and immediately running in a web browser, without any prior installation of software.
| In the Government's answer to question on row 8, the Government is expecting 2,638 hours for IMI Level 0. Is this accurate? If so, then 2,638 hours of courseware per year over the five years would 13,188 hours of Level 0 IMI. The previous pricing matrix only had 77 hours. | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. |
| Is the IMI Level Quantity amounts for Level 0, 1, 2, 3, and 4 correct? For example, for level 4 we are to price 10,417 hours of courseware per year. Did the Government expend over 10,000 hours of IMI level 4 on the previous AVLE contract per year? | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. |
| Should the contractor try to maintain the ceiling of $99 million for this effort? The way that the Government has the hours of IMI seems to suggest that we should price the work and not worry about the ceiling. | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. |
| From the previous AVLE contract, can the Government please provide the average number of courseware hours for Level 0, 1, 2, 3, and 4 per year? | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. |
| Can the Government confirm that for IMI Level 1, they expect to have 21,100 hours of courseware developed over 5 years? The previous pricing matrix only had 96 hours. | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. |
| Can the Government confirm that for IMI Level 2, they expect to have 45,720 hours of courseware developed over 5 years? The previous pricing matrix only had 15 hours. | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. |
| Can the Government confirm that for IMI Level 3, they expect to have 48,810 hours of courseware developed over 5 years? The previous pricing matrix only had 20 hours. | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. |
| Can the Government confirm that for IMI Level 4, they expect to have 52,086 hours of courseware developed over 5 years? The previous pricing matrix only had 10 hours. | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. |
| What does Column D (Total Estimated) QTY measure? |
Column C, Unit of Measure, indicates Courseware Hour. Does the Government estimate that they could require the following hours of development per IMI level?
Level 0: 2638 Level 1: 4220 Level 2: 9144 Level 3: 9762 Level 4: 10417 Are the values in Column D (QTY) for each Year, or do they represent the lifetime of the contract? If they are for each year, the Total for all years (Base + 4 OYs) will exceed the NTE amount of $99M. For example (Unit Price used is only a notional rate):
IMI Level 0: Courseware - 2,638 Courseware Hours - $2,500 ea = $6,595,000 (Base Year) IMI Level 0: Checks on Learning - 2,638 Courseware Hours - $2,500 ea = $6,595,000 (Base Year) IMI Level 0: Practical Exercises - 2,638 Courseware Hours - $2,500 ea = $6,595,000 (Base Year) IMI Level 0: Pre Test - 2,638 Courseware Hours - $2,500 ea = $6,595,000 (Base Year) IMI Level 0: Post Test - 2,638 Courseware Hours - $2,500 ea = $6,595,000 (Base Year) Base Year Total for only IMI Level 0 = $32,970,400
| Is this the Government's intent? | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. | ||
| Will the Government change the Qty EA for each IMI Package to reflect a "per one (1) runtime hour of delivery requested", to maintain the consistency with how the GFI Report, IMDP, and Storyboard layouts are configured? | No change, will stay as each and will stay as stated in the PWS. | ||
| Are the quantities identified in Column D to be executed in the Base Year, or do they represent the total quantity over the life of the contract? If only for the Base Year, will the estimated quantities be the same for each Option Period? | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. | ||
| Will the Government please provide the rationale for providing the total estimated quantities in Column D of the Pricing Matrix? |
Is it the Government's intent to develop 25,764 total hours of courseware across the five levels of interactivity?
| If this is the case, should the contractor spread this development across the one (1) Base Year and five (5) Option Years. For example 2638 runtime hours of Level 0 IMI would equate to 437.9 courseware hours each year? | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. | |
| Not all the formulas are correct or accurate. |
By not including accurate formulas in the pricing matrix leaves the pricing calculations open to interpterion and removes the equitable baseline for pricing.
If the Government does not provide the formulas, then how will all offerors be evaluated fair and equitably?
Will the Government correct formulas and provide an updated Pricing Matrix?
Examples of some of the inaccuracies include:
OY4 Extended Beginning with cell N78, the formula is referencing OY3 Unit Price * 3% + OY3 Unit Price.
This incorrect formula appears in cells N78 - N124 Error #1: The formula should be OY4 Unit Price * Qty Error #2: The formula is applying an escalation in the OY4 Extended Price Error #3: The Escalation formulas were removed from all of the OY Unit Price cells.
Invalid formulas Beginning with F-L78 through F-L91, the formulas are referencing the cells.
For example, the formula in cell F78 is D90*E90, and the formula in cell H78 is D90*G78.
Row 132 – Total Cells F132, H132, J132, L132, N132, and O133, are totaling the entire Column for each, rather than the Totals for each CLIN.
| This formula error ends up doubling each PoP, as well as doubling the entire Proposal. | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. Pleaase note: The Government does not warrant the accuracy of any formulas that may be in cells in a Government provided spreadsheet. Therefore, any errors that are caused by erroneous formulas will be considered offeror error. It is advised to review the spreadsheet before submitting the proposal. | |
| The Government's change to the CLIN 0002 – Design IMI Package calculations is misleading and will produce unusable information. |
Using IMI Package Level 0 as an example:
Prior: It was assumed that the Gov. anticipated 77 courses based on an average of 2,638 hours for each course to produce the following deliverables – Courseware, Checks on Learning, Practical Exercises, Pre & Post Tests.
Assuming 2,638 hours would produce one course for a cost of $100,000.00 – The total would be $7,700,000.00 Suppose the 2,638 units of measure are for the number of courseware (seat-time) hours and not labor hours to produce one hour of courseware. In that case, the calculations will be flawed to represent the work necessary to create anything. For example, imagine the effort required to develop 2,638 posttest courseware (seat-time) hours.
Also, if the Government is asking for the hourly cost for each of the five deliverables, it's impossible to provide them with an understanding of what a course might cost since the total number, 2,638, doesn't represent the effort to develop the posttest deliverable in either hours or dollars.
| Currently, – The number of Courses (Total Estimated) QTY has been removed, and the deliverables have been multiplied by what we assume to be the number of hours of 2,638. Therefore, if we estimated a cost of $100,000.00 per course to make the math work using the assumed hours of 2638 (100,000 / 2638 = $37.91), we would need to take the 37.91 and estimate percentages for the five deliverables. | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. | |
| Per amendment 4 Q&A, "The amount 2,638 is based on the total amount predicted for the IMI level 0 courseware, check on learning, practical exercises, pre, post test package." |
Assuming that 2,638 represents "hours" and that 4,220, 9,144, 9,762 and 10,417 are the total amounts in terms of hours predicted for IMI levels 1-4 courseware, respectively, the total amount predicted for the IMI level 0 through 4 courseware for the base year alone appears to be significantly greater than the total number of hours of IMI contracted under AVLE in the last 5 years combined. Please also note that the v4 Pricing Matrix is asking offerors to price a total of 36,181 hours per year for the IMI Package levels, when the total hours of development for all levels for the 5 years of AVLE 1.0 we calculate to be less than 1,750.
To illustrate the cost impact of the total amount (in student learning/seat hours) of courseware predicted for the IMI levels 0-4 courseware, we plugged in a fictitious unit price of $200 for each deliverable in v4 of the Pricing Matrix attached. The result, regardless of the nature and complexity of each IMI Package level or the other associated deliverables, is a five-year total of $191,604,111.38 after all identified formula adjustments have been made to ensure formulas in all cells are consistent . Again, the $200.00 unit price is for illustrative purposes only.
| Would the Government revisit the Estimated Quantities for each of the components within the five (5) IMI Package levels, as the v4 Pricing Matrix does not appear to take into consideration the complexity of the IMI Package levels and permit reasonable pricing on the part of offerors? | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. | |
| For each CLIN in Schedule B, a quantity of one (1) is indicated. In the Pricing Matrix, there are multiple SLINs under each CLIN, with various quantities for each. | ||
| Is it the Government’s intent for us to use the extended price from the Pricing Matrix (unit price x quantities) for the all items under each CLIN for each year (i.e., the value in cell F20 for CLIN 0001). If not, what value does the Government intend for us to put against each CLIN in the SF 1449? | The pricing matrix has been revised to reflect the correct correlation of the courseware IMI Levels and quantities. Additionally, the Pricing Matrix CLINs are highlighted in green and have been updated. |
File details come from the government source that posted it. Updated .