Questions_and_Answers_combined_10182018.pdf

1. Reference: RFP Section C.1, Paragraph C.1.2.1, Page 5, Paragraph C.1.2.2, Page 5, and Paragraph C.1.2.4, Page 6 These sections all reference examination administrations in 2018. Please confirm that administrations will first be held in 2019.

Response: Correct. These sections should reference the exam administration in 2019, not 2018.

2. Reference: RFP Section C.5, Paragraph C.5.1.2, Page 10 and the Test Locations spreadsheet The referenced paragraph indicates that there will be up to 220 examination sites, while the Test Locations spreadsheet only identifies 160 specific locations. Are there known areas where additional testing locations are needed?

Response: There are no specific areas. However, testing is typically around mayor metropolitan areas nationally. A good rule-of-thumb is that people should be able to test within a two or three-hour drive of their home.

3. Reference: RFP Section C.6, Paragraph C.6.1.3, Page 10

This section allows the Contractor to recommend pre-equating of the forms prior to administration. Are the existing operational forms equated already?

Response: Yes, but not pre-equated. They are equated after each administration.

4. Reference: RFP Section C.7, Paragraph C.7.2.1(3), Page 11

The referenced paragraph includes fees to be collected from candidates for the written and oral examinations. If fees for the examinations exceed the stated fees, will the Contractor invoice the AO for the difference? What are the most recently charged fees (candidate-paid and AO-paid) for the exams?

Response: Correct. Vendor will apply fees collected to the invoice. For example, if $50k is collected in fees, and the vendor invoices $110k, the invoice will reflect $110k due, minus an offset of $50k for fees collected for a total of $60k.

The regular test fees in 2019 will be $210 for the written and $300 for the oral.

There will be no charge for the retesting of the 419 candidates who qualify for the FCICE in the spring administration.

5. Reference: RFP Section C.7, Paragraph C.7.2.1(5), Page 12

Please provide details for the AO’s current refund policies.

Response: Refunds have been granted in the past in a tiered fashion. This means that a certain percent is refunded for a cancellation by a certain date, a lower percentage may be refunded by a date closer to the test administration and lastly no refund when sufficient administrative resources have been expended to justify retaining the entire fee. COR would work with a contractor to help make that determination and to include it in the Examinee Handbook.

6. Reference: RFP Section C.7, Paragraph C.7.2.3, Page 12

Will the re-test candidates test at the same time as other testing candidates?

Response: No. The re-test needs to be scheduled as soon as practical after award. A tentative date discussed is February, for rater training and rating in March.

Alternate dates can be considered.

7. Reference: RFP Section C.8, Paragraph C.8.4, Page 12

Would the AO accept a candidate comment form administered electronically as an alternate solution?

Response: Yes.

8. Reference: RFP Section C.9, Paragraph C.9.2, Page 13

Please confirm that passing candidates get their scores in addition to their pass/fail results.

Response: Candidates must receive their equated pass/fail scores, in a score from 1-

100. The cut score for the written is 75 and 80 for the oral, equated.

9. Reference: RFP Section C.10, Paragraph C.10.1, Page 13

Will the AO provide a listing of all candidates eligible to take the oral examination? If so, what format will the information be provided in.

Response: Yes. The information is in Excel and complete, but it is cumbersome to navigate.

10. Reference: RFP Section C.10, Paragraph C.10.5, Page 14

Please describe the statistical data available for form 4.0 of the 2015 oral examination that will be provided to the Contractor for equating purposes.

Response: All statistical data available is in the Technical Reports (AKA – Examination Statistical Analysis Reports/EASR). These reports reflect: N, Mean, St. Dev., Minimum, Maximum, SEM and reliability (alpha). The also include score distributions across exam forms, the dispersion of raw points per exam forms, part-and whole-test inter-correlations, means and standard deviations by test part and overall, descriptive statistics by forms administered (mean, standard deviation, number and reliability/KR21).

11. Reference: RFP Section C.10, Paragraph C.10.7, Page 14

Can candidates request a total of 2 repetitions? Or 2 repetitions per stimulus?

Response: In the consecutive, candidates may request two repetitions for the entire section, not per stimulus. This is similar to the idea of a “life line.”

12. Reference: RFP Section C.11, Paragraph C.11.2, Page 15

Is the 80% correct to achieve certification for the entire exam? Or does the passing candidate need to achieve 80% in each of the sections in order to pass?

Response: In the oral, 80% for the entire exam. In theory a person could fail a section and still pass the exam.

13. Reference: RFP Section C.11, Paragraph C.11.4, Page 16

Would the AO consider an alternate solution that utilized electronic transmittal of score report letters rather than paper?

Response: Yes.

14. Reference: RFP Section C.13, Paragraph C.13.3.1, Page 17 and Section C.16, Page 24

Depending on the required modifications to a form, particularly an oral form, 90 calendar days from date of approval may not be enough time. Would the AO be willing to agree to evaluate schedules on a case by case basis?

Response: Yes. For the OE re-test, we have already chosen a form which needs no changes. Any changes or development would be for the regular administration of the oral exam, which at this point could be June, July or August.

For the WE, no changes are needed in 2019.

15. Reference: RFP Section C.13, Paragraph C.13.4.1, Page 18

Depending on the requested development of a new form, particularly an oral form, 5 months from date of approval may not be enough time to complete development and piloting. Would the AO be willing to agree to evaluate schedules on a case by case basis?

Response: Yes. Please see response 14 as well.

16. Reference: RFP Section C.13, Paragraph C.13.4.2, page 18

For Phase Two oral examination forms, what is the anticipated sample size that statistical equating will be based on? What statistical equating methods have been used in the past?

Response: For both the re-test and the test, we typically test 400-500 individuals.

There is history on the exam administration going back to 2000 with detailed reports that include data for the psychometricians. Although we don’t know the equating methods, proyect psychometricians have called it “classical.”

17. Reference: RFP Section C.13, Paragraph C.13.4.5, page 18 What are the Contractor’s specific responsibilities with regard to field testing? That is, is the Contractor responsible for all aspects from recruiting participants to supplying facilities, or will the AOUSC share in any of these responsibilities?

Response: The vendor is responsible. The COR works closely with the vendor and can provide extensive insight on the administration over the past 10 years.

18. Reference: RFP Section C.13, Paragraph C.13.5.3, Page 19

What is the AO’s definition of a “mix-and-match” written examination form?

Responses: These are examinations that can be created and equated from a pool of questions, whereby candidate A is administered an equivalent, but different form to candidate B. This would depart from the present method using fixed forms, whereby candidates are administered form 1… 8, depending on which forms they have been administered before.

19. Reference: RFP Section C.13, Paragraph C.13.5.4, Page 19

Historically, what percentage of new items have met the criteria for acceptance

Responses: Historically, over the past 15 years, the requirements in the RFP have been met. (76 percent of the 40 or more).

20. Reference: RFP Section C.14, Paragraph C.4.1, Page 20

In the last bullet, the RFP references “mix-and-match” and “computer-adaptive” examination administrations. What is the AO trying to accomplish with these exam administration modalities?

Response: The overarching goal is to eventually be able to move towards testing oral and written every year, in a cost-effective manner, while preserving exam integrity without over exposing test content.

21. If an incumbent exists for the requirement Federal Court Interpreter Certification

Examination Program Support, Solicitation Number: USCA18R0166, please provide the following information?

• Vendor name: Paradigm Testing, LLC

• Contract number: USCA17C1008

• Value: $1,216,500.00

• Date of award: March 28, 2017

• Performance Ended: March 27, 2018

22. Reference: RFP Section Solicitation, Offer, and Award, Paragraph 9, Page 1 and RFP Section L-1, Paragraph L.1, Page 38 The first reference requires “Sealed offers in original and 1 copies” while the 2nd reference requires proposals to be submitted via email. Please clarify.

Response: Proposals are to be submitted via email Linda M. Williams at linda_williams@ao.uscourts.gov no later than 3:00 pm ET on Monday, October 29, 2018.

23. RFP Section L.2., page 39 This section specifies “a maximum thirty-five (35) pages excluding dividers, Table of Content, resumes, title sheets and indices.” RFP Section L.3., page 39 specifies 25 pages for Section I and 10 pages for Section II. Please clarify whether or not Section III (mainly SF 33, acknowledgement of amendments, and Section K) is included in the page limit.

Response: Section III – Price Proposal does not have a page limit. However, this section must contain only the information requested in L.3 Submission of Offers, Section III – Price Proposal, Standard Form 33. Offerors must NOT include any technical or pass performance information in their Price Proposal.

24. RFP Section L.3., page 39 This section requires that “Section II shall not exceed fifteen (10) pages.” Given the total page limit of 35 pages, please confirm that Section II should be limited to 10 (not 15) pages.

Response: Section II is limited to 10 pages.

25. RFP Section L.2., page 39 This section specifies “a maximum thirty-five (35) pages excluding dividers, Table of Content, resumes, title sheets and indices.” RFP Section L.3., page 39, states that the response should include “three sections and the executive summary.” Page 40 includes the executive summary as part of Section I. Please clarify whether or not the executive summary is excluded from the page count.

Response: The executive summary is included in the page count.

26. RFP Section L.3., Section II. Past Performance, page 41 This section identifies a limit of “10 pages (2 pages for each previous contract)” for Section II and requires that offerors submit no more than 5 past contracts. The Past Performance Questionnaire provided totals 4 pages without any content added. Please clarify whether or not the page limit applies to the collective completed questionnaires, or to other past performance information to be provided within the proposal itself.

Response: Section II page limit does not apply to completed questionnaires.

mailto:linda_williams@ao.uscourts.gov

27. RFP Section L.3., Section II. Past Performance, page 41 This section states: “Once completed by the customer agency, Past Performance Questionnaires shall be e-mailed directly to linda_williams@ao.uscourts.gov no later than the proposal due date and time.” Please confirm that directly means by each separate customer agency and that these questionnaires will not be included within the proposal submission with the other sections. If these questionnaires are to be sent separately from the proposal submission, please clarify whether or not Offerors should provide additional information in Section II using the page limit specified (10 pages) to be included within the overall proposal submission.

Response: Questionnaires are not part of the offeror’s proposal submission. The customer agency must email the completed past performance questionnaire directly to linda_williams@ao.uscourts.gov no later than the closing date and time for proposal submission. To be compliant, the offeror must provide additional information in Section II not exceeding the specified page limit.

28. RFP Section L.3., Section III – Price Proposal, page 41 This section states that the price proposal should include its own table of contents. Please confirm whether Section III should be included under separate cover or should be included under the same cover with Sections I and II but have its own unique table of contents.

Response: Section III – Price must be included separately from Sections I and II.

29. RFP Section L.3., Section III – Price Proposal, page 42 This section states that Offerors must submit a cover letter. Is this to be included under Section III specifically, or with the proposal in its entirety?

Response: The cover letter is to be included in Section III specifically.