2017_Merit_SOW.docx

Computer Administration of Merit Promotion Exam Statement of Work

A. Background:

The United States Marshals Service (USMS) consists of about 300 offices throughout the country and overseas. The primary mission of the agency is to support the Federal judiciary. Our employees assist the Department of Justice in virtually all Federal law enforcement activities. Further information about the USMS can be found on the Internet at www.usdoj.gov/marshals.

Criminal investigators (GS-1811) in the USMS compete within the agency for promotion to a grade 13, 14 or 15 position. Grade 13 criminal investigators are typically first-line supervisors or senior specialists in areas such as court security or enforcement. Grade 14 criminal investigators primarily serve as Chief Deputies in charge of our small to medium-sized offices, as well as Assistant Chief Deputies in our larger offices. Operational positions at this grade level also serve as Team Leaders in our Headquarters offices, and they manage the national programs of the U.S. Marshals Service. Grade 15 criminal investigators serve as Chief Deputies in our larger district offices or as Headquarters Chiefs/Staff Officers. Even though these positions have various duties and responsibilities, we have traditionally found that the job functions and required skills are consistent across positions, with a slight variation between grade 13 positions and grade 14 or 15 positions.

B. Description:

This USMS requirement is for a contractor to administer the writing skills (45 items) and job-knowledge (150 items-3 parts) exam to USMS employees at various locations throughout the United States and its protectorates, primarily via a computer-based system. Based on USMS policy directive 3.1.F.5.a.1), “Employees applying for promotion must …have a current Merit Promotion Exam score on file in accordance with established timeframes”. The USMS will provide the vendor with the completed exam. It is expected that the vendor shall perform the following requirements:

1. Transfer the provided exam items to a computerized system.

2. Register the test-takers.

3. Securely administer the 195-item multiple choice exam at sites specified in Appendix I on mutually agreed upon date(s), primarily on Saturdays.

4. Securely administer make-up exams (typically 2-3 months after original exam date).

5. Provide the USMS test-takers with the initial final exam scores and post-appeals modified scores.

6. Provide the Contracting Officer’s Technical Representative with the initial exam scores, a classical item analysis, and post-appeals exam scores.

The contractor shall administer the examination via a computer system that is secure, proven reliable, and provides constant back-up. For example, if the system were to crash in the middle of an exam, only exam data on the current question would be lost. The contractor shall administer the exam via a secure computer network, rather than on-line or through a web-based system. This method will help ensure a more secure exam, and less variation in screen load time delays for each test taker. In addition, the contractor shall administer and score a very limited number of paper-and-pencil exams in remote locations where a computerized testing center is not available.

The USMS will update its job analysis and test plan for the 2017 exam, and subject-matter experts will have completed writing the exam by December 31, 2016. It is expected that the exam will be provided to the vendor no less than two months prior to the primary exam dates.

Based upon the fact that the exam is designed to test USMS deputies, the contractor and any sub-contractor(s) must be aware that test takers may show up at the testing facility with concealed weapons.

C. Examination Procedures:

Exam Locations: The exam is to be offered at over 200 locations throughout the United States and its protectorates, along with a very limited number of foreign locations. The vendor shall provide a computerized testing site in at least 90% of the locations listed in Appendix I, or within 100 miles of these locations. The vendor will make testing arrangements for paper and pencil exams at those test sites where the vendor cannot provide a secure computer testing center.

See Appendix I for a listing of exam sites that must be matched at the rate of at least 90%. Please refer to Appendix I for more detailed instructions on how to respond to this part of Tasks G & H. It is mandatory that the contractor be able to match at least 90% of the locations listed under the attached Appendix I.

Exam Sites: Exam sites should be clean and provide one chair and computer workstation (with a computer mouse) for each test taker. Ample parking should be available for free or at a minimal fee. Exam takers may need to store a cell phone during the exam administration, so locked and secure storage must be available. In so much as possible, exam sites should provide a similar experience for all candidates, regardless of geographic location.

IT Systems: The exam must be stored and administered through a network (not Wi-Fi) secured and controlled by the vendor in order to ensure exam security and a consistent test-taker experience.

Exam entrance and Security: A) Test takers are required to show at least one government-issued picture ID upon entrance to the facility. B) Test takers must provide their signature on a registration form to verify that they arrived at the exam facility, or the vendor may provide some other type of proof to verify the names of all of the test takers who signed in to take the exam on that date and time. All registration documents shall be provided to the USMS COR. C) Proctoring: The contractor shall provide no less than one proctor per exam site. At least one proctor must monitor the examination room and observe test takers throughout the exam session. The proctor will allow for up to a 30-minute late entry to the exam. Proctor’s must check to make sure test-takers do not enter the exam room with contraband material such as cell phones, calculators, written notes, etc.

D. Testing Preparation and Administration:

Exams shall be administered by the contractor, to participants, on a primary Saturday exam date, with some secondary exam date flexibility allowed for high-demand testing sites. Based upon the dates provided in this solicitation, it is anticipated the test will be administered in early March 2017. The USMS preferred dates are March 25 and April 1, 2017. If these dates are not available, please propose alternate dates. There are approximately 3700 employees eligible to register for the exam, and historically, about 80% initially register for the exam and between 70% - 75% actually take the exam.

Test items should be randomized, so that the order of questions differs on each exam. This randomization should occur only within each of the three job knowledge sections (i.e., Part A, Part B, and Part C).

Test takers should be allowed to appeal questions by entering a free-form response (e.g., “This question is not accurate because none of the possible responses are correct” or “The policy which is referred to in this question changed last month, and the responses are no longer correct.”). A comment box format may be used for this purpose, as long as it is labeled with the term “appeal”. Ideally, this appeal item would be offered with each question or as a separate pull-down menu option, but may also be offered as an option at the end of the exam. The vendor will provide an electronic or computer copy of the proposed appeals format to the USMS COR for review with the pilot exam.

The total exam time will be for 4 hours. The writing exam section will be offered first, and each of the four sections WILL NOT be timed separately. The test taker should be free to go back and change answers to questions that have already been completed and move from one section of the exam to another. They are not timed separately. This is important, since a few of the test takers may be taking a paper and pencil version of the test, and will have that freedom. In addition, if an appeals option is not provided with each question, the format will allow the test takers to skip forward to the appeals section while an issue is fresh in their minds, and then go back to the appropriate test question to finish the exam.

Exam Results: The contractor shall provide test takers with exam scores, preferably at the test site, but otherwise within one week of the testing date. The contractor shall provide a statement of test scores to the test taker in a format provided by the USMS. Example: a key shall be provided to explain each part of the exam, and how they are added together (e.g., Part A + Part C = score for GS-13 exam). The contractor shall provide the test taker with a unique registration number or other unique identifier other than his or her social security number. That number must be printed on the exam results or otherwise provided to the test taker. This number will be provided to the USMS as a unique identifier along with the computerized database of exam results.

E. Proposed Tasks:

Task A Delivery Date: To be completed by vendor within 3 weeks after contract award.

Post-Award Conference. Within no more than three (2) weeks after contract award, a post award conference is to be held at USMS Headquarters. At the conference, the contractor is to provide the following:

1. The USMS COR will schedule and coordinate the post-award conference, which will be held in the Arlington, VA area.

2. The USMS and the vendor will discuss possible exam dates and determine an exam date in the spring of 2017 that provides the largest number of seats in the testing locations needed by the USMS.

3. The vendor will demonstrate 5-10 sample USMS test questions in the computerized testing system.

4. Within 2 weeks of the post-award conference, the vendor will provide

a. phone number (mandatory) or a web site (optional) that test takers may contact to register for the exam, and

b. a list of exam sites and times for distribution to test takers as reference during test registration. (The list will be provided in in Word or Excel format.)

Task B Target Delivery Date: To be completed by vendor no later than 2 weeks of the post-award conference (Task A).

The vendor shall establish an online practice site so that USMS test takers can familiarize themselves with the contractor’s test taking methodology, prior to the exam. This site does not have to be USMS-specific, but must be in a similar multiple-choice format. The proper web address shall be provided to the USMS COR.

The 195-item multiple choice exam will be transferred to the contractor no later than mid-January 2017.

Task D Target Delivery Date: To be completed by the vendor no later than 21 days prior to administration of the exam.

Transfer the 195-item multiple-choice exam from a word processing document to a computerized system. Each test item is in multiple-choice format with 4 or 5 possible responses. Most items and possible responses are only a few lines each, but some of the writing skills questions may be rather lengthy, and include several 4 - 5 sentence paragraphs as prompts.

Task E Target Delivery Date: Registration for exam to be provided by vendor at least 60 days, but preferably 90 days, prior to primary exam administration date (limited flexibility, depending on requirements of vendor).

The vendor will provide the USMS with a list of computerized testing locations or a web link to this list, so that USMS employees can view options prior to registration.

The USMS will provide the vendor with a list of eligible employees to include first name, middle initial, last name, and last 5 digits of social security number. Upon registration, employees will provide contact information required by the vendor, such as addresses, phone numbers, etc.

At least 60 days prior to the exam (90 days is preferable) up to the actual exam administration, the vendor will register eligible Marshals Service employees for the exam. Upon registration, the test taker will be provided with an exam date, location and time, which will be confirmed in writing via mail or email. At this time, a unique confirmation number will be assigned by the vendor, which can then be used as an individual identifier throughout the testing and score reporting process. The vendor will contact the test taker if there are any changes in the exam date, location or time. Be aware that test takers may call to change an exam site or time, based on operational assignments or mission needs. As a result, there is no exam registration cut-off or deadline to change test sites. If a test-taker shows up on the exam date and space is available, the vendor will register the test-taker on-site and allow him or her to take the exam.

Note: Employees will be informed by the USMS that prime locations and times will be more available to those who register early in the registration window. At any given time, a very small percentage of the USMS workforce may be called upon at the last minute to perform a special operation or detail away from their normal duty site. Accommodations should be made when space is available at the testing site. This may include allowing a non-registered test taker to register on-site and take the exam if seats are still available. If a test seat is not available to accommodate a change or late registration, a make-up exam may be approved by the USMS, depending on the circumstances.

Task F Target Delivery Date: The vendor will provide a pilot exam registration and test administration process at a limited number of sites 2 – 4 weeks prior to the primary exam date.

The vendor will allow 10 - 15 pilot test takers to pilot the exam process at a limited number of locations (approximately 3-5 sites). The pilot will cover the entire test administration process, from the registration phase to the individual score report received after the session. The vendor will also report the scores to the USMS as a group to ensure computer system compatibility and the adequacy of the score reports and the item appeal process. The vendor will make critical adjustments as needed to ensure a successful exam delivery. Pilot test takers will not take the exam over again on the exam date, due to security concerns. The above is to be coordinated by the vendor with the COR.

Task G Target Delivery Date: The target delivery dates are March 25 and April 1 2017 (there is limited flexibility for USMS exam date, depending on exam seat availability).

Exam Administration: Administer the computerized exam (195 multiple-choice questions with a 4-hour time limit) to USMS employees on up to two consecutive Saturdays, with seats at test sites primarily dedicated to the USMS.

The total exam time will be for 4 hours. Candidates will be able to determine time remaining via a constant timer on the computer screen, a timer pulled up on demand on the computer screen, or at minimum via a clock in plain sight in the testing facility. If a timer on the screen is available, the test takers will also receive a pop-up timer warning when 10 minutes remain in the testing window. Otherwise a 10-minute verbal warning will be provided. Each of the four sections WILL NOT be timed separately. In addition, if an appeals option is not provided with each question, the test taker will be allowed to skip forward to the appeals section while an issue is fresh in their minds, and then go back to the appropriate test question to finish the exam. Candidates will also be allowed to navigate back and forth to review items or to skip items and then come back later to answer and/or change answers.

Exam Date and Times: To be jointly determined by vendor and USMS to accommodate the 4-hour testing window. The goal is to administer all exams on up to two consecutive Saturdays, but there is limited flexibility to allow for sites with high demand and low supply.

Exam Results: The computer exam results, to include several different total scores, will be provided to the test taker upon exiting the facility in a format provided by the USMS. Otherwise, scores will be mailed to the test taker within one week of exam administration.

Exam Locations: The exam locations for the computerized version of the exam will vary based on the sites offered by the vendor, but must meet the 90% requirement described in Appendix I.

Exam Sites: Computer exam sites should be clean and provide one chair and desk/work station for each test taker. The atmosphere should be conducive to test taking (e.g., no disruptively loud noises). The work site configuration should be set up so that candidates cannot easily see each other’s screens. Ample parking should be available for free or a minimal fee. There should also be a secure locker available to store cell phones, etc.

Customer Satisfaction: At the end of the exam administration, an optional survey of customer satisfaction, to be provided by the USMS and consisting of up to 10 items, will be available to the test taker. Responses from this survey will be provided to the USMS, to include unique customer id, test site and test date.

Task H Target Delivery Date: The vendor will administer a paper and pencil version of the exam on one Saturday in March, to correspond with the computerized exam delivery dates (limited flexibility for USMS exam date, depending on exam seat availability).

Exam Administration: Administer a proctored paper & pencil version of the computerized exam (195 multiple-choice questions with a 4-hour time limit) to USMS employees on a single Saturday. The contractor shall provide USMS-approved instruction booklets for proctors, as well as answer sheets and a method for recording appeals. The vendor will deliver a paper and pencil version of the exam within 100 miles of the USMS office location, to candidates in remote locations or in a very limited number of locations where automated testing site capacity does not meet USMS demand for seats.

The total exam time will be for 4 hours. Candidates will be able to determine time remaining via a clock in plain sight in the testing facility, and will receive a verbal reminder with 10 minutes remaining in the test window. Each of the four sections WILL NOT be timed separately. In addition, if an appeals option is not provided with each question, the test taker will be allowed to skip forward to the appeals section while an issue is fresh in their minds, and then go back to the appropriate test question to finish the exam. Candidates will also be allowed to navigate back and forth to review items or to skip items and then come back later to answer and/or change answers.

Exam Times: To be jointly determined by vendor and USMS to accommodate the 4-hour testing window.

Exam Results: Proctors will mail exam booklets, answer sheets, and appeal sheets to the vendor using a traceable/accountable delivery method at the vendor expense. The vendor will grade the paper and pencil exams and provide final exam results to employees in same manner as the computerized final exam results.

Exam Locations: The exam locations for the paper and pencil version are still to be determined and will vary based on the automated sites offered by the vendor. Historically, there are approximately 8-10 paper and pencil sites in remote locations, such as Fairbanks, Alaska; Yuma, Arizona; Key West, Florida; Great Falls, Montana; Missoula, Montana; Pierre, South Dakota; Alpine, Del Rio, and Laredo, Texas. Most of these sites will have less than 10 test takers, while the Yuma site may have slightly more, and the Texas sites may have up to 40 each.

Exam Sites: The contractor shall provide the locations with sufficient paper and pencil for testing. Paper and pencil exam sites should be clean and provide one chair and desk for each test taker. (If desks are not available, sufficient space should be allowed around a table so that candidates cannot easily see each other’s answer sheet (e.g. one empty chair between each candidate.) Ample parking should be available for free or a minimal fee. There should also be a secure locker available to store cell phones, etc.

Customer Satisfaction: At the end of the exam administration, an optional survey of customer satisfaction, to be provided by the USMS and consisting of up to 10 items, will be available to the test taker. Responses from this survey will be provided to the USMS in electronic format, to include unique customer ID, test site and test date.

Exam entrance and Security: A) Test takers are required to show at least one government-issued picture ID upon entrance to the facility. B) Test takers must provide their printed name and signature on a piece of paper to verify that they arrived at the exam facility, or the vendor may provide some other type of proof to verify the names of all of the test takers who signed in to take the exam on that date and time.

C) Proctors must check to make sure test-takers do not enter the exam room with contraband material such as cell phones, calculators, written notes, etc.

Proctoring: It is expected that there will be no less than one proctor per exam site. At least one proctor must monitor the examination room and observe test takers throughout the entire exam session. The proctor will allow for up to a 30-minute late entry to the exam.

Task J Target Delivery Date – offered by vendor, make-up exam timeframe is typically from 2 weeks to 3 months after primary exam date.

The vendor will provide automated make-up exams for eligible employees designated by the USMS (typically 150-200 make-up exams).

Due to travel schedules, special assignments, illnesses, etc., some USMS employees will not be able to take the exam at the designated date or time. Within 2 weeks of the primary test date, the USMS will provide the vendor with a list of names of those eligible for a make-up session and the window of testing opportunity (generally 3 months after the initial exam date.) A full list of names and make-up locations will not be finalized until 2 weeks after the actual nationwide exam. Make-up sessions may occur at multiple sites, depending on the locations of the employees who need to take the exam. In general, make-up sessions can be offered on the vendor’s testing schedule, during regular testing hours and as seats are available at a particular testing site.

Note: Since the make-up window extends beyond the requirements for the provision of a final electronic file of scores (see Task L), the vendor will need to provide the USMS with a second final score report for the remainder of the make-up exams. This report needs to only include those not provided on the first report, and should be received 2 weeks after the last scheduled vendor- administered make-up exam.

Task K Target Delivery Date – within 2 weeks of last/second primary exam date or no later than April 17, 2017.

Provide, within two weeks of test completion, the preliminary exam results (to include survey responses) and a classical item analysis for the 195 items, using the exam results from the original test takers (makeup exam scores do not need to be included in this analysis). The preliminary results will show the number of test takers and the score distributions and averages for each section (e.g., Part A, Part B, Part C, and writing skills). At a minimum, the item analysis should show the number and percent of valid responses, a score distribution for each question, point bi-serials, and also split the test takers into a high group (those who scored in the top 50%) and a low group (those who scored in the bottom 50%), and show the number and percent of responses for each of the four choices. For example, the percent responses could be shown as:

Part B, Question 1 A B C D % of hi 75% 10% 5% 10% % of low 35% 50% 10% 5%

This data will be used during the appeals process, to help determine if a question should be thrown out or have more than one correct response. The item analysis data will also be used when writing the next exam, to determine if a question is worth using over in a new exam.

In addition, the vendor will provide all appeals in an electronic spreadsheet format, to include test taker unique ID, exam section (i.e., writing, Part A, Part B or Part C), item number, correct answer, answer provided by test taker, and the text of the written appeal.

Task L Target Delivery Date(s) - Rescoring – no later than May 5, 2017.

· Delivery of new scores (final) to USMS (electronic and hard copy)

· Mailing new scores (final) to test takers

As stated in tasks F and K, test takers will be allowed to appeal individual exam items. An electronic or computer copy of this appeals information, with the question number identified, will be provided by the vendor to the USMS within 2 weeks of the last primary exam date or no later than April 17, as stated in task F. In April, the USMS will convene a group of subject matter experts to review each appeal. Over the past 25 years, appeals have always been received. Typically, the appeals for approximately 5 questions are upheld. In a very few cases, appeals have been upheld for up to 10 questions.

Once the appeals meeting has taken place, the USMS will update the scoring key and send it to the vendor to re-score all exams, including completed make-up exams. For example, Part B, item 31 may now have both option A and option C scored as correct, or for Part B, item 35, all responses may be scored as correct. Based on the updated scoring key, the vendor will rescore all tests (i.e., original exams and makeups) that have been received to that point, and email the new exam results to the test takers at the email or mailing address designated by the test taker during registration. This format of results will be USMS specific, and include information such as the unique ID of the exam taker, average exam scores and ranges.

In addition to providing the updated scores to the individuals, the vendor will provide the USMS with an electronic result files. Data downloads are typically not an option due to firewall issues. The results file will contain the following information: Test taker name, unique identifier number, exam date, exam site, and score for Part A, Part B, Part C, and writing skills. This database should also contain the test-taker responses for each item in the exam. This information can be split from the overall score information or can be combined into one database. The format for the computer file is flexible, but must be able to be read into Microsoft Excel. Acceptable formats are Excel, tab-delimited data, Rich Text format, etc.

(The rescoring and provision of final scores will need to occur a second time, for the small number of test takers that complete make-up exams through the end of the make-up exam window.)

Task M Target Delivery Date(s) - Billing – The vendor will provide the USMS with an itemized bill within 4 weeks of the primary exam date, and for make-up exams, within 4 weeks of the closure of the window for make-up exams. (If the vendor prefers, make-up exams can also be billed on a monthly basis throughout the make-up exam window.)

The vendor will provide the USMS with an itemized bill that includes the name, unique registration id, registration date, exam site, and exam test date for each individual test taker.

Due to the variability of mission requirements, test takers will be able to cancel their registration and/or change test sites up to 48 hours prior to the exam start time without an additional charge or test cancellation fee. After the 48-hour window, the vendor will be able to charge the USMS for no-shows and/or late site changes.

N. Special Notes:

All test results are the property of the USMS. Once the make-up exams are completed, and the computer files are received and approved by the USMS, the vendor should destroy all exam materials and test-taker records and provide written certification of such. Test-taker records cannot be sold or shared with other vendors for statistical purposes, or to create or add to mailing lists.

All above listed dates are tentative. If the vendor cannot meet the proposed dates, it is requested that alternative dates be proposed. Please provide any alternative dates with your proposal submission.

Appendix I

The selected contractor must be able to match at least 90% of the USMS exam sites on the attached spreadsheet, by being within 100 miles of those locations. The Excel spreadsheet contains a column for zip code, state/country, and city or location of the USMS exam sites, along with the estimated maximum number of employees who work at or near that site.

Proposed Tasks: For every USMS exam site, the vendor must compare the USMS exam sites and number of seats needed with a list of their exam site locations (including an address) available for Saturday testing, and the number of seats available for Session I and Session II on the primary test date(s) for each location. It is not acceptable for a vendor to simply state that they can meet the site requirements. The vendor must provide the address and zip code of each computerized testing location in comparison to each USMS exam site listed on the Excel spreadsheets and approximate distance from the USMS exam site. Without addresses, zip codes and approximate distances, the vendor will not be considered as meeting the requirements.

For example, the response should include:

mandatory exam site from USMS spreadsheet vendor exam site w/address and zip (If not available on Saturday, please note)

Approx. distance from USMS exam site (miles)

# of seats needed based on excel

Session I # of seats available

Session II # of seats available

For each match to the USMS testing sites where a sufficient number of seats does not appear to be available for single date, two-session testing, or where Saturday testing is not available, the vendor must list alternatives, such as “employees located at zip code 55555 can take the exam on the Friday evening prior to the testing date,” or “if registration at site XXX exceeds 50, a third testing session will be added on Friday night,” or “vendor will rent space and computer systems at site YYY to accommodate unusual number of test takers,” or “vendor will open site for Saturday testing at a fee of $$$ if less than xxx employees take exam at that site.”

In addition, please provide a separate, detailed plan for each of the following three high USMS population/low testing center capacity regions. (For example, the requirements described above are not met if the vendor lists a single test site with 20 seats and claims to meet requirements by mapping this single testing site with 20 seats as the solution for five different locations with a USMS population of 20 each for a total of 100 required seats.) These three areas are:

· Washington, DC Metropolitan area to include all MD, DC and Northern, VA locations (except for Norfolk, Roanoke, Lynchburg and Abingdon in Virginia). The USMS population in this area approaches 500 – how will this requirement be met?

· Southeast Georgia to include Atlanta, Newnan, Valdosta, Macon, Athens, Brunswick, Waycross, Glynco, Savannah and Dublin, Georgia and Jacksonville, FL. The normal population for these offices is 133, but the population may swell by at least another 50 if there is a large class taking place at the USMS Training Division in Glynco, GA. How will this requirement be met?

· SW Border districts in Arizona (Yuma and Tucson), Southern California (San Diego, Los Angeles and El Centro), Nevada (Reno), New Mexico (Las Cruces) and Texas (Alpine, Brownsville, Corpus Christi, Del Rio, El Paso, Laredo, McAllen, Pecos and San Antonio). The Southwest Border offices are geographically dispersed and often employee a large population of USMS employees away from major metropolitan areas. How/where will these 500+ employees be tested, especially in areas such as Yuma, Del Rio and Laredo?