OSEP HiringThird-PartyEvaluator.pdf

Hiring a Third-Party Evaluator1

SHOULD I HIRE A THIRD-PARTY EVALUATOR?

As you think about evaluating your project, you may want to consider working with a third-party evaluator if it’s not a requirement.

These qualified professionals have training and experience in designing and conducting evaluations and can conduct any or all steps of the evaluation for you. Deciding whether to hire a third-party evaluator may not be easy, as there are several issues to consider. These include:

Benefits. Third-party evaluators can:

• Bring technical expertise in research methodology, statistics, or related topics to the project

• Provide credibility and objectivity

• Allow the project staff to focus on implementation

Limitations. Third-party evaluators may:

• Add cost to the project

• Add to the project management tasks (i.e., managing the third-party evaluator)

• Not know the project background or content area as well as project staff

• Be less available than project staff

Overall, the more complex your evaluation is, the more helpful a third-party evaluator is likely to be. For example, if your evaluation might use complicated sampling or statistical procedures, the expertise of a third-party evaluator is likely to be helpful or even necessary.

I HAVE DECIDED TO HIRE A THIRD-PARTY EVALUATOR. WHAT SHOULD I CONSIDER?

Hiring early. You will want to hire a third-party evaluator as early as possible during the application preparation or planning stage. This early start will give the evaluator time to help design your evaluation, develop a detailed evaluation plan, and conduct part or all of the evaluation activities. If you hire a third-party evaluator later in the project, you may not be able to take full advantage of the evaluator’s expertise or time. Also, it may be too late to implement certain data collection activities or to correct earlier mistakes.

Conducting a needs assessment. An evaluation needs assessment can help you identify the specific tasks the evaluation will require, including those that will be contracted to a third-party evaluator.

Allocating Resources. The evaluation needs assessment can be used to identify approximately how much money you’ll want to allocate for your third-party evaluator. If necessary, the cost of different types of evaluations can be roughly estimated as a percentage of the project budget: low cost = <10%;

moderate cost = 10-20%; high cost = >20%.

Devoting staff time. It’s important to recognize that hiring a third-party evaluator doesn’t eliminate the need for project staff time on the evaluation. At a minimum, project staff will need to communicate with the third-party evaluator and monitor the evaluation. Remember that ultimate responsibility for the evaluation still rests with the Project Director, not the evaluator.

1. Adapted from: Heinemeier, S., D’Agostino, A., Lammert, J., & Fiore, T.A. (2014). Guidelines for Working with Third-Party Evaluators. Rockville, MD: Westat.

Below we have outlined steps to hiring and successfully working with a third-party evaluator. For more information, see the CIPP resource Guidelines for Working with Third-Party Evaluators, available on the OSEP IDEAs That Work website.

HOW DO I FIND AND WORK WITH A THIRD-PARTY EVALUATOR?

Develop a Scope of Work. The third-party evaluator’s scope of work contains the same types of information as the overall project evaluation’s scope of work, but focuses on the third-party evaluator’s contributions. If you conducted an evaluation needs assessment use it to help define the scope of work. The scope of work for the third-party evaluator should include specific evaluation tasks, expectations for meetings and communication, and schedules for submission of draft and final products. Importantly, it should also contain performance management expectations and milestones, to help with effective oversight of the evaluation.

Develop an Evaluation Budget. The evaluation budget identifies the resources (e.g., personnel, instruments, incentives, travel) needed to complete the evaluation, and it’s usually part of the overall project budget. It won’t be possible to finalize the budget before the evaluator is hired, since the final budget will be informed by the exact tasks the evaluator will do. However, it will be important to have a sense of how much money is available for the evaluation before the evaluator is hired. For more information see the CIPP brief Budgeting for Evaluation: Key factors to consider, available on the OSEP IDEAs That Work website.

Search for Qualified Evaluators. Ask colleagues to recommend an evaluator or talk with staff from other projects about their evaluators. Search websites for projects with similar evaluation needs, and identify their evaluators.

You can also connect with evaluators at local and national meetings; monitor listservs; and search websites, professional publications, and newsletters. Once you have compiled a list of potential evaluators, carefully consider their qualifications and experience as they relate to your specific evaluation, and contact evaluator-supplied references.

Prepare a Contract. Once you have selected an evaluator, prepare a written contract specifying the evaluator’s roles and responsibilities. The contract is a legally binding document that details the evaluator’s activities, the amount of time to complete the evaluation, and the cost for services. The contract should include timelines for the evaluator to deliver interim reports and other work products at regular intervals.

Ensure That Your Evaluation Plan Aligns with Your Proposal. As part of your contract, be sure to identify who will write the evaluation plan component of your proposal:

project staff or your newly hired third-party evaluator. No matter who takes the lead on writing this section of the proposal, it is extremely important that project staff carefully review the section to ensure that the evaluation plan aligns with the rest of the proposal and the project work that is being proposed. For example, proposed project activities and outcomes in your logic model should line up with evaluation questions and data collection activities in the evaluation plan.

Establish a Strong Working Relationship. Throughout the evaluation, it’s important to monitor and manage the third-party evaluator’s work. Plan to communicate regularly.

While it is important for the third-party evaluator to maintain objectivity, the grantee and evaluator can develop a strong working relationship that aids completion of evaluation tasks.

Strategies for establishing a strong working relationship include setting reasonable goals and expectations, establishing regular communication, and defining decision-making roles and responsibilities.

The Center to Improve Program and Project Performance has been funded with Federal funds from the U.S. Department of Education, Office of Special Education Programs, under contract number ED-ESE15-A-0016/0004. The Project Officer is Dr. Kristen Rhoads. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.

https://osepideasthatwork.org/sites/default/files/Guidelines_3rdEvaluators_508_0.pdf https://osepideasthatwork.org/sites/default/files/CIPP2_Budgeting_for_Evaluation_Brief_2-13-15.pdf https://osepideasthatwork.org/sites/default/files/CIPP2_Budgeting_for_Evaluation_Brief_2-13-15.pdf