Read an Evaluation Plan Before Results Arrive
Read an evaluation plan before results arrive to see what a programme will be judged on, which measures matter and what the findings cannot establish.
The common mistake when an evaluation plan appears is to skip straight to the programme description and assume the eventual report will answer the question everyone cares about. That can leave you judging findings against a promise the evaluation never set out to test. Read the plan before results arrive, and you can see what evidence is being gathered, for whom and within what limits.
Find the plan’s purpose
Start with the document’s title, date, commissioning body and stated purpose. Check whether it is a protocol, an evaluation plan, a terms of reference or a brief summary: these documents may describe different levels of detail, and a summary may leave out important design choices.
Look for the programme being evaluated and the decision the evaluation is intended to inform. A review might help officials improve a service, decide whether to continue it, or judge whether it met a public commitment. Those purposes are related, but they call for different evidence.
Mark any boundary around the work: a particular service, location, group of participants or period of delivery. If the plan covers only selected sites, later findings cannot automatically describe every place using the programme.
Write the stated purpose and boundary in a sentence of your own. This gives you a reference point as you read the technical sections.
Separate the questions
Find the evaluation questions, often listed near the aims or set out in a table. Sort them by what they ask: implementation questions concern what was delivered; outcome questions concern what changed; value-for-money questions concern costs in relation to results; and causal questions ask whether the programme contributed to a change.
Do not treat these as interchangeable. A plan may ask whether a service reached its intended users without asking whether it improved their situation. It may track a change without being designed to show that the programme caused it.
For each question, note the evidence the plan proposes to use and the point in the evaluation when it will be gathered. An interview with staff may illuminate how delivery works, while administrative records may describe service use. Neither source answers every kind of question.
Say you are reading a fictional plan for a council advice service. It asks whether residents can access appointments and whether they later resolve a benefits problem; you would note that access and resolution are separate questions, requiring different evidence and potentially different interpretations.
Trace each measure
Next, follow each intended measure from the question it is meant to answer. A useful plan explains what will be counted or assessed, where the information comes from, who is included and when the measure will be taken. If those details are absent, record the gap rather than guessing what the eventual report will contain.
Distinguish an activity measure from an outcome measure. Appointments offered, forms processed or referrals made describe activity. A change in access, knowledge or circumstances may be an outcome, depending on how the plan defines and assesses it.
Check how key terms are defined. “Successful contact”, for instance, could mean an appointment was booked, a conversation took place or a person received the help they needed. The plan’s definition determines what its measure can support.
- Measure: What exactly is recorded or judged?
- Source: Is the evidence from service records, a survey, interviews or another source?
- Population: Which people or cases are included, and who may be missing?
- Timing: Is the measure taken at delivery, later on or at more than one point?
- Comparison: Is there a baseline, a comparison group or no planned comparison?
These checks help separate a clear measure from a broad aspiration. If a plan promises to assess “lasting improvement” but names no follow-up evidence, note that the intended outcome may not be observable through the proposed measures.
Read the comparison carefully
If the plan includes a comparison, find out what is being compared and how the comparison is formed. It might compare outcomes before and after a service begins, or compare people offered a service with a group that did not receive it. Each approach has limits: circumstances can change over time, and groups may differ before the programme starts.
Look for the plan’s explanation of how it will account for those differences. A comparison can strengthen an assessment, but its presence alone does not settle whether an observed change was caused by the programme. Check whether the design, available data and evaluation question fit one another.
When there is no comparison, the evaluation may still describe delivery, participant experience or change over time. Read the stated aims to see whether they promise a description or a causal judgement; do not assume the latter from outcome measures alone.
Check the limits
Find the section on limitations, risks or data quality. Look for missing records, low response, changes to the programme during evaluation, differences between sites, or groups who may be hard to reach. These are not minor footnotes: they shape how far a finding can be applied.
Notice whose perspective appears in the proposed evidence. Staff accounts, participant feedback and administrative data each make some aspects easier to see than others. A plan that relies on service records may say little about people who never reached the service.
Also check whether the evaluation can distinguish the programme from other support or policy changes happening at the same time. If it cannot, the report may be able to describe patterns and experiences without attributing them confidently to one intervention.
Questions to settle first?
What can an evaluation plan tell me?
It can show what the evaluation intends to ask, which evidence it plans to collect and what conclusions its design may support. It cannot tell you the findings before the evidence has been gathered and analysed.
Does an outcome measure prove impact?
No. A measure can show that an outcome was recorded or changed, but causal claims depend on the evaluation design and the strength of its comparison and evidence.
What if the plan is vague?
Mark the unspecified question, measure or population, then check for a fuller protocol, appendix or data plan. If none is available, treat that point as an unresolved limit rather than filling it in yourself.
Can the evaluation apply to other places?
That depends on who and where the evaluation covers, how delivery varies and whether the report explains those differences. Findings from selected settings may offer useful lessons without representing every setting.
Keep a reading record
Before closing the document, make a compact record with five entries: purpose, boundary, evaluation questions, measures and stated limits. Add a note beside each question describing what evidence would answer it and what would remain uncertain.
When the findings arrive, compare the report with this record. Check whether the promised measures were collected, whether the evaluation questions were answered, and whether the report respects the limits already visible in the plan.
Reading an evaluation plan before results arrive is most useful when you keep its promises distinct from the programme’s broader ambitions. That small distinction lets you ask a sharper question of the final report: not whether it says everything worth knowing, but whether it answers the questions its design could reasonably address.
