Evaluation Design

Type of Evaluation

Evaluation design means the type of evaluation that you choose to do, and also the timing of when you take measurements, e.g. before and after the road safety project takes place. It does not refer to discussions of data collection methods such as surveys or interviews. For more information on methods please see the help page on Ways to Collect Data.

The first question you should ask yourself when thinking about what evaluation design to use is: What is the purpose of the evaluation?

Spend time thinking about why you are evaluating, and how you want to use the results of the evaluation. Do you want to:

  • Improve your intervention?
  • Demonstrate the effectiveness of your intervention?
  • Both improve and demonstrate effectiveness?

Improving the Intervention

An evaluation to identify how to improve a road safety project or intervention usually takes place when the project is first set-up. The intervention may not have started yet or a few ideas may have been trialled.

To understand how to improve the project you need to examine how the intervention is carried out, including its inputs and outputs. Inputs are all of the resources that go into the road safety project; such as money, staff time or expertise. Outputs are anything that is directly created as a result of the inputs; such as a leaflet or a number of presentations that are delivered. Describe these inputs and outputs in your logic model or project plan. You are, therefore, investigating the day-to-day mechanics of how the intervention is prepared and carried out.

Although the main focus of this type of evaluation is to investigate the day-to-day mechanics of how the intervention is prepared and carried out in order to identify improvements, ideally you would also examine some outcomes. Outcomes are not to be confused with outputs (although the similarity in their name doesn't help matters). Outcomes are more about the end-game and are usually the reason why the intervention is taking place. The outcomes in road safety education, training or publicity projects are usually:

  • To increase awareness or improve knowledge
  • To improve attitudes
  • To improve behaviours

However, it is likely that you will only be able to measure short-term outcomes in this type of evaluation. This means measuring knowledge, attitudes or intended behaviour change, immediately or soon after the intervention has taken place. The advantage of examining some outcomes is that you can establish weaknesses in the project's content which will require improvement. For example, if a particular message in an educational workshop is not well remembered then it will need to be improved.

Measuring Effectiveness

Does the road safety project work?

When there is a focus on measuring effectiveness, the project may have been running for a while, or effectiveness may now be being considered after some initial testing on the design and delivery of the project.

This type of project will need to measure outcomes. For road safety education, training or publicity projects, outcomes are usually:

  • To increase awareness or improve knowledge
  • To improve attitudes
  • To improve behaviours

At minimum these outcomes should be measured in the short-term. This may mean having the audience take part in the evaluation immediately after they have been involved in the intervention. Ideally, however, there should also be some further follow up, to see if any effects of the intervention are sustained over time.

In addition to measuring outcomes, ideally processes will be examined. These are the day-to-day mechanics of how the intervention is prepared and carried out. This investigation can provide clues as to why the project was, or was not, successful. For example, the outcomes may reveal that students found it difficult to remember a specific topic area. The process information may indicate, however, that workshop leaders found it difficult to cover all of the material in the time allocated.

Common ways to measure outcomes

Outcome evaluation designs are the different ways in which you can collect data to measure if the intervention has been effective in improving knowledge, attitudes and/or behaviours. They are used when you would like to put a number on how effective a project was (also known as quantitative designs, quant relating to quantity or number). For example, road safety attitudes were improved by 20%.

Outcome evaluation designs fall into three main categories:

  • Experimental (most robust)
  • Quasi-experimental
  • Non-experiment (least robust)

The category that will be chosen relies on a number of factors. The evaluator should conduct the most robust (or strongest) type of evaluation they can, within the resources that are available. show more >

Doing Both: Making improvements and Measuring Effectiveness

It is likely that the purpose of an evaluation, will be to examine both where improvements can be made to the design and delivery of the project, and to measure effectiveness.

If this is the case both the day-to-day mechanics of how the intervention is prepared and carried out and the outcomes (changes to knowledge, attitudes and/or behaviours) will need to be examined in detail. The sections above (Improving the Intervention and Measuring Effectiveness) provide more information on what each part requires.

Evaluation Design Exercise

For an exercise on evaluation design, suitable for an evaluation training event, please see our Workshop Materials page.