Weaker Designs: Non-Experiments
The designs (types of evaluation) listed on this page are usually the least complex and time-consuming to run. They can be a good starting point for those who are not confident in conducting evaluations. They are, however, the least strong form of evaluation and so it is difficult to say whether any change seen was due to the road safety project that you implemented.
There are different types on non-experimental design but all only involve those who receive the intervention, i.e. there is no comparison against those who do not receive the intervention.
Before and After (no control or comparison group)
A pre and post design without a control or comparison group is a way to measure change in one group of participants, after the intervention, compared to before the intervention. Participants are surveyed at two time points – once before the intervention, and once afterwards. This is a non-experimental design.
Instead of surveying the participants before the intervention begins, the pre-test measurement could be obtained from existing documents. For example, a driver's record book recording the amount of private practice conducted before the intervention started could be used as a before measurement, as opposed to collecting that information through surveys or interviews. The before-data may already exist but it will most likely need to be collected specifically for the purpose of the evaluation.
The change is calculated by detracting the before measurement score from the after measurement score (before score – after score = amount of change).
Strengths
- This is a simple and relatively inexpensive design as only one group of participants needs to be sampled.
- The same group of participants is surveyed at two time points – once before the intervention, and once again, after the intervention. Therefore, you need to have repeat access to your participants.
- You could test the sustainability of your intervention by doing a longer term follow up with the same group.
Weaknesses
- Without a control or comparison group there is no way of knowing whether or not it was the intervention that caused the observed effect. It is always possible that the observed change occurred regardless of the participants' exposure to the intervention.
- The act of repeat testing (asking the same questions in the after intervention survey, as were asked in the before intervention survey) can simulate the effects of the intervention by making your participants more aware of the issue.
- It can be difficult to retain contact with your participants over time and your findings may be biased depending on the characteristics of those who 'drop out'.
Case Study
A case study is a non-experimental type of evaluation and provides rich descriptive information about the experience of a small number of participants. Case studies are commonly carried out using a mix of different research methods. Due to the in-depth detail of the data collected, case studies are especially useful for looking at how an intervention was implemented, and why it was successful (or not). Data is collected after the intervention has been introduced. The cases chosen for study can be typical or extreme examples, so either a participant who does particularly well (or badly) with an intervention, or a participant who shows an average level of engagement. Case studies can follow participants at any stage of an intervention.
Strengths
- Case studies offer holistic understanding of an issue, and help to explain what happened and why.
- They are valuable in describing interventions operating in complex circumstances. Case studies are a rigorous method where an experimental or quasi-experimental design is not possible.
- They also complement quantitative data collected through other research methods.
Weaknesses
- Case studies can take a long time to follow the participants and to collect and analyse large amounts of data from multiple sources.
- Participants will need to be contacted more than once or twice which may increase the cost of the evaluation.
- Findings cannot easily be generalised to the wider population.
After-then-Before design
In an after-then-before survey, participants who received the intervention are asked to make self-judgements about their behaviour, or knowledge of an issue, both before and after the intervention. The participants are surveyed at one time point only, after the intervention, and they are also asked to retrospectively rate their behaviour or level of knowledge for the period before the intervention. As you are only taking one measurement there are no problems retaining contact with participants.
Strengths
- It is not necessary to contact participants a second time so there are no problems with drop out.
Weaknesses
- Participants' recall of their earlier knowledge or behaviour may be imperfect, or they may exaggerate the difference to please the researcher.
After intervention only (no control or comparison group)
This is the simplest, quickest and cheapest type of evaluation, although it is also the least robust. It involves one measurement of one group at only one point in time.
Strengths
- This design may be useful where participants' existing level of attitude/behaviour/knowledge can be assumed to be low. However, you would need to be able to justify this assumption.
- As you are only taking one measurement at one point in time, there are no problems retaining contact with participants.
- Sometimes this may be the only type of evaluation possible where evaluation was not planned at the same time as the intervention was.
Weaknesses
- Without a control group, or a base line measurement (pre-test) it is not possible to say if the level of attitude, behaviour, knowledge or other desired outcome, is due to the intervention, or if the participants did not already have this level of attitude/behaviour/knowledge, before the intervention took place.
- Using this design it will not be possible to say that observed effects were caused by the intervention.