The toolbox aims to provide a one-stop-site for the evaluation of community sustainability engagement projects that aim to change household behaviours. Through the toolbox you can learn how to conduct your own evaluation of a behaviour change project using the guides and templates provided.

This toolbox is the product of a Victorian Local Sustainability Accord funded project (Round 2). The project involves the partnership of four local governments:

• Whitehorse City Council
• Knox City Council
• Bayside City Council
• Shire of Gannawarra

The academic partner is the National Centre for Sustainability at Swinburne University of Technology and they have been tasked with constructing the toolbox, templates and resources. The toolbox was developed by Damien Sweeney and Martin Pritchard from the NCS.  Martin and Damien have gone on to create Pacific Research & Evaluation Associates (PREA) which is focused on sustainability, international development and evaluation.

The toolbox is designed to be easy to use and transferable across local government in Victoria. Though it is a comprehensive resource that covers a range of methods to guide you in evaluating your projects, there are numerous other guides available that provide detailed advice on specific methods. These are pitched at different levels of evaluation knowledge. Where possible, this toolbox makes reference to other sources of information so that the user can use this resource as a gateway to the fascinating world of evaluation.

This development of this toolbox has been a staged process, commencing with a literature review. This research reviewed the evaluation methods undertaken by behaviour change projects both in Victoria, as well as internationally, in order to look at the effectiveness, benefits and constraints of various techniques. This overview led to the piloting of several evaluation methods with participating councils, and the development of case studies.  A range of other methods have also been included, and where possible, case studies have been sought to provide practical examples.

The need for an evaluation toolbox

Sustainability is seen as a major issue by local governments and their constituents. Behaviour change projects are increasingly being planned and implemented to achieve sustainability targets in the residential sector in areas as diverse as energy use and greenhouse gas emissions, water use, transport, waste and consumption. The increase in number of, and funding for, behaviour change projects leads to an assumption that such projects are effective in achieving their intended outcomes. The Department of Sustainability and Environment (Victoria) recently commissioned a stocktake of environmental behaviour change projects to identify best practice and gaps in delivery. The research identified gaps in evaluation practice and recommended the development of evaluation tools and indicators that covered both qualitative and quantitative measures.

Evaluation is a critical component of behaviour change projects due to the complexity and urgency of the issues that are being tackled, and the human element that creates a great deal of unpredictability. Monitoring and evaluation provides a process to test the effectiveness of the theory of change being implemented. There is limited literature that critically reviews the impact of behaviour change projects as a means to progress sustainability. There is equally limited literature that critically reviews the effectiveness of current evaluation methods for behaviour change projects.

There are however numerous articles and case studies on individual behaviour change projects. Case studies can particularly be found on Internet sites dedicated to behaviour change for sustainability (for example, www.toolsofchange.com and www.cbsm.com). What is interesting in reviewing these case studies is that most report positive outcomes, yet communities worldwide face increasing problems in meeting sustainability outcomes. This raises a number of fundamental questions:

• Are only successful case studies being documented, or is there pressure to demonstrate positive results in behaviour change projects (to receive further funding, or to justify the project)?
• Do the evaluation methodologies being used capture the full and true outcome?
• Are the largely positive results in many behaviour change projects a result of targeting people (or people self-selecting to participate) who are already undertaking sustainable behaviours, and if so, is this captured in the evaluation, and what does it mean in terms of project learnings?
• Are projects that show positive results with a small target group capable of being successfully scaled-up?
• Can the collection of data over a project’s implementation provide a  meaningful evaluation, or do behaviour change  projects require long-term evaluations?

These questions are pertinent especially with regards to residential energy, and greenhouse gas reduction, as this is the focus of many behaviour change projects, and poses significant challenges in terms of evaluation.

When to use experts

This evaluation toolbox is designed to provide Council officers and other project staff who do not have expertise in evaluation with the basic knowledge and skills to undertake an evaluation of a project. However, there are times when you may want to seek expert assistance, either to assist with a specific task, such as developing an evaluation plan, or to undertake a specific evaluation method.

In such circumstances, rather than outsourcing the task to an external party, you may want to consider having the experts build your organisation’s capacity to undertake the task, thereby not only getting something done, but getting training at the same time. The Australian Evaluation Society has a list of evaluation consultants, or you may ask around your network for suitable experts.

A Living Toolbox

Users of the evaluation toolbox are encouraged to provide feedback on their experience with the toolbox as a means to evaluate the project as well as to improve on its usability. We also encourage users to provide case studies on their use of the toolbox and in particular the evaluation methods so that lessons can be shared and the practice of evaluating behaviour change projects progresses.