Programme Issues

Project reporting mechanisms

Projects rarely used their blogs as a means of publishing lessons learnt as they progressed through their work. There are questions around the effectiveness of current reporting mechanisms to capture important information from projects. This is illustrated by the fact that interim reports delivered at the end of April 2012 hardly mentioned senior management engagement at all. On being requested to include this in our report to the Senior Advisory Committee meeting we had to go out to projects and specifically ask for information about their senior management engagement. We asked them specific questions:

• How engaged your senior managers are with OER?

• What are you doing to engage them?

• Whether it seems to be working?

We received back some very interesting responses which resulted in the development of a wikipage on senior management engagement (this has since been augmented by information in final reports.) However this served to highlight that we would have missed this if we hadn't specifically asked for the information.

This may have been obtained if projects had been asked to report using the Synthesis and Evaluation framework or Evaluation Toolkit - as this is a specific question in that framework. This is discussed further in the next section...

Evaluation Toolkit and Framework

The Synthesis and Evaluation Team spent considerable time developing a Evaluation Toolkit to help projects engage with the Framework because of challenges experienced in previous phases of the programme. The toolkit provided various visual ways to connect with the framework and offered routes through the programme themes. Dialogue with projects took a range of formats, including online meetings with evaluation buddy groups and input at programme meetings, direct email and telephone conversations, and this provided an opportunity to inform and engage projects with the toolkit to support evaluation.  We also produced written and video guides to encourage engagement.

One of the key features of the toolkit is the use of interactive google forms that allows projects to report on specific questions and receive back their inputs collated for inclusion in programme reporting mechanisms. However this was not linked to project report templates or encouraged as a reporting mechanism. In final buddy evaluation meetings several projects reported that they liked the toolkit and wanted to use it as it made sense to them. However they had to focus on reporting to programme templates so were unable to use the toolkit for this purpose. This was disappointing for our Team because it also meant that synthesis activities could not be ongoing and relied mainly on final reports. The final report template tends to encourage projects to repeat information (due to the nature of the section headings*), does not deliver engaging readable stories for the wider sector, results in key messages being omitted and makes synthesis challenging.

Evaluation buddying mechanisms and the Evaluation toolkit were perceived by projects as 'something extra' to do because they were not included in project plans so were challenging for projects working to very tight schedules and workplans. Despite this we had some excellent feedback about both mechanisms and feel that this approach could be of value to future programmes if they were:

• included in project plans
• encouraged by programme officers
• linked to formal reporting mechanisms

* projects are asked to write to the following section headings which results in repetition and confusion over what to include in which section.

Project Outputs and Outcomes

How did you go about achieving your outputs / outcomes?

What did you learn?

Immediate Impact

Future Impact

Conclusions

Recommendations

Implications for the future

Short Timescale and programme timings