Programme Issues

Project reporting mechanisms

Projects rarely used their blogs as a means of publishing lessons learnt as they progressed through their work. There are questions around the effectiveness of current reporting mechanisms to capture important information from projects. This is illustrated by the fact that interim reports delivered at the end of April 2012 hardly mentioned senior management engagement at all. On being requested to include this in our report to the Senior Advisory Committee meeting we had to go out to projects and specifically ask for information about their senior management engagement. We asked them specific questions:

• How engaged your senior managers are with OER?

• What are you doing to engage them?

• Whether it seems to be working?

We received back some very interesting responses which resulted in the development of a wikipage on senior management engagement (this has since been augmented by information in final reports.) However this served to highlight that we would have missed this if we hadn't specifically asked for the information.

 

This may have been obtained if projects had been asked to report using the Synthesis and Evaluation framework or Evaluation Toolkit - as this is a specific question in that framework. This is discussed further in the next section...

 

Evaluation Toolkit and Framework

The Synthesis and Evaluation Team spent considerable time developing a Evaluation Toolkit to help projects engage with the Framework because of challenges experienced in previous phases of the programme. The toolkit provided various visual ways to connect with the framework and offered routes through the programme themes. Dialogue with projects took a range of formats, including online meetings with evaluation buddy groups and input at programme meetings, direct email and telephone conversations, and this provided an opportunity to inform and engage projects with the toolkit to support evaluation.  We also produced written and video guides to encourage engagement.

 

One of the key features of the toolkit is the use of interactive google forms that allows projects to report on specific questions and receive back their inputs collated for inclusion in programme reporting mechanisms. However this was not linked to project report templates or encouraged as a reporting mechanism. In final buddy evaluation meetings several projects reported that they liked the toolkit and wanted to use it as it made sense to them. However they had to focus on reporting to programme templates so were unable to use the toolkit for this purpose. This was disappointing for our Team because it also meant that synthesis activities could not be ongoing and relied mainly on final reports. The final report template tends to encourage projects to repeat information (due to the nature of the section headings*), does not deliver engaging readable stories for the wider sector, results in key messages being omitted and makes synthesis challenging.

 

Evaluation buddying mechanisms and the Evaluation toolkit were perceived by projects as 'something extra' to do because they were not included in project plans so were challenging for projects working to very tight schedules and workplans. Despite this we had some excellent feedback about both mechanisms and feel that this approach could be of value to future programmes if they were:

• included in project plans
• encouraged by programme officers
• linked to formal reporting mechanisms

 

Comments about the Evaluation buddying process:

The DeFT project wrote an excellent blog post around their approach with buddies ORBIT and said

The meeting also helped us to see that despite the fact that we cover different discipline areas (the majority of DeFT teachers are in English or media while ORBIT focuses on science subjects) we have much more in common than we initially thought.

and

Yet another satisfactory outcome of the meeting was a joint strategy for sharing evaluation outputs

 

OMAC strand Teeside project reflected on their blog following the OMAC buddy meeting in April...

Just reflecting on how academic practice has changed having completed another skypeevaluation meeting with other projects in the OMAC strand. It was a very useful meeting sharing what we had learned and thinking about how to ensure the key lessons from the projects are captured in final reports. As someone relatively new to OER practice who has learned a great deal over the past year with this project, I am acutely aware that although OER practice and the infrastructure to support it across the sector is pretty well developed and there are colleagues with a great deal of expertise in this area, there are still many academics who are not engaged or experienced in this aspect of academic practice, nor is it necessarily very easy to access quickly the type of support needed. This is something we must look at if  mass participation in OER is to be achieved – assuming that is a shared goal.

 

 

 

* projects are asked to write to the following section headings which results in repetition and confusion over what to include in which section.

Project Outputs and Outcomes

 How did you go about achieving your outputs / outcomes?

What did you learn?

Immediate Impact

Future Impact

Conclusions

Recommendations

Implications for the future

 

 

Short Timescale and programme timings