Back to ukoer3 Final Synthesis Report

 

Programme Issues

1. Challenges relating to Project reporting mechanisms

The Evaluation and Synthesis Team have identified some questions around the effectiveness of current reporting mechanisms to capture important information from projects.

• Projects rarely used their blogs as a means of publishing lessons learnt as they progressed through their work. This meant that ongoing synthesis was very patchy.
• Project reporting templates do not appear to capture some of the important information that would feed in to programme level outcomes. For example,  interim reports delivered at the end of April 2012 hardly mentioned senior management engagement. Our team was asked to include this in our report to the Senior Advisory Committee meeting which resulted in a specific email request to projects for information about their senior management engagement. We asked them specific questions:

• How engaged your senior managers are with OER?

• What are you doing to engage them?

• Whether it seems to be working?

   

We received back some very interesting responses which resulted in the development of a wikipage on senior management engagement. This has since been augmented by information in final reports to develop a briefing paper). This important information would have missed this if we hadn't specifically asked for the information.

 

2. Project use of the Evaluation Toolkit and Framework

Information from projects could have been captured in a more comprehensive and systematic way if they had been required to use the Synthesis and Evaluation framework.

 

The Synthesis and Evaluation Team developed an Evaluation Toolkit  which provided various visual ways to connect with the framework and offered routes through the programme themes. Support was offered to help projects engage with the toolkit, through online meetings with evaluation buddy groups and input at programme meetings, direct email and telephone conversations, written and video guides.

 

One of the key features of the toolkit is the use of interactive google forms that allows projects to report on specific questions and receive back their inputs collated for inclusion in programme reporting mechanisms. However this was not linked to project report templates or encouraged as a reporting mechanism. In final buddy evaluation meetings several projects reported that they liked the toolkit and wanted to use it as it made sense to them. However they had to focus on reporting to programme templates so were unable to use the toolkit for this purpose. This was disappointing for our Team because it also meant that synthesis activities could not be ongoing and relied mainly on final reports. The final report template tends to encourage projects to repeat information (due to the nature of the section headings*), does not deliver engaging readable stories for the wider sector, results in key messages being omitted and makes synthesis challenging.

 

Evaluation buddying mechanisms and the Evaluation toolkit were perceived by projects as 'something extra' to do because they were not included in project plans so were challenging for projects working to very tight schedules and workplans. Despite this we had some excellent feedback about both mechanisms and feel that this approach could be of value to future programmes if they were:

• included in project plans
• encouraged by programme officers
• linked to formal reporting mechanisms

 

Comments about the Evaluation buddying process:

The DeFT project wrote an excellent blog post around their approach with buddies ORBIT and said

The meeting also helped us to see that despite the fact that we cover different discipline areas (the majority of DeFT teachers are in English or media while ORBIT focuses on science subjects) we have much more in common than we initially thought.

and

Yet another satisfactory outcome of the meeting was a joint strategy for sharing evaluation outputs

 

OMAC strand Teeside project reflected on their blog following the OMAC buddy meeting in April...

Just reflecting on how academic practice has changed having completed another skypeevaluation meeting with other projects in the OMAC strand. It was a very useful meeting sharing what we had learned and thinking about how to ensure the key lessons from the projects are captured in final reports. As someone relatively new to OER practice who has learned a great deal over the past year with this project, I am acutely aware that although OER practice and the infrastructure to support it across the sector is pretty well developed and there are colleagues with a great deal of expertise in this area, there are still many academics who are not engaged or experienced in this aspect of academic practice, nor is it necessarily very easy to access quickly the type of support needed. This is something we must look at if  mass participation in OER is to be achieved – assuming that is a shared goal.

 

 

 

* projects are asked to write to the following section headings which results in repetition and confusion over what to include in which section.

Project Outputs and Outcomes

 How did you go about achieving your outputs / outcomes?

What did you learn?

Immediate Impact

Future Impact

Conclusions

Recommendations

Implications for the future

 

 

Short Timescale and programme timings