OER Synthesis and Evaluation / phase3ProgrammeIssues
  • If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • You already know Dokkio is an AI-powered assistant to organize & manage your digital files & messages. Very soon, Dokkio will support Outlook as well as One Drive. Check it out today!

View
 

phase3ProgrammeIssues

This version was saved 11 years, 4 months ago View current version     Page history
Saved by Lou McGill
on November 2, 2012 at 2:01:23 pm
 

Back to ukoer3 Final Synthesis Report

 

Programme Issues

1. Challenges relating to Project reporting mechanisms

The Evaluation and Synthesis Team have identified some questions around the effectiveness of current reporting mechanisms to capture important information from projects.

  • Projects rarely used their blogs as a means of publishing lessons learnt as they progressed through their work. This meant that ongoing synthesis was very patchy.
  • Project reporting templates do not appear to capture some of the important information that would feed in to programme level outcomes. For example,  interim reports delivered at the end of April 2012 hardly mentioned senior management engagement. Our team was asked to include this in our report to the Senior Advisory Committee meeting which resulted in a specific email request to projects for information about their senior management engagement. We asked them specific questions:
    • How engaged your senior managers are with OER?

    • What are you doing to engage them?

    • Whether it seems to be working?

       

We received back some very interesting responses which resulted in the development of a wikipage on senior management engagement. This has since been augmented by information in final reports to develop a briefing paper). This important information would have missed this if we hadn't specifically asked for the information.

 

2. Project use of the Evaluation Toolkit and Framework

Information from projects (including stakeholder engagement issues) could have been captured in a more comprehensive and systematic way if they had been required to use the Synthesis and Evaluation framework.

 

The Synthesis and Evaluation Team developed an Evaluation Toolkit  which provided various visual ways to connect with the framework and offered routes through the programme themes. Support was offered to help projects engage with the toolkit, through online meetings with evaluation buddy groups and input at programme meetings, direct email and telephone conversations, written and video guides.

 

One of the key features of the toolkit is the use of interactive google forms that allows projects to report on specific questions and receive back their inputs collated for inclusion in programme reporting mechanisms. However this was not linked to project report templates or encouraged as a reporting mechanism. During final buddy evaluation meetings it became evident that projects were aware of the toolkit and it's potential value. Several projects reported using the framework and toolkit to shape and refine their evaluation questions. However they had to focus on reporting to programme templates so were unable to use the toolkit for this purpose. This had a negative impact on ongoing synthesis activities as the team relied principally on final reports. The final report template tends to encourage projects to repeat information (due to the nature of the section headings*), does not deliver engaging readable stories for the wider sector, results in key messages being omitted and makes synthesis challenging.

 

Engaging with the Evaluation toolkit was perceived by projects as 'something extra' to do because it was not specifically included in project plans.  Despite this we had some excellent feedback and feel that this approach could be of value to future programmes if it was:

  • included in project plans
  • encouraged by programme officers
  • linked to formal reporting mechanisms

 

The report was underpinned by the UKOER evaluation and synthesis framework and drew on an overview of project documentation (project wiki, blog and website), a focus group with the teachers involved in the project and e-mail consultations with project team. (DEFT Final Report)

 

3. Evaluation buddying teams

Evaluation buddy mechanisms were established in consultation with the Programme Manager and encouraged at the Programme start-up meeting. Some projects did refer to this in project plans. This proved a useful mechanism for the team to engage directly with projects and had the following positive outcomes:

  • provided an opportunity to discuss use of the framework and toolkit and identify any issues or challenges relating to evaluation
  • helped them to identify links with other projects. We had some measure of success in getting projects to link evaluation, and other, activities.
  • proved an effective means to gather some of the outcomes of project activities and supported ongoing synthesis
  • provided projects with someone to share their concerns with - particularly projects new to the UKOER programme
  • highlighted significant lessons for the wider community and resulted in the production of some collaborative briefing papers

 

All projects were offered an opportunity to attend two online evaluation buddy meetings (before interim and final report due dates). Not all projects took this opportunity as this was also perceived by some projects as extra to their agreed workplans. Some projects also took the opportunity for extra one-to-one meetings with team members.

 

Comments about the Evaluation buddying process:

The DeFT project wrote an excellent blog post around their approach with buddies ORBIT and said

The meeting also helped us to see that despite the fact that we cover different discipline areas (the majority of DeFT teachers are in English or media while ORBIT focuses on science subjects) we have much more in common than we initially thought.

and

Yet another satisfactory outcome of the meeting was a joint strategy for sharing evaluation outputs

 

The team also benefitted from the support of Evaluation and Synthesis team who facilitated peer evaluation based on involvement with the ORBIT (The Open Resource Bank for Interactive Teaching at Cambridge University) project. Through that arrangement, we were able to take advantage of synergies between projects both of which focused on OER issues within the school sector. The teams met in Sheffield on 16 May and also took part in a series of Skype conversations (23 April, 8 and 29 August). These interactions led to numerous collaborative outputs
•A deeper understanding of the synergies between the two projects and exchange of ideas for collaboration beyond the lifetime of the project
•An exchange of ideas for further collaborative development and involvement of DeFT project participants in ORBIT initiatives such as a survey for teachers
• A joint ORBIT-DeFT Kanban (visual process management tool) for sharing evaluation outputs and exchange approaches to evaluation
•The creation of a DeFT “family” page within ORBIT wiki listing all DeFT case studies (see http://orbit.educ.cam.ac.uk/wiki/DEFT). The ORBIT team have re-purposed the DeFT case studies into interactive lesson plans. On top of being an excellent example of content reuse, sharing the case studies via ORBIT wiki increases exposure and aids dissemination efforts. (DEFT Final Report)

OMAC strand Teeside project reflected on their blog following the OMAC buddy meeting in April...

Just reflecting on how academic practice has changed having completed another skypeevaluation meeting with other projects in the OMAC strand. It was a very useful meeting sharing what we had learned and thinking about how to ensure the key lessons from the projects are captured in final reports. As someone relatively new to OER practice who has learned a great deal over the past year with this project, I am acutely aware that although OER practice and the infrastructure to support it across the sector is pretty well developed and there are colleagues with a great deal of expertise in this area, there are still many academics who are not engaged or experienced in this aspect of academic practice, nor is it necessarily very easy to access quickly the type of support needed. This is something we must look at if  mass participation in OER is to be achieved – assuming that is a shared goal.

 

ALTO project final report included an acknowledgement to the Evaluation and Synthesis team

     for comment, guidance and convening some very useful online discussions and providing the evaluation toolkit, which we have found very useful.    

 

* projects are asked to write to the following section headings which results in repetition and confusion over what to include in which section.

Project Outputs and Outcomes

 How did you go about achieving your outputs / outcomes?

What did you learn?

Immediate Impact

Future Impact

Conclusions

Recommendations

Implications for the future

 

 

Short Timescale and programme timings

 

The timing and length of projects should be considered carefully when issuing funding calls. The short turnaround time for this project made it difficult for many part-time tutors to commit to it – a longer project life would have allowed tutors to engage with the project at different points in its cycle and would have suited their working patterns better. In addition, the timing of the project from October to October meant that key project activities coincided with heavy institutional workloads (beginning of the new academic year), and that student evaluation of new materials was not possible at the close of the project. (FAVOR Final Report)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Comments (0)

You don't have permission to comment on this page.