Review of HEFCE-funded Open Educational Resources (OER) initiatives: report on Score activities
Jay Dempster, Lou McGill, Isobel Falconer and Allison Littlejohn
Evaluation and Synthesis Team
November 2012
INTRODUCTION
This report offers a retrospective evaluation of the Support Centre for Open Resources in Education (SCORE) supported work, as it pertains to a wider review of HEFCE funded Open Educational Resources (OER) initiatives encompassing three phases of the JISC/HE Academy's UKOER programme (2009-2012) and the Open University's (OU) national role via the SCORE initiative (2009-2012). Both UKOER and SCORE aimed to advise on and support development and publication of resources in more ‘open’ form as well as to raise awareness, draw out key issues and deepen understanding with regard to open educational practices.
The HEFCE OER review is being undertaken by the existing UKOER Synthesis and Evaluation Team. It includes evaluation of SCORE supported work, and the incorporation of this into the core UKOER methodology, modified and expanded as appropriate to the distinct aims and activities of SCORE. This SCORE evaluation is based on documentary evidence and consideration of qualitative outcomes. (We understand from the SCORE Director that quantitative evidence concerning volume and uptake of deliverables against stated KPIs for OER production/publishing, fellowships and events is being reported separately by the SCORE team.) Central to supporting the OER community was SCORE’s online presence and, again, data concerning usage statistics and participation via the web, blog, social media, for instance are likely to be valuable evidence of success against KPIs concerning sector engagement, but are excluded from this analysis.
The evaluation focused on gathering new evidence from stakeholders; with respect to SCORE, Including recipients of SCORE fellowships (both teaching fellows and residential course attendees), advisory board members, institutional representatives, and (as far as possible or indirectly) included course leaders and students. An interim report which focussed on SCORE activities was produced in May 2012 and this was augmented and published in July 2012. These reports and synthesis of key outcomes will be incorporated with UKOER findings into the HEFCE OER Review Final report.
BACKGROUND
The UKOER and SCORE supported OER work funded by HEFCE builds on past sector investment in sharing resources, including JISC Exchange for Learning x4L, Jorum, JISC Digital Repositories Programme, JISC Repositories and Preservation Programme and JISC ReProduce Programme. Some activity has arisen independently of formal reuse or OER projects as an extension of more general learning and teaching activity, e.g. through the HE Academy subject centre or national teaching fellow activity. Since its inception in 2009, The OU's national role through SCORE sat alongside two phases of the JISC/Academy funded UKOER programme initiated in 2009, and completed during phase 3 of the UKOER programme. Following this investment, there is appetite from HEFCE to see a cumulative synthesis and evaluation of their entire funded intervention in OER (hitherto referred to as the HEFCE OER review). The purpose is two-fold: (a) to deepen understanding of open educational practices and resources, and (b) to produce a solid evidence base and enhance the status of the work supported in the UK and in the international OER field.
Throughout the UKOER programme, evaluation findings from projects were externally synthesised by a team led by Glasgow Caledonian University. The team developed and refined an evaluation framework and associated methodologies, which have contributed to the developments of reports and background data collated on a wiki platform. Notably, the synthesis has highlighted aspects of culture and practice change (at individual, community and institutional levels) which has been enabled and supported by engaging in the release and use of OER. Another significant focus of UKOER work has been in identifying benefits of different models for a wide range of stakeholders across a number of sectors, both within and outside the educational sector. The UKOER programme has led to increased understanding around the complexity of the open educational landscape - discussed in both the Open Practices Briefing Paper and a blog post by one of the JISC Programme managers visualisations of the wider open education landscape.
The HEFCE OER review extended the synthesis and evaluation framework used for the UKOER programme to include the aims and activities of SCORE, thus promoting a unified emphasis on the programmes as part of a whole HEFCE investment in OER. In this respect, visualisation outputs produced by staff in CETIS were utilised to assist in communicating the overlap in participation between the UKOER and SCORE programmes, underpinning the argument for looking at the HEFCE investment as a whole.
The framework for UKOER evolved throughout all three phases of the UKOER programme. Evaluation & synthesis was an iterative, two-way process such that projects contributed to the development of the framework throughout and that each iteration of the framework reflected the three phases. The OER phase 1 pilot programme enabled the large scale release of OER, the three strands of funding allowed different approaches, benefit cases and technical solutions to be trialled in a genuinely diverse mix of contexts. Phase 2 extended OER release but also supported activity areas around OER use and discovery. Phase 3 projects investigated the use of OER and open approaches to work towards particular strategic, policy and societal goals through a thematic approach: Theme A : Extend OER through collaborations beyond H; Theme B: Explore OER publishing models; Theme C: Addressing sector challenges; Theme D: Enhancing the student experience. All three phases explored individual, institutional and community issues around embedding, sustainable practice and widening engagement with OER.
The framework had a number of key focus areas, each with a range of evaluation questions that might be addressed by project teams as they carried out their individual project evaluations. Individual project and strand/theme findings were mapped to the questions in the synthesis framework, providing an overview of key issues and trends across the programme. This approach highlighted both key outcomes and significant outputs that demonstrate evidence of these.
PURPOSE, SCOPE & METHODS
While the intention was to undertake an evaluation of UKOER and SCORE as a collective whole, an initial review of SCORE's documentary evidence, reported here, enables consolidation of the UKOER framework with SCORE's unique aims and activities. The Board noted (in its minutes from a meeting held 12-03-2010) that "there are important benefits to harvest from the SCORE project" and "the need for clear analysis of any outcomes", and recommended preparation of a draft impact assessment plan {action 6.4.3 in NARB/10/3/M meeting 27-10-2010}.
This review based on SCORE documentary evidence forms part of our wider HEFCE OER review; its main aim is to update the OU's National Role Advisory Board with findings. The full review supported four discrete areas of activity which reflect the fact that it began during phase three of the UKOER Programme, and this synthesis of SCORE documentation contributes to the second and third areas:
- Retrospective re-evaluation of the findings from UKOER phases 1 and 2, to include the gathering of further data from projects beyond their funded life.
- Evaluation of currently active SCORE supported work, preferably using a methodology compatible with the core UKOER methodology.
- Retrospective evaluation of SCORE supported work, and the incorporation of this into the core UKOER methodology.
- A cumulative synthesis of findings from the above strands and the parallel evaluation/synthesis of UKOER phase 3, and the presentation of this in formats suitable for researchers, policy-makers and a general audience.
The Board recommended that qualitative data should be used to indicate the impact being made, particularly at a strategic level within the higher education sector, including who has been impacted and how; why people aren’t doing this already; what the benefits are. Suggestions for questions around OER use and exploring impact included:
- What is happening to educators and how are they responding?
- How OERs are impacting the design of education?
- What opportunities have arisen that have not been seen before?
- How is the use of OERs changing what we do?
These questions remain pertinent and have been addressed in all phases of UKOER work through the synthesis framework, which was utilised to capture stakeholder views as part of the overall HEFCE OER evaluation. The Synthesis & Evaluation team designed an overarching modified framework and engaged the SCORE team in the shaping process. An overview was presented at the May meeting of current SCORE fellows/mentors to invite comments on its relevance for SCORE work and identify any distinctions arising from SCORE's specific context or activities. Responses informed further refinement of the framework, including audience, scope and focus towards a working set of questions, which formed the basis of evaluation instruments for the full review.
HEFCE funded OER work in the UK has been extensive and impacts on strategy, policy, practice (of a wide range of stakeholders, including learners), research, curriculum design, delivery and support. In order to make the fairly complex framework accessible we offer four top level areas: Culture & Practice, Releasing & Using OERs, Processes for Sustainability and Impact & Benefits. These are then broken down further within the framework to reflect more nuanced aspects of each broad area.
On first view, the SCORE supported activity (see below) appears to map well across these focus areas. As one would expect, there is strong overlap, including in terms of the individuals involved in both UKOER (some all three phases) and SCORE. There are also potentially some distinct elements of SCORE's approaches, such as 'research', which have been included as cutting across all four focus areas and its locus will be explored further as part of the review with stakeholders. The SCORE community have already demonstrated a desire to engage with an overarching framework for OER and to share collective wisdom and help to gather a solid evidence base around the interventions and outcomes of these UK wide initiatives.
To fit in with the SCORE reporting deadline in May, the Synthesis & Evaluation team initially engaged in the following activities:
- Gathering existing documentary evidence from the SCORE project team
- Mapping this data onto an updated evaluation framework suitable for both UKOER and SCORE initiatives
- Discussion with SCORE director & fellows around the framework and a set of draft questions for this UK wide evaluation
- Analysis and synthesis of this data, in order to draw out early thoughts (themes and significant issues).
Documentation included in this initial review of SCORE included:
- 31 SCORE Fellows project proposals, progress reports
- 5 Short Term Fellows Residential Course feedback (based around experience of the sessions immediately following the course)
- 12 Short-term Fellowship 6 month Post Course Review of Progress (based on a more reflective proforma, 2 were also SCORE Teaching Fellows);
- 7 National Role Advisory Board meetings: papers, including SCORE plans & activities, key decisions & achievements, project outputs
- 12 SCORE workshop events feedback from participants
- SCORE Fellows outputs - publications, conference presentations/posters, workshops, online activities;
- Notes & observations from SCORE fellows meeting (26th April)
- Informal interview with the SCORE Director & Communications Manager
- Website blog posts, twitter activity, cloudworks, google analytics
These sources were reviewed using the core UKOER framework to extract key themes, lessons learned and tangible benefits as part of this retrospective evaluation and demonstrate the extent to which intended (and unanticipated) outcomes have been achieved.
The nature and volume of activity such as fellowships and events are easily evidenced from the records and have been reviewed by the OU National Advisory Board. A SCORE Dashboard & KPIs sheet provided actual data for activity against targets for the three year period.
- volume (number run/supported);
- type (projects, courses, disseminative conferences, exhibitions, interactive workshops, training & staff development);
- uptake/participation (numbers attending/completing);
- partnerships/collaborations (e.g. CC workshops with STEM, successful bids for other OER work);
Further data was collected between May and November 2012 via structured surveys, and interviews to draw out further themes, explore issues and findings more specifically, towards a narrative supported by evidence and illustrations from across the UKOER & SCORE initiatives.
FINDINGS
How SCORE have approached the work
The business plan for SCORE (Dec 2008) set forward a case for supporting strategic development of OER across the HE sector as a whole, drawing on the OU’s competencies in this area to address sector-wide needs. As noted by the OU's National Role Advisory Board (paper NARB/10/1/3) responsible for monitoring progress of SCORE supported activities, the plan suggested that "it is time to move the sector from a supplier-led approach to the creation of web-based OERs to one which reflects the needs of both institutions and students".
The SCORE work draws on the OU experience in OpenLearn and its other OER projects (e.g. TESSA, OEL) to: disseminate skills for populating and maintaining open content sites; address intellectual property and rights management issues; extend sector-wide understandings of the needs and behaviours of users of open content material; and inform and share national and international research findings. There was also an additional undertaking to facilitate engagement with international developments, particularly the Open Courseware Consortium and the European MORIL project.
The SCORE supported work over the 3-year funding period has set out to support and inform individuals, projects, institutions and programmes across the higher education sector in England. The funding has been allocated to support four types of activity: sector engagement, fellowships projects, events and production/publishing of OERs. SCORE's original plan included:
- 36 SCORE fellowships to fund project activities which inform and influence policy and practice around OER creation, sharing and use;
- an additional 3,600 hours of OER content;
- an enquiry-based support and advice service;
- a range of events to support dissemination across the sector;
- a vibrant Web 2.0 based virtual community.
Various changes were implemented in year one in light of the Online Learning Task Force recommendations and OU developments. SCORE targets and KPIs were consequently revised to reflect new priorities and directions for SCORE activities (see Appendix A). A retrospective review of achievements against these revised targets (in the SCORE team's final report to the Board) reveals that a wide reaching range of structured support has been delivered, including funded projects to individuals, residential courses, staff development workshops and dissemination activities. SCORE's Dashboard and KPIs report (11-01-2012) to the Shared Solutions Steering Group indicates that targets have been met or exceeded, including the release of almost 1500 hours of OER published internally within the OU, and nearly 600 hours OER published externally by SCORE members.
The work of the SCORE team, and specifically the work of the fellows and other OER activity at the OU, has been disseminated widely (at events, in formal publications and online) with the aim of informing sector-wide OER policy and practice. It was noted that opportunities are being provided to connect the work of the fellows to the JISC/HE Academy UKOER Programme and other relevant programmes such as JISC Digital Media. For example, Sharon Waller from the Higher Education Academy attended the Fellows meeting and there is much useful overlap between SCORE fellows in institutions participating in UKOER. Other connections were also actively sought, for instance with the HE Academy Open Materials for Accredited Courses strand and use of LabSpace by OMAC projects.
Responding to the massive changes in the economic and political climate around the close of year 1 (2010), there was a requirement for SCORE to be responsive to the outcomes of HEFCE’s Online Learning Task Force (OLTF), together with lessons learned from its own activities, research and fellowship project activity, which informed a modified work plan for years 2-3. As stated in an early paper tabled at the OU's National Role Advisory Board {NARB/10/1/3}:
"There is no shortage of ‘leads’ for future OER activity, and some of these are already taking the project in unexpected directions ... The variety of opportunities and the newness of OER activity to all institutions, even the OU, creates challenges for identifying where to support activity across what is a very wide front."
Response to change
Reflection on key priorities prompted the team to rethink what SCORE did, particularly in terms of the selection of content which is most in demand across the sector as OER, and support activity which meets real needs by institutions, projects and individuals engaged in higher education. Given sustainability as one of the remits of the OU's National Role Advisory Board's involvement, examples of how the SCORE work has evolved to meet changing needs & demand demonstrate effective response to need through formative monitoring and evaluation:
- The Academic Partnerships programme, which had been part of Shared Solutions was removed, which meant that some of the work being undertaken by it was incorporated into SCORE work. This inclusion enabled SCORE to undertake some of the consumer market testing and proved important to its responsiveness to the changing needs of its market.
- Applications for SCORE Teaching Fellowships were predominantly from staff already quite experienced in OER. This benefited SCORE in terms of delivering outcomes relatively quickly, yet fell short of its staff development aims. It was therefore felt that a 'taught course' route via a new and separate ‘Short-term Fellowship Scheme’ provided a means by which newcomers to OER can acquire the knowledge and understanding necessary to get started in working with OER. The one week long residential courses (with pre- and post-residential tasks) offered more opportunity for the SCORE team to fulfil a staff development need and proved very successful.
- Feedback from participants at workshops and conferences hosted or supported by SCORE helped identify and assess indicators of impact for this area of work. Similarly, residential participants were surveyed through six month follow-ups in order to explore what changes to their own practice occurred, and what influence they’ve had on their colleagues.
- A significant element of the SCORE budget was originally allocated to developing OER Publishing, building on the significant work of the OU investment in open learning in partnership with Hewlett (and at this point with no HEFCE funding). As the OU moved from creating content for OpenLearn & copyright checking services, to more mainstream approaches, OpenLearn was seen as an integral part of creating new courses, managing rights and structuring content. With the availability of the Open University's structured content system, which enables open content to be produced at low cost, this reduced the need for SCORE funding to be used to create new OER.
- Checking back with the community around their priorities for OER proved challenging, with information coming back indicating "a little bit of everything" rather than identifying any key areas of interest to educators. However, in response to key messages from the OLTF, the SCORE team took the opportunity to move away from OER publishing as a service towards a development project. Two sites were identified and progressed to meet the needs of two different communities : (1) international students interested in studying in the UK (Ready2Research) and (2) those looking to enhance their digital scholarship activities (Digital Scholarship). Both sites made use of OER identified through a consortium that was put together for this purpose and that provided the necessary skills in a user-friendly format. The developments filled a need to inform and enhance the culture, attitudes and behaviours in the use & sharing of open content. In fact, this offered an opportunity to test the power & suitability of OER to meet a clear need.
- A key benefit to the sector of this work is that it was led by the consumers rather than producers of open learning content (assuming the two are not the same, which may not be the case). A survey conducted through VCs' offices and a study involving Subject Centres confirmed a slight preference towards STEM courses, minority subjects, OER supporting skill development and employability and whole courses rather than extracts. There was otherwise little steer from the sector as to where to go with OER publishing and some concerns about the platform for contributing OER (over and above institutional and Jorum repositories).
- Existing networks, such as the SUPER-OERand National Teaching Fellowship Scheme were utilised to draw interest and promote the opportunities to a relevant (and eligible) segment of the sector. Potential new and significant communities of practice were also tested on early career staff, such as those participating in HEA-accredited courses and those interested in research scholarship in teaching and learning.
- SCORE's help & advice service was less well utilised; the SCORE Director/Communications Manager felt this was probably because enquiries came more from face to face events and questions were addressed in person by the SCORE team or within the community.
SCORE sustainability
In the move from buy-in and awareness of OER to maintenance, of key concern to SCORE is bringing about significant, lasting change to attitudes on OER. Recommendations from the OU's National Role Advisory Board therefore favoured plans centred on the creation of a self-funded network. Sustainability was evaluated on how collaborative activity could be made self-funded and how the project worked with partner institutions to mainstream OER activity. Members of the Board noted {NRAB/11/1/M 3.2.7} that while some institutions are embracing OER, an increasing number are questioning their involvement in the movement, resulting in polarisation within the sector and potential break off towards specialist groups. The Board {NARB/11/1/4} suggested that engaging with Professional Bodies could also help reduce the risk of institutions disengaging from OER work. Approaches to institutional embedding would help inform decisions of senior managers, drawing on evidence from the third strand of SCORE's sustainability plan to demonstrate tangible value to The Open University by facilitating the incorporation of open educational resources into OU course materials.
As part of its sustainability plan, there was felt also to be merit in working towards a possible commercial plan. Slightly more than half of the income was expected to come from events, so testing the market for its willingness to pay provided vital evidence as a basis for decisions and pricing structures. Noted indicators of the viability of charging for services and events included:
(i) Demand for Short Term Fellowships (and willingness to pay the residential course fee) by individuals from international institutions, but notably not from other UK nations (outside of England).
(ii) On occasions where SCORE charged for workshops, there did not appear much fall in bookings.
(iii) Many times the number attending the physical events were actually viewing the recordings online. {NARB/10/1/M}
(iv) The annual OER conference has now run for two consecutive years as a pay-to-attend event, so there is reasonable confidence that this aspect of the proposals is viable.
Focusing on just one activity may simplify decisions and planning and reduce tensions between activities, but may not foster the breadth of appeal needed to develop and sustain a network. Members of the Board strongly urged the project to develop a number of different sustainability scenarios for post July 2012 {NARB/10/3/M}.
Broad indicators of success
The OU's National Role Advisory Board noted that members felt "SCORE had met expectations to-date, and that in some cases the project had exceeded them" and that "SCORE’s record was considered to be impressive" {NRAB/11/2/1 minutes from meeting held on 25-11-2011}. SCORE attracted 108 fellowship applications including for both Teaching Fellowships & Short Term Fellowship (STF) Residential Courses, which is 35% above its three year target KPI of 80.
SCORE Teaching Fellowships
SCORE recruited a total of 31 Fellowships across 19 institutions (30 funded Teaching Fellows and one unfunded Fellow), 29% above its target of 24. The National Role Advisory Board acknowledged that "this has been a very fruitful part of the project, with Fellows working on many different aspects of OER and holding links to a wide range of institutions and networks." {NARB minute 3.1.1} Subject disciplines have been mapped against the list produced for UKOER, and show a predominance of work in developing academic practice in a broad sense, filling in a few subject gaps and extending others.
Of the SCORE fellowship themes, 'Sustainability and impact' and 'Curriculum and learning design' dominated. Project themes were very varied, from generic level studies around policy, sustainable partnerships (with employers, cross sector, global), academic practice, including learning design and developing postgraduate researchers, user communities, to technology-focused developments (digital visual/interactive media, e-assessment, virtual experiments, reuse and adaptation of OERs). Some projects had a specific subject focus (for example open educational resources or practices in mathematics, statistics, arts, social sciences, biological sciences).
Outputs from the Fellows in relation to their SCORE Fellowship and their project were compiled on the SCORE website. The research, experience and networking opportunities acquired through the SCORE fellowships is seen to have informed further bids, discussions and collaborations across the sector {from NRAB/11/2/1: Appendix C: OVERVIEW OF SCORE TEACHING FELLOWSHIP OUTPUTS}. Examples include:
- "SCORE research informed the planning and development of the successful ALTO UK bid [UKOER phase 3], [in which SCORE will continue to] support early collaborative work with UK institutions, HEIs and Fes within the arts sector."
- "As a direct consequence of my fellowship I initiated a collaborative proposal for the JISC phase 3 OER strand, which was unsuccessful."
- "Establishment of collaboration with University of Hertfordshire (in parallel with a STEM funded project), promoting and developing effective virtual laboratory resources."
- "Meeting with PVC T&L Newcastle University (and Chair, PublishOER Board)."
- "As I direct result of this fellowship I was successful in getting funding of 5K to explore the theme of openness in research design as part of a larger project. In addition to this I have applied for an internal Excellence in Learning and Teaching award at [the ] University."
At the April 2012 SCORE fellows' monthly meeting, a show of hands indicated that the majority of those present had previously been involved in at least one phase of the UKOER programme, some in all three and some in support & guidance roles. Most were involved in other OER related national and/or international networks and communities. An interesting area to consider is how that cross over happens. Insights drawn from the social media bridges and inter-relationships (see twitter visuals below) might suggest that even where OER initiatives are separately funded/badged, this does not negatively impact the resultant communities within which individuals operate in the OER arena.
Short Term Fellowship Courses & SCORE hosted events
Evidence clearly shows that the weeklong residential courses (Short Term Fellowships) as well as the programme of national and regional events and annual conference, which have been designed and hosted by the SCORE team/community were very well received, as evidenced by the many benefits noted in event feedback responses. A large majority of attendees found the events very useful and informative; presentations and discussions "raised all the crucial issues" (such as staff incentives, IPR copyright, infrastructure necessary for successful OER), filled in gaps in knowledge, clarified many aspects and helped delegates to position themselves: "show that we are on the right track". Participants felt both the week long residential course and one day events were well organised. The content was generally relevant to their needs, speakers were motivational and the sessions offered a good balance of input from high quality presentations and time for questions and debate, illustrated by comments such as: "a great platform for sharing and discussion" and "I came away having found out lots, feeling inspired and having met some great contacts." Notable in terms of impact, delegates indicated that the knowledge and awareness of OER issues and practice gained from participating in the course or workshop(s) had informed and empowered them. Several delegates remarked that they felt they had developed sufficiently well to implement immediately an OER initiative within their own practice and, in some cases, influence thinking within their own institution.
A few remarked on low numbers as a limitation and a desire for more on practical issues of creating OER, including "practical/technical know-how", "how they [OERs] developed over time" and "how OERs might be re-used and repurposed", while others remarked more on the benefits of the events in terms of networking, variety of examples and learning from others, including examples from different subject areas, and international perspectives and a sense of the "global OER movement". There was significant interest in understanding the national context, including several delegates suggesting an input from HEFCE or JISC on national policies and future directions would be of value as well as guidance on approaches to evaluating impact. Some attendees felt also that they would have benefitted from more focus on institutional contexts and practicalities, as one delegate remarked: "more detail, less theory". Suggestions included "some input from senior staff to look at high-level issues" and a need to understand more clearly the "mechanics of the work flow", including key barriers, resource requirements, practical examples of legal agreements/waivers; and "strategies for how students can be involved in the OER development or reworking process". Others, however, remarked on the "emphasis on pragmatic(ness)" as the most valuable outcome of the event they attended: "All the ideas for where to look after the event, and a structure to understand where this fits.". Similarly, another delegate indicated she had created checklists for her own practice.
A total of 25 SCORE hosted events were delivered (around half within the OU and half outside the OU), including the establishment of a 'one event per month workshop programme'. Actual numbers were lower than expected, particularly given some were regionally targeted. However, attendance represents around twice the three year target for the number of HEIs having a physical attendance at a SCORE event {data from SCORE Dashboard and KPIs report, 11-01-2012}. Furthermore, statistics for online visitors (based on unique IP) show a strong appetite for remote attendance, both internally (99 visitors from within the OU) and externally (479 visitors). All events were recorded and are offered as OER, which supports the long term objective of engagement across the sector. Residential week-long courses (through Short Term Fellowships) proved popular with the number of participants exceeding SCORE's three year targets by 12%.
Sector engagement & dissemination
A high volume of wider awareness raising activity was undertaken throughout the sector, in the form of letters/emails to key target groups (e.g. Heads of eLearning/Library Services at all HEIs in the UK) & mailing lists; liaison, workshops, exhibitions & conferences in collaboration with national agenices (including JISC, SEDA, HE Academy, Leadership Foundation). Regular updating via the SCORE websites and Web 2.0 communications were evident: blog posts, tweets, wiki updates, slideshare, news feeds.
A key message from the documentary feedback from event participants (including STFs) is that balancing awareness raising, training needs and community engagement is essential for benefit realisation and impact of SCORE's key aims. Overall, both uptake and feedback from delegates suggest a need for clear distinctions in marketing events in terms of target audience, i.e. strategic institution wide level for managers etc and a practical level for teaching and support staff. For instance, those new to OER will require general awareness and some starting points for using or developing OERs, those in an institutional strategic management role, and those who are experienced in the field and benefit most from sharing approaches and experiences of open practices. By way of evidence, attendance at the session of "OER & Creative Commons" had double the number of delegates than any of the more general events, which may reflect its clear focus and specialist audience.
Collectively, there is evidence of a highly engaged and expert community of practice that networks across institutions, regionally, nationally and in most cases internationally, representing a vibrant UK presence in the field. This represents a significant achievement and impact from the SCORE initiative to date, as noted by the OU National Advisory Board {NARB/11/1/M }:
"... there is now a community of OER enthusiasts who weren’t there when SCORE started, and so there is now a need to support them. Support offered by SCORE would include events, regular updates and online support via the website, and continued support for research through the Teaching Fellowship scheme."
One indicator of SCORE's contribution to a UK wide OER community stems from its social media activity, such as Twitter networks. Some visualisations of the activity data have been produced for this review by Martin Hawksey of CETIS. The analysis was conducted based on data collected on the 1st May 2012 and show comparison and overlaps between the SCORE and UKOER Twitter following.
Each node represents a single follower, the size of the node being scaled; individuals most central in terms of community activity are identifiable by a much larger circle in the groups (figure above). Sheila MacNeill's post on this captures the value of such visualisations: she states "...being able to illustrate aspects of our betweenness centrality is increasingly important. Like others involved in innovation and community support, it is often difficult to qualify and quantify impact and reach, and we often have to rely on anecdotal evidence." By switching between the visuals, we can see the community bridging capacity of particular Twitter profiles (e.g. the 'BetweenCentrality' visual shows that @SCOREProject has three well connected intercommunity bubbles acting as bridges to the SCOREProject community. While the @UKOER following appears more interconnected (as represented in the 'In-Degree' visual), @SCOREProject may have greater reach given some followers have a higher 'Followers count'.
What is interesting here is the general inter-connectedness of the OER community. Such community structures, growth and cohesiveness are certainly worthy of further analysis (not just around twitter profiling). For instance, following up with individuals to learn about how much people have gained from this activity (perhaps through a little amount of tweeting they gain a lot, or perhaps the converse...) or about what people were interested in, the level of engagement in messages or discussion, and whether this brought about change in their thinking/practice. Community dynamics may then start to shed light on questions about viability of creating an association or network as a sustainable model for SCORE and other UK wide OER activity, as proposed by the OU's National Role Advisory Board {meeting minutes 20-05-2011/NARB/11/1/4 and 25-11-2011/NRAB/11/2/2}. This has yet to be explored with the community. For instance, to reduce costs and sustain an active UK OER community, the Board suggested such a network could potentially make use of members to undertake services such as running training sessions or reviewing proposals for the annual conference. It would also seek ways in which it could more directly benefit the OU and so potentially gain financial support from it. There is a proposal to create an OER Hub at the OU under consideration, and it was noted that SCORE could potentially take on the national element of this Hub.
SCORE project experiences, outcomes & impact
In the early review report for the SCORE project (May 2012), we presented some early thoughts on a partial set of documents. Final reports from SCORE fellows also include some early data analysis, but it is evident that impact from the work is still likely to gather pace only some time after the funding period. However, we have now attempted to map findings from the SCORE supported work against the UKOER synthesis framework (see above under Purpose, Scope & Methods). As with UKOER findings, the evidence is predominantly towards changes in 'Culture and Practice'. However, this report does consider issues around 'Releasing & Using OERs' and 'Processes for Sustainability'. The available evidence to date suggests that 'Benefits & Impact' are predominantly at the level of individuals participating in the SCORE Teaching Fellowship scheme, but there are strong early indicators of wider impact on colleagues, their institution, UK HE sector and beyond. Benefits were expressed particularly in terms of the project developments and/or research they were able to focus on, and also in terms of the immense value of the regular SCORE supported meetings, mentoring and community networking. For example, feedback in fellows' reports:
- "I have learnt that I can go further through collaboration with others than by relying on myself only."
- "Much networking has been required to enable this Fellowship to ‘take off’."
Changes in thinking & working practices
Fellows remarked on how the project experience had changed their personal way of working, for instance through:
- A broadening of their awareness and knowledge of the OER field: e.g. "SCORE has given me a good insight into the global picture, and role that governments play."
- Its value to academic practice and their own readiness to consider reusing OER for teaching alongside other (traditional) resources: e.g. "I am currently leading the e-learning component of a new 4 year nursing programme for 450 students. ... The possibility is to have a new major programme with a very high percentage of open content. This will be a very big change for us."
- Appreciation and understanding of the different issues involved, and links between publication and release: e.g. "Issues of accessibility are key to ensuring the success of the OER movement and to integrate OEP into the sector".
- More flexible working practices in how they organise the work, select tools, collaborate with colleagues: e.g. "The development of this resource reinforces the importance of having access to a network of colleagues/OER practitioners, as the work often involved consultation with colleagues’ expertise."
- Access to cascading levels of engagement: e.g. "Such involvement enables me to engage further with the political context of a regional, stereotypically research-oriented UK HE institution."
Several related themes emerged from SCORE fellow project outputs, regardless of discipline and teaching role of those involved. In one SCORE fellow’s final report, discussing the benefits of searching for OERs, one participant in the study expressed how "she enjoyed the “hunt” for other resources, and that it gave her ideas for what to incorporate into her teaching." Another fellow reflected on the "clear sense of pedagogical needs that arose among participants in this project" using OERs to help them accomplish some very specific learning goals, suggesting that academics had gone back to their disciplinary roots for answers: e.g. "Many participants took the pedagogical aims further in discussing the kinds of resources that might be useful, and retreated to the epistemological bases of their fields in discussing what they needed from OERs." In one case study, a SCORE fellow highlighted the importance of design and visual impact in students’ assessment of the value of learning materials: "Clearly this relates to wider issues about digital literacy and the need to transfer students’ critical thinking skills into the online space."
Barriers faced by individuals
Barriers that participants in SCORE fellow projects faced in attempting to find, evaluate, and use OERs were identified as "those about not being able to successfully complete the cycle of implementation or reuse of OERs in their own teaching practice", including technological barriers (such as incompatibility for accessing and downloading resources), lack of appropriate quality of, or meta-data for finding, resources (and the consequent drain on academics' time to source suitable OERs for reuse). VLEs were felt to "silo students into their different courses and don’t encourage sharing across courses, cohorts or with the wider public". One case study was particularly insightful about the implications of barriers in terms of take up of OERs:
"The barriers faced by these colleagues are not driven by lack of desire to use OERs, or indeed, as part of this project, they are not barriers to do with having support or guidance in approaching OER use in their own practice. They are legitimate concerns and documented barriers from academics who would like to use to OERs as a way of innovating in their practice. If these highly motivated participants have difficulties bringing OERs into their practice, there is the potential that such difficulties might be magnified in more mainstream academics."
Impact of individuals on others
The evidence supports the claim that OER initiatives such as SCORE and UKOER are impacting positively on practitioners' thinking about their own practice and that this cascades into impact on other colleagues. One nursing academic participating in a SCORE fellow's project described "finding an OER that had a powerful impact on her work, leaving her reflective about her own practice as a nurse, as well as reflective about her teaching." {from SCORE fellow project final report}.
One SCORE fellow remarked: "In the course of this fellowship I have fundamentally changed my practice in an open and sustainable way, and in doing so I have effected change with my colleagues beyond the usual dissemination."
Evidence from final reports and case studies also strongly suggests that because of the research-based nature of SCORE projects, fellows are natural 'critical ambassadors' for OER both internally and externally. The majority of SCORE fellows had a high level of dissemination activity planned, with the intention of releasing reports, articles, resources, staff development materials and various other outputs, using available networks and special interest groups to transfer knowledge, embedding materials and tools, further collaborations, and promoting their work locally, nationally and internationally through conferences and social media.
Some SCORE fellows had had opportunities to consider the same issues framed in different countries' contexts and, as one reported, they found they were faced with the same questions: those of "affordability, infrastructure, and cultural barriers or demands". Some SCORE fellows reported difficulties in recruiting participants for their project, and highlighted both negative and positive responses, for example: "the tension in terms of using a “competitor’s” material in your own “product”, which is problematic in a market economy" versus "By aligning our support to ideals of outreach, public understanding, impact and giving we attracted colleagues whose own academic ethos and identity fit with the values of OER."
Institutional impact was being evidenced by several SCORE fellows. In a final report, one fellow commented on her increased expertise in this area resulting in greater involvement in organisational development: "The outcomes of the project could have a further effect on the institution strategy or policy on OER. For example, the University is now considering making OER widely available through different channels, including iTunes U." Another remarked "Being a SCORE fellow and studying this topic has opened doors for dissemination and cooperation." Several others indicated increased opportunities their SCORE supported work had provided in terms of thinking about their individual practice in organisational terms:
"This project gave me time to promote and engage with ‘joined up thinking’ about OER across the board beyond that which I might have normally managed to do."
"As a professional learning technologist I have always seen OER projects as learning technology initiatives. The skills are the same and the drivers very similar. The key to learning technology success has always been in matching the technology to the task and the activity to the institution."
There is general recognition that without educators buy-in, an OER culture will not become embedded within the academic practices of institutions. These shifts in thinking and practice beyond their own individual teaching practices, in sharing and debating, are crucial in supporting changes in culture and acceptance of OER in the HE sector. As highlighted in the conclusion of a paper by one SCORE fellow and UKOER pilot project leader:
"Even in a supportive policy environment, in order to translate OER from mere 'stuff' to something much more useful, we need a sensitisation programme for both academic and also professional support staff and a radical view of how our learning and teaching practices need to be modified to embrace OER. At [our university], we are just beginning to build awareness of OER into our HEA accredited staff development programmes. This approach lies at the heart of our view on sustainability."
Sustaining practice
Most SCORE fellows felt they were in a strong position to secure further funding to continue their research and/or OER developments beyond the SCORE funding period. E.g. "This project's work towards a model for sustainable use of OERs is attracting significant academic interest." Many are already being invited to lead on initiatives or areas of provision in their institutions or externally, to act as critical friends, give key presentations, and other indicators of esteem. Beyond education, where OER are at least partly established, one SCORE fellow working in the public, voluntary sector, noted that OER are unfamiliar territory, remarking "[OERs are] very poorly understood outside universities, so a lot of time is spent explaining what OERs are and why."
In their feedback on fellows' projects, SCORE mentors also remarked on the wider impacts and support of the SCORE work. For instance:
"This project has impacted the sector of interprofessional education much wider than in just HE."
"There is a lot of support for this project and it will sustain activity beyond the end of the SCORE funding."
"This impact is not just at a national level but international ...This shows great support for the project outside of the fellowship."
Key ‘take home’ messages highlighted by delegates at OER 2012, an annual conference hosted for the second time by the SCORE project, indicate that the OER movement is international and forging ahead. The SCORE final report to its National Advisory Board noted that "questions and issues are keeping pace with the evolving nature of the [OER] movement. This means a constant cycle of knowledge gathering, development and justification." Comments from the delegates included:
“OER is here to stay, it needs to be supported.”
“More people and institutions are active in OER than I thought.”
“Educators are trying to bridge gaps and make information more freely available.”
“To mix metaphors: we've passed the inflection point; we've passed escape velocity; the movement cannot be stopped.”
CONCLUSIONS
These findings situate the SCORE activities within the broader HEFCE OER Review. Various outcomes and outputs were reviewed and mapped against the Review Framework. Retrospective work was collated and certain stakeholder groups from both SCORE and UKOER initiatives were consulted through a variety of mechanisms to provide an in depth and reflective view of activities and outcomes across the three year period. Two surveys were conducted during summer 2102 and results are discussed and included as an appendix to the final HEFCE Review Report:
- The first was a short 'poll' of five questions distributed widely across professional contacts, HE/FE mailing lists and social networks, which aimed to to snapshot cross-sector awareness of HEFCE's OER initiatives (namely the three phases of UKOER and SCORE).
- The second was a longer survey circulated specifically to individuals directly participating or indirectly involved in the HEFCE funded UKOER programme and OU's SCORE work.
A selection of 10 survey respondents were invited to participate in interviews and care was taken to ensure the sample (i) incorporated both UKOER (8) and SCORE (4) directly funded staff; (ii) senior staff more indirectly involved (2 individuals in institutional management or mentoring roles); and (iii) did not favour only the champions or enthusiasts of OER. This aimed to balance and triangulate the data as far as possible within the scope of the review.
Interviews were semi-structured and built on respondent’s survey responses. Respondents were asked to describe individual or institutional journeys, outlining factors that influenced the evolution of practice. A central principle is that evolution is far from linear and continual - periodic critical events determine the direction of travel. Interviews enabled respondents to recount their story to build up a picture of what has changed since the initial UKOER pilot programme in 2009 (e.g. how they have moved from where they were to where they are now). Results are discussed in an appendix to the final HEFCE Review Report
We have found there to be significant cross-over between the various UKOER and SCORE communities and activities and final reports and documents from this review will reflect an holistic picture of the benefits of engaging with open practice and OERs across several sectors in the UK, including various educational sectors, NHS and government sectors, 3rd sector, public sector, and private sector (including publishers).
The professional learning opportunities afforded by the SCORE models of support (fellowships, residential courses, workshops) are particularly relevant in relation to evidence of the impact of individual teacher's expectations of e-learning on learners' approaches to learning (recent JISC e-learning programme work; Sharpe, 2010; Margaryna & Littlejohn, 2011; McAndrew,2011).
Meta- trends identified through, and contributed by, SCORE activites include:
- Focus shift from OER production to use of OER in professional practice (teaching and learning practice);
- Shift in emphasis of OER production from producer-led to consumer-led;
- Shift in discussion of OER from production/practice issues to consideration of meta-level factors such as epistemic approaches and beliefs.
Overall synthesis activities indicate that HEFCE funding in this area has had significant impact on culture and educational practice, sector awareness and understanding of OER release and use, and on defining the benefits of various institutional, community and individual models. Key questions remain as to who is using/reusing OER and, more importantly how, (see also JISC funded study in 2010-2011 on the Impact of OER) but both SCORE and UKOER activities are clearly addressing issues around:
This review of SCORE documentation and formal reports has revealed some interesting indicators of changes in institutional culture and practice, which links strongly to similar findings from the UKOER Programme. In particular, we note the following:
- treating OER as a sub-set of existing strategies (such as curriculum innovation, e-learning, digital literacy, professional development 'good practice', quality enhancement initiatives);
- increasing the volume and culture of sharing and reuse (with the effect of course teams gaining visibility on the web, collating digital resources as one mechanism for collaboration);
- blending in assessed work (as a means of promoting quality & trust and countering plagiarism);
- expanding the boundaries of study skills (e.g. by moving beyond text skills, citing non-text based resources).
- a move from narrow activity in the production/use of OERS by enthusiasts to broader production/use/release of OERs as an aspect of academic practice. SCORE Fellows could help in the push towards broadening out use if they are encouraged to take a more strategic role in this regard. Losing the momentum of SCORE Fellowships at this critical time could be detrimental to the OER movement within the UK and could devalue the HEFCE investment to date. This is also mirrored by a need to take forward UKOER Project Team expertise within institutions and to the wider community.
- the polarisation of those who see value in the production/use of OERs and those who don't. This trend is related to the priorities of academics and institutions which has shifted significantly since 2010. The decoupling of teaching/research is likely to exacerbate this trend - there is a tangible acceptance of the value of open access in the research communities, which is not evident to the same extend in teaching communities, despite the potential benefits in both domains. The third phase of the UKOER programme found significant merit in linking OER to research outputs as a way of increasing engagement and contributing to the notion of open academic practice.
Further information and existing synthesis documents can be found at:
Briefing papers are available at:
References:
McAndrew, P. (2011) Inspiring Creativity in Organisations, Teachers and Learners through Open Educational Resources. European Journal of Open, Distance & E-Learning. Retrieved 13th May 2012 from http://www.eurodl.org/?article=458
Sharpe, R. (2010) Conceptualizing differences in learners' experiences of e-learning: a review of contextual models: Report of the Higher Education Academy Learner Difference (HEALD) synthesis project. Retrieved 13th May 2012 from http://www.heacademy.ac.uk/assets/EvidenceNet/HEALD_Report.doc
Margaryan, A., Littlejohn, A. & Vojt. G. (2011). Are digital natives a myth or reality? University students' use of digital technologies. Computers and Education, 56(2), 429-440.
Appendix A: Revised SCORE targets and KPIs
|
SCORE Activity
|
Target
|
Achieved
|
Comment
|
|
Teaching fellowships
|
24 fellowships
|
14 fellowships have been completed
|
17 more are in progress and due to complete by July 2012
|
|
Short-term fellowships
|
50 fellowships
|
58 fellowships
|
March 2012 was the 5th face-to-face presentation of this course. Development of an online course is in progress.
|
|
Dissemination events/workshops
|
20 events – the majority also recorded and published as OER
|
20 events
|
1 event is still to take place plus an exhibition stand at the HE Academy conference
|
|
Media training
|
Additional activity – no target set
|
15 training days, 100 participants
|
Media training has been moved to the OU eSTEEM Centre
|
|
Conference
|
2 SCORE annual conferences
|
2 annual conferences
|
OER 11, May 2011, Manchester – 130 delegates from 63 institutions
OER 12 combining with the OCWC annual conference as “Cambridge 2012”, April 2012 – 236 delegates from 136 institutions.
|
|
General OER publishing
|
1440 hours from the OU and 600 from other sources
|
Target exceeded: 1463 hours from OU and approx. 960 from other sources
|
This activity ended July 2011 with resources reassigned to the Consortium OER activity
|
|
Consortium OER activity
|
“Ready for Research” and “Digital Scholarship” micro-sites
|
115 hours of OER sourced for Ready2Research and a further 27 for Digital Scholarship. Test platform usability testing complete.
|
On track for go live 2 July 2012. Amendments made based on results of usability testing. End May there will be a populated site for further testing.
|
Comments (0)
You don't have permission to comment on this page.