Back to: UKOER/SCORE Review Final Report

Appendix 1. Analysis of Surveys

To connect with the OER community and wider sector perspectives we designed two online questionnaires during July 2012.

• The first was a short 'poll' of five questions distributed widely across professional contacts, HE/FE mailing lists and social networks, which aimed to to snapshot cross-sector awareness of HEFCE's OER initiatives (namely the three phases of UKOER and SCORE).  
• The second was a longer survey circulated specifically to individuals directly participating or indirectly involved in the HEFCE funded UKOER programme and OU's SCORE work.

 

Survey questions explored the way open educational practices are being perceived in terms of who OER are being produced for, as a key indicator of potential longer term impact. For instance, questions were intended to identify changes in attitudes towards risk that can lead to a greater and more widespread release and use of OER by teaching staff, enhanced student centred approaches, as well as the role of OER in marketing, leading to reputational gains for the institution. Central to supporting the OER community is the online presence of both UKOER and SCORE participants. Some data concerning activity and usage via the web, blog, social media, has been reviewed, as this offers valuable indicators concerning sector engagement. What is interesting here is the general inter-connectedness of the OER community. Such community structures, growth and cohesiveness are central to the success of the HEFCE OER funded initiatives.

 

However, for those not necessarily involved or funded directly, we recognise that different terminologies, contexts and associations may apply. It is important that these are identified and incorporated into our cumulative knowledge and understanding. We therefore have welcomed and encouraged wide engagement with a short, cross-sector 'crowdvoting' type online poll that aims to introduce a broader set of opinions and judgment - "wisdom of the crowd" - against which we can overlay our deeper analysis of open practices from individuals within the OER community.

 

The wider 'crowdvoting' poll elicited 129 responses, predominantly from the UK (58%) but also a fair spread of international engagement mostly from English speaking countries. 96% were working within an organisation/institution and most were working in the HE sector: US (7%), New Zealand (6%), Australia (4.5%), India (4.5%), Canada (4%), South Africa (3%) and one or more responses from 12 other countries. Some differences in countries represented by respondents working in the HE sector. Those working in school education were largely UK and India; other sectors included further/vocational education (predominantly UK), public sector (UK & US only), private sector (UK, India, Brazil, Netherlands & South Africa), charity/voluntary sector (entirely UK & US).

 

The detailed survey with the OER community elicited 50 full responses, 98% of which were HE, which is unsurprising given the HEFCE funding was HE focused. Respondents represented a spread of roles; the majority were teachers/tutors (52%) yet with a fair percentage indicating their role (solely or additionally) as managers (22%), pedagogic support (26%), researchers (28%) and OER initiate support project (26%). 10% indicated a technical support, 10% librarian/information worker, few or none in marketing or admin, and 2% were students. Responses represented people involved in all initiatives as individuals (UKOER projects and/or SCORE fellowships, residential course or workshop attendees) or as part of an institutional project (as in UKOER projects) or subject community (UKOER pilot and phase 2).

 

The survey attracted a good mix of people involved across both initiatives, as individuals (predominantly SCORE), support services, lead institutions (only UKOER), partner institutions and/or user/recipients. Only UKOER phase 1 (pilot) and 2 indicated subject community involvement. UKOER phase 3 involvement was predominantly as 'lead institution' (50%). These profiles mirror the focus/nature of the UKOER programme phases and SCORE opportunities (personal fellowship projects, residential courses & workshops).

 

Quantitative statistics of responses presents a valuable picture of current perspectives and evidence for awareness, priorities and outcomes in some areas.  It is anticipated that qualitative data will yield the deeper understanding arising from 'softer' indicators of benefits and impact expressed through the narratives of personal OER journeys. See also the interview analysis

---

Awareness, engagement & participation

Themes : UK to international interest, individual champions, impact on others

The OER survey, distributed to UKOER/SCORE related communities and networks specifically, indicates the importance of the funded work to raise awareness, link and empower individuals and communities of developers and users (both students and staff as "Open Education Practitioners"), and galvanise the institution (see also the section below on benefits/impact). When asked how far they felt people in their institution/organisation are aware of HEFCE funded OER initiatives (outside the project team members), responses indicate that this is still predominantly limited to pockets within institutions:

• awareness in pockets (67%)
• very low awareness (27%)
• generally wide-scale awareness (6%).

 

Although a difficult question to ask or answer, as it relies heavily on a personal 'sense' of an institution-wide phenomenon, it nevertheless reveals that even with all the funding and the significant institution-wide engagement, the awareness is still fairly limited. Not least this indicates that there is still a long way to go in terms of awareness and maturity of organisation level OER, and open practices more generally. With regards to OER specifically, there is a danger within the community of a sense of moving beyond OER onto other higher profile and 'buzzwordy' initiatives, such as MOOC. However, the majority of institutions are actually only part-way along the OER journey, despite all these activities and funding. As a sector, therefore, there is a need to consider how this work can best be consolidated and how  institutions might be encouraged to continue to move forward with 'open' developments in a climate of financial constraints.

 

However, this level of awareness, engagement and participation remains potentially an inner circle. Respondents (129) in the wider sector poll mostly indicated they had had no or some engagement with either the OER infoKit, UKOER projects, SCORE supported activities or some other international OER related initiatives (namely OERu, University of the People & EdX). (Engagement was defined as involvement in a project, using a service or resources, attending events, supporting roles, collaborative roles.) The relatively high response rate from those engaged in OERu was the result of an email posting on the OERu discussion list encouraging these respondents. Analysis of the 38 "other" responses threw up MOOCs (5), OPAL (3), Coursera (3), DS106 (3), learnhigher (2), OERtest (2), OER Africa (2), OpenLearn (2), Creative Commons (2). A minority indicated directly funded engagement or their intention to engage. 

 

The wider sector poll data shows good awareness and engagement among the international OER community of UK initiatives. As anticipated, a high proportion of the 56 non-UK responses come from those involved in OERu (12% indicated directly funded engagement and 27% some engagement), although there was also a significant engagement with the OER infoKit (20%), UKOER projects (14%) and SCORE fellowships/events (9%). Comparing UK respondents with non-UK for overall responses, we can infer that SCORE's impact is felt more strongly within the UK. 15% of respondents had no engagement with any OER initiatives, either those named or "other". This shows that the survey achieved, to some extent, its aim of reaching beyond the OER community.

 

SCORE and UKOER had different aims and models of engagement. Both appear to have achieved their aims, but the differences in engagement models show up with SCORE showing a much higher ratio of "some engagement" to "direct funding" than UKOER, but UKOER showing greater absolute numbers of both "some engagement" and "direct funding".  The Infokit shows both better ratio of "some engagement" to "direct funding" and greater absolute levels of "some engagement" than either UKOER or SCORE.

 

Inter-connectedness of the OER community

Themes : networking, the role of social media, influence of/on subject communities, collaboration/sharing

 

The majority of OER survey respondents indicated their primary sphere of practice to be as a member of an educational institution (81%), but there was also crossover with both wider and specialised communities and networks. Research and subject disciplinary communities also featured largely (47% & 39% respectively) as well as reasonable representation across international networks (27%), professional/learned communities (29%) and social media networks 36%).

 

 

Subject disciplines of respondents were predominantly education (52%) and languages (32%), but included both sciences and humanities as well as generic skills. Education and CPD are likely high due to OMAC strands of work, but also a focus of much teacher education on OER production. Many other categories, such as research skills, information literacy etc. might legitimately be included in 'generic skills' resources, which may have further dominated the spread. In previous reports, we have shown that generic OERs (i.e. those that cross disciplines) present attractive options for institutions in terms of efficiencies of sharing generic materials across faculties. Furthermore, their low perceived value may appear low risk and 'an easy win' in terms of the perceived requirement for high quality in subject content. These subjects therefore, unsurprisingly, featured strongly.

 

It is interesting to note that from the whole list of subject areas, not one subject was left blank, suggesting OER was covered in all subjects. A text analysis of responses concerning the perceived impact of respondents own OER work within subject disciplinary community of practice revealed 'awareness' as the most important term (20%), alongside 'practice' (14%), 'raised the profile' (8%) and 'students' (8%). In the free comments, there was some evidence that the UKOER/SCORE activities had introduced new ways of teaching in the disciplines, particular sub-disciplines (e.g. media, photography), with an undercurrent of repurposing ideas or producing bespoke versions. Although there remain concerns and challenges, some felt OER was gaining acceptance and influencing course development processes with greater sharing across traditional boundaries. "Sustainable models for OER as a service" was also mentioned by at least one respondent with several indicating that other OER initiatives and mentoring schemes within their organisation had developed as a result of the HEFCE funded activity.

 

Open educational practices & culture 

Themes : organisational positions, OER release/use/re-use, attitude to risk/competition

 

Open educational practices (OEP) as users/learners appears more prevalent than OEP as producers, even among the more OER-community respondents to the wider poll.  The most common practice as a producer is "I consider myself to be an open scholar (making content openly available and collaborating openly to further research)".  It might therefore seem surprising, and says a great deal about prevailing hierarchical culture in HE of teachers vs students/learners, that by far the smallest category (30%) is "I design courses where learners contribute to public knowledge resources". (We note, however, that one of the open comments raised ethical concerns about expecting fee-paying students to contribute their outputs for free.) The responses may stem from how the question is asked (and in a future iteration of the poll, we might include an option for "I encourage learners to use OER" in order to get a better handle on this observation).

 

600

Comparison with the segment of 19 respondents not engaged in OER initiatives, as might be expected, adoption of OEP is generally lower than for respondents overall. The profile of practices also differs, showing a much more marked prevalence for OEP as users rather than producers for those not engaged in OER initiatives than for the respondents overall.

 

For those who have received SCORE or UKOER funding (24 respondents) (see chart below), the profile is markedly different.  Interestingly, use of OER ("I use open content freely available on the web for my learning") is lower than for respondents as a whole, as is participation in open courses (slightly). Production of OER appears to be higher than any usage of OER, as might be expected since much of the UKOER funding was for OER release. It occupies first spot along with use of open source software.  Along with production of OER, facilitation of open courses is also significantly higher for this group than for respondents as a whole. The practice "I share ideas and examples of my teaching practice on the open web" shows a slightly lower rate than for respondents as a whole, in contrast to the greatly increased response for releasing material as OER.  This may indicate an increased awareness among those engaged in this funding stream of the importance of licensing, i.e. that they are formally releasing as OER rather than informally making materials available on the web.

 

Comparison of these findings with that for respondents as a whole, or even more with those not engaged at all, shows the impact that HEFCE funding has had in shifting practice towards staff as open producers and facilitators rather than simply as consumers. However, this shift may be one that actually entrenches existing cultural practice, as implied by one of the free text comment: "There is a major shift in mindset to be overcome if more educators and institutions are to engage in OEP and with OER. The sense that 'we are what we produce' continues to prevail in terms of educational materials and resources, and until we can disassociate 'the material we produce' from notions of our perceived worth as educators there will continue to be fear of OEP and OER and the implications for reputation, job security, and what the role of an educator should be". The implication of such a view is that use of open materials marks a greater shift towards open practices than is releasing open materials.

 

However, design of courses where learners contribute to public knowledge resources remains low and is, indeed, slightly lower than for respondents as a whole. It is debatable, at this stage, whether this finding highlights a facet of UK culture and the perception of learners/students in the UK. Comparison of non-UK and UK does indeed show a marked difference in the practice of designing courses where learners contribute to public knowledge resources.  Rates for those involved in UKOER/SCORE/OERinfoKit are slightly higher than for UK respondents as a whole suggesting that there might be some shift in this practice also towards production. Comparison of non-UK and UK also suggests that rates of adoption of OEP outside the UK are higher than within the UK, at least for respondents to this survey.  The general shape of the profile remains similar, however, and as noted above, is in marked contrast with that of those receiving funding from UKOER/ SCORE/ OERinfoKit.  However, this data may be skewed by the high proportion of the 'non-engaged' who come from the UK (15 out of 19). Excluding those not engaged at any level in OER from the UK sample still leaves UK engagement in OER looking lower generally than that internationally, especially as users of OER.

 

The picture of institutions/organisations' position in relation to OER release, use, open educational practice (OEP) and open licensing reflect more the focus of the funding. What is interesting to see is that OEP seems to be fairly well established in both documentation and institutional commitment, even though we only really started talking about open 'practices' last year. This therefore appears as a real move forward from thinking specifically about content (OER), although it may simply be a term picked up by the community, because of involvement with OER funded programmes using it. However, there is evidence also of growing interest now in open assessment and open courses. This momentum is also evident in the number of institutions currently investigating open courses as a result of high profile MOOCs (Massive Online Open Courses).

 

Overall, the survey responses offer a very positive picture of engagement from the institutions represented. It would be very interesting to see just how institutions that have not been involved in funded initiatives compare. Nevertheless, our sense is that the data shows some real impact of HEFCE's funding in OER, particularly in relation to institutional documentation changes and developments.

 

Legitimate barriers in creating & using OER

Themes : the cycle of implementation, access, co-creation

 

In the wider sector poll, similar issues appear to dominate both individual and institutional barriers, notably lack of awareness of OER and their benefits, lack of coherent overall institutional educational strategy, and (by inference) lack of supportive workload planning. Legal concerns, although high, are not as high as might be expected and are not one of the major barriers, especially at institutional level (which may indicate a lack of institutional awareness of OER issues).  Concerns are relatively higher (compared to lack of institutional strategy and support) for those working in the FE/vocational sector.

 

 

 

Similarly, quality issues are not a major concern either for institutions or individuals (less important than legal issues). However, the evidence from poll responses indicates that insufficient digital literacy is a major barrier for individuals, coming just below the institutional factors mentioned above.  It leaps way into first place for those working in the school sector, and is also just first for those not based in the UK. Looking at USA and New Zealand (the other two countries with a significant number of respondents - 9 and 8 respectively), shows interesting differences, especially for individuals, with "does not fit with existing work practices of staff" becoming the major barrier in NZ and equalling lack of institutional support/strategy in USA. This suggests that, as might be expected, the barriers depend on the national character and practice of HE.

 

Perceptions of barriers, both individual and institutional, are similar among respondents who have received HEFCE funding (UKOER & SCORE) as among the overall respondents. Interestingly, perceptions of barriers among those with no engagement with OER initiatives appear similar to perceptions among the overall respondents - the same barriers in top positions, led, as might be expected, by "lack of awareness". However, for this group, technical challenges appears as an institutional barrier that is as important as lack of strategy and institutional awareness of OER. One conclusion would be that in the UK much more work needs to be done with institutional senior managers, and on raising the digital literacy of individuals.

 

In our detailed survey with the direct OER community involved in UKOER & SCORE funded activities, unsurprisingly, barriers that featured highly were:

• time to adapt and repurpose
• legal aspects and licensing, and
• OEP not fitting with current work practices of staff.

 

As with the wider sector poll, we might have expected to see more responses around quality - although in the survey, this was split this into (i) concerns about quality of OERs out there to use and (ii) concerns about their own materials being released as OER. Quality was a significant issue during pilot phase of UKOER and this may indicate that increased experience with OER allays some of those concerns. Nevertheless, a significant number still indicate that it is difficult to locate relevant OER, indicating either a potential lack of resources or issues around discoverability (due to limited meta-data, time and/or skills). Lack of digital literacies does feature highly as a barrier too, although our first few interviews suggest that finding/evaluating quality OER is a time issue not a skill issue and likewise many staff avoid releasing OERs due to the time involved in making them sufficiently polished (reputational concern) and fully compliant (legal concerns).

 

 

There remains a view that lack of institutional support, strategy or investment is a barrier. It is important to note that these respondents may be responding with an overview of how barriers have affected staff in their institutions as many of them have had to expend considerable energy raising awareness, and supporting staff to engage and change practice. UKOER project teams are very aware of the barriers and have been most articulate about this throughout the programme. 

 

When looking at this question in more detail, it is interesting to see that respondents involved in the pilot phase of UKOER identified the same top three barriers equally. Phase two respondents however had legal aspects and time factors as equally high, but awareness of benefits much lower. This perhaps reveals that phase two work built very much on the pilot phase work and some of the work around raising awareness of benefits had started to be successful. SCORE (fellowship, workshop and residential) respondents reflected the same top three as the overall group. UKOER phase three, on the other hand, broadened activities to other sectors and they report 'lack of awareness of benefits' as their top barrier by a significant amount. 'Time to adapt & repurpose' was replaced in the top three by 'lack of institutional investment'. The latter may reflect increasing financial constraints beginning to impact on work outside what is perceived as core work.

 

Involvement of students & impact on learning

Themes: types of student partnership, types of learners

 

The OER survey specifically asked about the capacity in which students had been involved in the OER initiative. The responses revealed the wide range of ways that students had been (and are) participating.  Currently, there is evidence of students' involvement at multiple points: creating OERs (22%), evaluating OERs (41%) or simply as recipients of OERs (32%). A smaller set of responses indicated students were part of the project team (16%), students as researchers (11%) and/or students on open (or partially open) courses (9%). However, nearly 30% of respondents answered N/A, indicating that students had had no involvement in their OER activities.

 

 

When asked about impact on students/learners, many respondents indicated it was "too soon to tell". However, the long set of open comments provide many early indicators of a positive response to, and interest in, OER, garnered anecdotally/informally from feedback in teaching/class, student support and discussions with colleagues as well as more formal student focus groups and surveys. Evidence suggests students/learners are gaining confidence through their engagement with OER, including greater confidence in their learning, higher use of online 'open' resources (e.g. YouTube views/followers), enhanced student projects, collaborations and shared initiatives, including internationally, such as blogging and OER editing/production. Evidence of students' positive engagement with OER included emails to tutors about their usefulness.

 

This positive impact is equally true of staff as learners on professional development programmes, with impact on staff awareness both of OER licensing and copyright issues and of the benefits of sharing. This has an indirect impact on students, as improving the practices of teachers also indirectly impacts the experiences of their students. In fact, the majority of user evaluations mentioned by OER survey respondents were with staff within their institution/organisation (55%), but also students formally enrolled on a course (40%); some were people outside the institution (36%), informal learners (21%) and/or general users (19%). However, only a low proportion (22%) indicated they had carried out any follow-up evaluations of the use of OER released as a result of funding after the funded period, but comments suggest in many cases that evaluations are just getting underway.

 

Motivation & benefits

Themes : professional development, pedagogy, sharing/cost-benefits

Our detailed survey with the directly supported OER community indicated a range of motivations for being involved in UKOER and/or SCORE initiatives. However, the predominant response (78%) was that of being "experimental' - to investigate the potential of OER". This is not surprising given HEFCE funding was aimed to investigate the potential of OER release. However, coupled with other motivations such as building on previous OER work by individuals, departments & institutions, and availability of funding, the finding is significant since it recognise the value of HEFCE funding in this area.

When asked about their top 3 perceived benefits of releasing and encouraging use of OERs, survey respondents predominantly indicated:
1. increased access for learners (55%)
2. enhanced pedagogy (49%)
3. increased sharing between educators in the same discipline (41%).

 

 

There is little comparative data as to whether these potential benefits (in hindsight) were perceived in the same way before the initiatives ran. However, UKOER participants have reported that OERs present an ideal opportunity to enhance pedagogy and these results really confirm that previous finding. This is a very strong argument to support engagement with OER release and use. While it is interesting to find increased access for learners and sharing amongst educators standing out as significant as an overarching benefit (and important motivation for releasing OER), there is actually very little evidence that learners are using the OER or educators are in fact sharing (and using/repurposing) each others' OER. The first two phases of UKOER had significant involvement of subject communities and this is most likely to have really supported this kind of sharing.

 

Others indicated motivation around the potential benefits furthering OER infrastructure (24%), cross-sector sharing (24%) and institutional reputation gains (22%); with a lower proportion indicating supporting disadvantaged learners (20%), increased efficiencies (14%) and student recruitment (14%). A very low percentage suggested increased personal reputation as a benefit. However, the majority of respondents reported personal benefits in terms of professional development, with increased understanding of issues around licensing featuring strongly. As we evidenced in phase 2 (and reported in the Open Practices Across Sectors Briefing Paper), many respondents reported as benefits "an increased awareness of wider sectoral approaches and increased collaboration across sectors", although UKOER phase 3 respondents reported this most significantly - which reflects the cross-sector focus of activities.

Impact & sustainability

Themes : sustained institutional policies & new strategies, support/collaboration

When asked about sustainable development, 55% of respondents indicated that their institution/organisation had continued support of the OER initiative and vision since project funding ended, 8.5% had not. For the other 36%, the project had not yet ended.  This is also reflected in phase 1 and 2 reports - although a lot of these have had ongoing funding from the programme. Overall, this does appear to indicate effective sustainability approaches in the majority of cases, with some exceptions where institutions had not (yet) taken OER developments or strategies further. However, examples from respondents included:

 

• Financial support - e.g. £10k "to help the institution to share understanding of high quality approaches to ensuring that the institution complies with legal good practice".

 

• Development/technical support - e.g. "to implement recommendations" arising from the project, such as staff time allocated, regular monitoring and evaluation of key sources of OER in our specific areas of interest, "The fact that the repository now exists and has replaced a myriad of smaller schemes and (less open) ways of distributing resources has been welcomed widely."

 

• Partnership support  e.g. "supporting opportunities to apply fo further funding ... to develop bespoke versions" for specific purposes, work allied to other ongoing initiatives in the institution.

 

• Embedding support e.g. "work to continue to explore, support and exploit the potential offered by OERs is included within the strategic objectives", "systems for supporting teachers and providing teaching materials for reuse", "Contributions form tutors have been coming in, and new collaborative practices emerge, but the time-factor and worries over copyright are probably the main limiting factor."

With regards to the wider sustainable outcomes, some of the open comments from our wider sector poll suggested a need for better and more strategic coordination of open practice initiatives, beyond the local HE institutional level, if HEFCE's intervention in this area is intended to have a global impact. Key drivers mentioned were the need for business models for institutions, senior management buy-in and development of assessment and accreditation that align with open practices by academic staff.  However, the OER survey and interview data (collected so far) reveal that by far the greatest driver for sustained activity in this area has been the enthusiasm of staff. If this is the key aspect then sustainability may be precarious if staff leave or enthusiasm wanes or shifts focus. This answer, and findings from our earlier SCORE review, tends to imply the notion of 'champions' who will take the ideas and awareness forward. 

 
As expected, respondents also identify senior management buy-in as important, mirroring findings from the pilot UKOER project. UKOER phase 3 had higher responses than other two phases with regard to increased awareness at institutional/organisational level. This is expected as some institutions had been through one or more iterations of UKOER by then. SCORE respondents show very similar pattern to UKOER 3 choices - increased awareness and enthusiasm of staff being very important across both groups.

The main factors hindering sustainability in OEP/OER appears to be the desire and time to first put in place good compliance with high quality procedures in relation to staff development, clearing copyright and licensing consent. Some technical issues remain, but appear less prominent in terms of slow progress. The use of both local OER repositories and systems and national services, like Jorum is growing, yet there appear to remain challenges with finding relevant OERs.

Evidence for involvement in the HEFCE funded UKOER/SCORE work having an impact on institutions/organisations, overall, was very encouraging. A significant percentage of responses indicated increased institution-wide awareness, changes in culture and practice, collaboration (both internally and with external partners & wider sectors). There were also some excellent reports of new and adapted strategies and policies. While a small percentage reported little or patchy impact across the institution, the findings are a significant indicator for the most part of institution-wide change and commitment (impacting on sustainability of funded activities discussed above).

 

Some more poignant comments around sustainability of OER release from OER survey respondents include:

"I believe that the direction of travel is permanent, we will not go back from here, and the institution isn't anti-OER, just slow to respond. However it is responding as an organisation (not as a cottage industry) and my group has been commissioned for £10K to work with staff and schools (departments) to educate them in using OER from elsewhere and having a high standard of copyright compliance. It is a first step."

 "[over the past few years]... we have sustained a gradually increasing portfolio. In 2012-13 we intend to expand the number of Open classes further within the department and to offer a model for their development [to others]."

"[Sustainability] is both a question of technical digital literacy skills and understanding the specificities/ and philosophy of OER approaches .... [it's the] time input [for staff development and] to develop high quality OERs, lack of recognition/reward for doing so, [as well as] some technical and licensing legal issues." (mix of responses collated)

Indeed, this reflects what projects report across all phases, strongly suggestive that policy change and strategic buy-in is a very important part of the institutional journey (and perhaps 'OER/OEP maturity') with clear evidence of institutions commitment to continuing it at an organisational level.

 

Date September 2012