Summary

The closeness of the resource authors to the resources in the individual projects was evidenced in a lack of articulation of heavy initial quality control before release. Several projects state clearly that the materials released are the same as those used in teachiing within the institution, and one (ChemistryFM) argues explicitly that if OER is to be integrated into normal course development, then the normal quality processes should be used. A further point is that if, as in many of the individual projects, the general public informal learner is among the target audience, then at least a limited degree of initial release is necessary before user testing with this audience can take place.

After initial release, quality processes may be deliberately initiated one-off or cyclical events, such as a user survey or annual course reports; or they may be an ongoing part of community involvement in the OERs. A high proportion of individual projects displayed the latter approach, and many also included the former. Many of the individuals projects, are aiming at the public as a community of users, and are using this community for quality enhancement; they are adopting a web2.0 philosophy, formally described by ChemistryFM as “public peer review”. This would imply that the absence of a review cycle, which might be seen as a reason for robust initial QA processes for OER, becomes an issue of OER management – tracking usage and user feedback, which provides continuous ongoing review – rather than of initial heavy quality control.

Detailed questions and evidence

What quality processes are appropriate for different communities?

Quality processes may be deliberately initiated one-off or cyclical events, such as a user survey or annual course reports; or they may be an ongoing part of community involvement in the OERs. A high proportion of individual projects displayed the latter approach, and many also included the former, eg. openSpace’s beta testing:

openSpace went into Beta Testing – a phase that is still in progress at the time of writing this report. Beta Testing is when end users trial a website, a VLE platform, software or applications – including design - and return feedback. Our feedback solution is an online survey. openSpace Beta Testers include: UCF staff, external academics, OE practitioners, Subject Academies, other UKOER projects, creative practitioners and professionals and invited members of the general public. (openSpace final report)

Individuals projects, in general, are aiming at the public as a community of users, and are using this community for quality enhancement: eg.

An online beta tester survey is part of the beta testing process. Beta testing is important in terms of quality control and user assessment of the virtual learning environment and our OE pedagogy. Questions cover the easy of navigation, quality of supporting/help information, software and application toolbox, the user’s learning experience, learning materials and clarity of the information covering Creative Commons, reuse, re-purposing and editing. (openSpace final report)

In many cases, they are adopting a web2.0 philosophy, formally described by ChemistryFM as “public peer review” eg.

Throughout the duration of the project, the videos have been available to view on YouTube and have received an accumulated 15,475 views and a number of positive comments. One video has been retracted because a viewer noticed a minor error in a formula, which neither staff nor students had noticed, highlighting the benefits of public peer-review. This will be corrected and the video reinstated. (Chemistry FM final report)
ChemistryFM poster analysing YouTube rankings and student survey results

Polls about content will be added to the site to allow users to provide feedback. Lastly, we will be monitoring the lesson forums for user feedback or queries they may post... The use of online polls, unit study forums and feedback invitations will inform the development of openSpace, particularly in areas of student support. From our experience administering successful online community environments, .. has shown that polls, forums and feedback invitations are amongst the favoured means online communities/users use when raising problems, concerns, barriers and issues. (openSpace final report)
[openSpace includes an “add new comment” link at the bottom of each page for registered users]

In taking a community-based approach to quality enhancement, projects are evidencing the less hierarchical philosophy and flattening of learner-staff relationship, advocated by ChemistryFM in their "Teaching in Public framework"

User feedback may be supplemented by expert evaluation: eg.

Evaluation is to be completed in several ways. (a) review of materials by a qualified educator, (b) informal evaluation by students by making the developed resources available, (c) by monitoring the uptake of the open source software for development. Case (c) is a long term objective and will not be completed within the timescale of the project. (Java Breadboard final report)

How do quality processes for OER release relate to other institutional quality processes? Are there tensions/barriers?

Some projects feel that xisting processes should be sufficient already to ensure quality: eg.

Our original proposal of creating an ‘editorial team’ comprised of staff and students was not followed through. Mark and Jose felt that because the course materials are used and will continue to be used as teaching and learning resources for the course taught at the University of Lincoln, there are already robust QA processes in place: External examiners, Subject Committee (inc. students), Faculty Quality Committee, Annual course reports from subjects, module student evaluation, peer observation.... Again, in terms of sustainability, it seemed counter-productive to try to introduce new QA measures into the course simply because the resources were being made public. In our view, the spirit of OERs is about sharing the work we do, rather than introducing exceptions into our practices. (Chemistry FM final report)

This would imply that the absence of a review cycle, which might be seen as a reason for robust initial QA processes for OER, is an issue of OER management – tracking usage and user feedback, which provides continuous ongoing review – rather than of initial heavy quality control:

The ChemistryFM project will provide us with an excellent example to show other staff and a real-world example for us to measure the reach and impact of the course, through the use of web analytics, comments left on the website by students and members of the public, formal end of year reviews and possibly even course enquiries resulting from exposure to the course in this way. In this way, we expect to advance the idea of ‘teaching as a public good’ and Jose and Mark have already been impressed with the feedback on the videos on YouTube, remarking that “it made us think, how can we improve on this? … The comments made us more critical of our own work.” (Chemistry FM final report)

Summary of approaches to quality