OER Synthesis and Evaluation / Generic Synthesis and Evaluation Framework
  • If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • Files spread between Dropbox, Google Drive, Gmail, Slack, and more? Dokkio, a new product from the PBworks team, integrates and organizes them for you. Try it for free today.

View
 

Generic Synthesis and Evaluation Framework

Page history last edited by Lou McGill 9 years, 5 months ago

This framework identifies issues and questions about OER Release which shaped the pilot programme and that were raised by the projects as they progressed. This framework was offered to the projects to help them identify their evaluation questions and how they might answer them. It also informs the final pilot programme synthesis framework

 

Synthesis framework for OER Pilot projects

Focus area

Questions

Issues raised by projects

OER release processes

  • What have we learned about good practice in OER release?
  • The issues faced when developing and running a technology enhanced module
  • Issues in/guidance for using public hosting sites e.g. SL, youtube
 
  • What issues are presented by the release of particular types of content (multimedia, student-created content, interactive resources)?
  • How best to create manage and share simulations?
  • Issues in converting materials to non-propriatary and standards-based formats
  • Interoperability: issues of granularity, language, editability, dis-aggregation, translation
  • Particular issues in open release of student-produced multimedia content
 
  • How can effective processes be shared and embeded?
  • Most projects assuming the issue is one of awareness raising, rather than equal sharing, though collaboration within consortia is already an issues for Subject strand projects - and raising issues e.g. differential QA processes, different levels of commitment to OER and to production processes
  • How to raise the google ranking of OER materials?
  • What are the relative impacts/effects of using conferences, journal articles, twitter, google rankings, blogs, Intutue catalogue, British Library, other scholarly sites?
  • Pedagogic 'content' of resources an issue - what if 'pedagogy in' does not match pedagogy 'out'? Is this a problem or does the user decide?
 
  • How do existing repositories support the release of OERs in the UK?
  • Explorations of JorumOpen must be open ended i.e. not assume this is best solution for all.
  • Many projects investigating use of existing institutional or SC repositories and using project to strengthen case for them/enhance existing content
  • Raises issues of deposit and repurpose protocols; available local expertise; technical issues such as server space
  • Issues in using third-party web sites and services.

Developing, managing and sharing OERs

  • Which models are appropriate for different contexts?
  • How do different models benefit different stakeholders?
  • How is this articulated?
  • Many projects are exploring models of technology-enhanced learning, alongside models for supporting the OER lifecycle. Implying the two may be closedly connected, e.g. need clear shared model of technology-enhanced learning if OERs are to be effectively embedded, shared and re-purposed.
 
  • Which models are sustainable?
  • What effects sustainability?
  • Providing templates as exemplars of how to create reusable learning objects (allowing others to create their own)
  • Hard for many institutions to resource high quality e-learning tools/resources development across all subject areas, so an open source community model may gain purchase.
  • Will availability of OER mean more students can study? Will these extra students make the OER approach sustainable?
  • Getting more keen people involved (reduces the cost of producing resources)
  • OER approach must clearly feed into and support established courses and curriculum processes
  • Ability to deconstruct and reuse is essential to sustainability
  • Funding models for OER my differ in small specialist institutions
  • Involve OE students themselves in the lifecycle: user as producer
  • Issue related to project based centralised model needing to become decentralised within an institution to become sustainable and embedded within standard practice

Guidance and support mechanisms

  • What guidance and support needs to be offered (a) nationally (b) at institutional or even departmental level?
  • Which support mechanisms are appropriate for different stakeholders?
  • Guidelines on disaggregating and repackaging learning objects
  • Guidelines on how to develop re-usable materials in Wimba and other open source applications
  • Guidelines for institutional contributors to JORUMOpen
  • Guidance on open source release in general, including marketing and assessing educational impact
  • Sample copyright documents, including transfer, request letter, licenses
  • Guidelines on assessing and assuring quality of open resources
  • Accessibility guidelines
  • Guidelines on delivering courses to incorporate open content
  • Staff development materials on OE & IPR
 
  • What forms of evaluation are most appropriate and how best can benefits be assessed?
  • What forms of quality assurance are appropriate and who should be responsible?
  • Focus on evaluating:
    • progress of the project, project activities
    • effectiveness of project events and interventions e.g. workshops
    • quantity and quality of resources released
    • evidence of use and re-use, feedback from pilots and trials
    • evaluation of educator/staff perspective (e.g. through surveys, interviews, peer review, observations, system data, critical friends)
    • evaluation of learner perspective (e.g. through surveys, interviews, observations, system data)
    • evidence of sustainability
  • Develop criteria for: accessibility, usability, fitness for purpose, meeting external requirements e.g. JorumOpen
  • Evidence of efficient/effective processes for OER production, adaptation, release, management
  • Properly evaluated case studies are of value

Business cases and benefits realisation

  • What are effective business cases for different stakeholders?
  • Is business case an appropriate way of conceptualising the costs and benefits for stakeholders?
  • Different users (CPD users, informal learners, Lifelong Learners, students on programmes, teaching staff, librarians, resource developers, and HEIs) each have a different case, which should be reflected in the overall evaluation/synthesis.
 
  • What benefits could HE and wider society expect to see from open educational resource release?
  • Raise reputation/rankings of Institution / dept / centre of excellence
  • Enhance international dimension of educational experience
  • Evidence of benefits (efficiency and cultural) from collaborative approaches to teaching/learning resource provision
  • Support for CPD in the workplace
  • Freedom of access and enhanced opportunities for learning
  • Evidence of more learner-centred approaches thru providing access to OERs
  • Evidence of professional/peer-to-peer learning about the processes of OER release
 
  • What benefits do subject communities, institutional communities and other communities receive?
  • Building sustainable, active communities of practice around re-usable learning objects
  • Increasing numbers accessing OERs (staff, students)
  • Courses able to support increased numbers through use of OERs
  • Support for skills (e.g. numeracy) through release of OERs that can be used in different subject areas
  • Buffer against decline of specific subjects or topics (may not be sustainable at institutional level but can be sustained across several institutions through shared resources)
  • Students benefit from 'taster' materials before enrolling
  • Teaching in Public Framework (??)
  • Evidence of enhanced opportunities for: self-directed learning; peer-to-peer and social learning
  • Enhanced curriculum content at individual HEIs
 
  • What are the costs of OER release and who typically has to bear them? Are the benefits perceived as being worth the cost?
  • What proportion of these costs has been borne by the project: are the costs sustainable without project funding?
  • At present the perception is that OER release takes much longer than imagined, and would be difficult to achieve without seed funding.

Cultural issues

  • What are the current norms for sharing educational content in different communities? What global or local trends are in evidence?
  • There is a real need to explore the ethics of OER release by UK HEIs in the context of a global market for education.
  • There is also a need to explore the different attitudes to OER release within HE, FE, Industry & the professional bodies.
  • In many cases the pilot projects are defining the norms for their institution/community
  • The degree of sharing would appear to be influenced by how proactive different communities are, and the perceived quality of the OERs available within the collective.
  • Subject area also influences openness, as does the extent of collaboration (e.g. international collaborations can expose communities to different norms, values and practices)
  • Note that staff on different types of contract can have a different stake in educational resources and consequently different models of cost/benefit may apply.
  • Large scale institution-wide implementation of technologies can act as catalyst for change. *Increasing trend re development of institutional repositories for learning and teaching materials providing an opportunity to engage with openness and encouraging change in attitudes and practice. Also change of VLE to moodle seen as opportunities.
  • Staff willing to be open if ties in with/triggered by other needs (such as course review, adapting content for new business partner)
  • Marketisation has led to increased engagement with web 2.0 technologies so individuals and institutions more engaged with opening content
 
  • What motivates and supports/enables individuals to make their content open? What are effective mechanisms of reward and recognition?
  • Achieving academic staff buy-in continues to challenge projects
  • Use of like for like licences supports sharing
  • Understanding of IPR within the institution and/or community is essential to support individuals.
  • Having open practices and open licenses recognised in Institutional IPR policies is supportive of openness.
  • Offering progressive release/staged openness – choice in who to share with as confidence builder
  • Private and secure space to engage in frank discussions about concerns re open sharing
  • Mandating OER at senior level is one approach adopted.
  • Reward: could there be remuneration for CC licensing? Formal reward and recognition systems for uploading content.
  • Again see point above about staff on different contracts and in different institutional locations having different stakes in educational resources.
  • Linking OERs to highly regarded research which then drives the pedagogy
 
  • What are the institutional, legal, cultural barriers to open content?
  • Whether originator of OERs has used Third Party content (a barrier to openness)
  • Commercial implications e.g. regarding textbooks and IP the institution wants to marketise, are barriers to openness. Licenses are difficult to negotiate except for purely non-commercial use.
  • Risks associated with: plagiarism, defamation, flaming, spamming, bullying, use of 3rd Party material in OERs
  • Managing and recording provenance of back catalogues
 
  • Who benefits from release of content? How do they perceive and understand those benefits?
  • Some communities recognise that more can be done with resources if they are visible outside the institution
  • Students benefit by seeing/applying knowledge in a wider context
  • CPD users & their employers through upskilling
  • Other HEIs through adding peer reviewed material to their curriculum
  • The OER originator through reflection gained via student/user feedback
  • Institutions gain recognition and market benefits through wider availability of their academic content.
  • Increased quality of resources as they go through new processes and are ‘seen’ by different groups of staff within an institution
  • Increased collaboration and new partnerships/linkages within institutions – increased sharing of ideas and practice.
 
  • How does the opening of learning resources affect the roles of individuals?
  • Increased collaboration and partnerships within institutions means different knowledge and issues being encountered
 
  • Within what kinds of communitites does open sharing take place readily and effectively?
  • What are these communities actually sharing? What can we learn from them?
  • Some subject communities are already sharing teaching resources, particularly around specific sub-disciplines and topics. Norms and values may bind more closely at this level than the whole subject or whole programme.
  • The research culture of a topic/discipline area and the educational culture of a teaching team may be relevant to how openness is regarded.

Institutional issues - strategy, policy, practice

  • To what extent do existing policies and strategies support the opening of learning resources?
  • How are learning resources currently managed and made available within institutions?
  • Who is identified as responsible for legality, accessibility, re-usabillity and quality of open content?
  • In what ways do institutional practices (need to) change? How is transformation best approached?
  • How do existing management and departmental structures and staff roles need to be transformed to facilitate the opening of existing content?
  • Which existing institutional strategies does the opening of learning resources impact upon?
  • Many institutions are raising issues around corporate identity – wanting to embed this into any open content that is released.
  • Some participating institutions will feed lessons learned by pilot projects into their strategy and corporate plan. Most expect to develop guidelines for OER release.
  • IPR policies are commonly regarded as unhelpful and need to be changed
  • Institutions have a challenge to incorporate CC licences into their existing mechanisms
  • Institutions are also having to review how open, flexible, and/or distance learning is managed at institutional level – may require new centralised approach, or more integrated strategy
  • Technical support needs to be in place for OER design and development, server/hosting issues, and content management
  • International partnerships are a feature of many institutions' strategic approaches to OER.
  • Contact/relationship management becoming important to track the number of people involved in releasing content

Legal issues

  • Are ownership and legal issues still perceived as a major barrier?
  • Have perceptions changed during the timescale of the programme? Is new guidance needed?
  • Who in institutions and communities takes responsibility for the legality of open content release? What barriers do they present and what support do they offer?
  • What are the IPR issues relating to hybrid, multiply-authored resources?
  • Impact of creative commons licensing and ethos may be offset by increased marketisation of content, full cost recovery agenda.
  • Very variable support at institutions: in some places just 'asking the question' about IPR can be seen as dangerous, leading to increased scrutiny and problems.
  • Different institutions have different policies on who owns the IPR on teaching materials
  • Low general knowledge of IPR in the community.
  • Most projects are using CC licences. Some are seeking written contracts with contributors, and looking at different ways of accrediting individuals and recognising originating institutions
  • Concern that creative commons licensed resources will be buried online through search engine ranking
  • Projects are seeking advice on dealing with both incoming and outgoing rights clearance (provided through JISC Legal)
  • Copyrighted images can be replaced by other resources. Risk of people seeking open release looking for work-arounds and under-the-radar approaches rather than making visible the problems they are facing.
  • Practical and legal complications of tracing IPR correctly on materials created from multiple sources
  • Concerns that IPR issues will impact in a negative way on quality and pedagogic theory of resources

Technical and hosting issues

  • Are there any messages around tools and standards that come from the programme?
  • Projects are using a very wide range of standards due to the wide range of materials being released:
  • Standards: SCORM, SWORD, SENDA/W3C/WAI/Section 508, IMS LD/CP/QTI, IEEE LOM, RSS/Atom, METS content packaging, Simple Dublin Core, OPML, OAI-PMH
  • Document formats: pdf/ua/odf, java, javascript, html, xhtml, css, wcag, word, mpeg, mp3, ajax, jpeg, flash, vra/cdwa, rss
  • CETIS is advising projects on technology and standards
 
  • What kinds of metadata are essential, what desirable, and what are the issues in creating and managing metadata?
  • Tension between rich tagging to ensure shared understanding of how resources address key academic issues, and lightweight, usable metadata solutions
  • Several approaches to tagging which will be explored to surface how lightweight the metadata requirements can be while facilitating discovery and reuse. e.g. adding multimedia support to e-*Prints, using existing repository tools
  • Will better tagging improve Google ranking of OERs?
  • Long standing issue of who adds metadata still a consideration for projects - professional/academic/resource users
 
  • How do existing repositories support the release, management, discovery preservation and access to OERs e.g. OpenJorum in the UK, institutional repositories within an institution, web sources globally, etc
  • Dealing with expired content and keeping content up to date
  • Duplicated content or linked-to?
  • Different degrees of openness available to depositors?
  • Choices re deposit into JORUMOpen? Open content ethos = specific hosting solutions don't matter. Also the development path of JorumOpen may be out of sync with project requirements
  • Many projects grappling with issue of consistency, tracking and management of resources available through web, institutional repositories and JORUMOpen, particularly in relation to preservation and archiving
 
  • What issues arise when using public/third-party hosting solutions?
  • What are implications of using e.g. SL islands, youtube, i-tunes, twitter...?
  • How well do Web 2.0 sites support interactivity e.g. Flash animations?
  • Web2.0 sites are at different stages of development and inconsistent in the media they will support - some specialise in specific media types and vary in how they allow access
  • There is an issue of resource ownership in relation to some or all third party web sites, e.g . in Flickr the person uploading the resource is, by default, the resource owner. Many sharing sites require consent to conditions of use statements that may violate CC and other licences.
 
  • How best to make hybrid, interactive and multi-media resources available for open access.
  • Use of web 2.0 services to support flexibility, portability etc.
  • Issue of granularity to support repurposing.
  • Simulations are interesting example of hybrid multimedia resources which demand particular approaches to open release.

Quality issues

  • What quality processes are appropriate for different communities?
  • Quality issues need to be raised as a question for the pilot programme, not a solution, vis: what approaches to quality are most supportive of open release? Options include:
    • Peer review as part of a pilot (assessing quality, usability and re-usability) – how can this be sustained, encouraged and resources?
    • Student feedback (survey and questionnaire)
    • Expert/critical friend (quality assurance, procedural checks)
    • Community use, comments, ratings etc
 
  • How do quality processes for OER release relate to other institutional quality processes? Are there tensions/barriers?
  • Academic quality checks largely provided by existing institutional or subject community processes: 'e-learning' or 'online learning' quality processes may be less mature or robust
  • OER quality processes need to be robust and sustainable over the longer term as open content has no 'review' cycle, unlike institutional content.
 
  • Are OERs perceived to be of high quality? What impact do perceptions of quality have on release process/sustainability?
  • What is the educational value of the OERs? Pedagogical quality must remain the critical success factor

Pedagogy/end-use issues (not a primary focus of evaluation)

  • Which types of OER are used by different stakeholders?
  • Types of OER under investigation:
  • Online and podcast lectures, lecture notes, audio files, ppt slides, worksheets, Open source software; tutorial materials, videos, lectures, notes, reading lists, online assessment tools, student stories, learning outcomes and objectives; course outlines; workshops; web resources; self test quizzes; essay revision; exam materials, questions/answers, multiple choice questions, self-study assignments, guidance, RLOs, simulations
  • Note that it will be difficult to get feedback from downstream users of the OERs released under the pilot projects due to timescales, but most have measures in place to monitor uptake or to support adoption at other sites and gain feedback from workshops and pilots.
 
  • Can we see a pattern in relation to level of granularity and use, re-use, re-purposing?
  • Do OERs need to separate resources from tasks, enabling the re-use of the whole module or just individual tasks as users choose?
  • Disaggregation of resources – what enables this? Do contributors always want this to be possible?
 
  • How far are use patterns influenced by: the subject discipline and/or topic area; type of resources made available?
  • Evidence from subject strand that topic area may be more significant than subject alone: different cultures of sharing, different histories of shared curriculum
  • Need to be sensitive to different cultures of education, in context of glogal OER release: consider the ethics and likely impact of OER programme
 
  • How is pedagogy manifested in open content, if at all?
  • What additional information needs to be packaged with OERs to make them more reusable?
  • Balancing 'pedagogy in' (intentions of original contributors) with 'pedagogy out' (intentions of users, whether staff or students). Does the former matter at all, if users are clear about educational purpose?
  • Different business/market models imply different balances here, from complete educational resources (inherently pedagogically meaningful) to re-usable assets.
  • Role of templates to support consistency in preparing OERs for release
  • Case studies, comments, reviews etc
  • Examples of learning activities
  • Guidance on how materials currently used
  • Encourage contribution by users regarding contexts and strategies of re-use.
 
  • In what ways, if at all, do learning and teaching practices (need to) change when OERs are widely available?
  • Is the web a suitable broadcasting medium?
  • How can open education be accredited?
  • How do we evaluate different methods of delivery/support for OERs?
  • Need to build an OER mentality into curriculum design and delivery
 
  • What skills/literacies do staff and students need to adapt to using and creating content in an open way?
  • Medium to high levels of technical skills/IT literacy.
  • Need for training and OER guidelines
  • Need to feed requirements to teacher training accreditation bodies
 
  • How can student-created content be made openly available for sharing, peer review and collaboration?
  • Raise awareness of the OER movement, academic commons, CC licensing etc among learners
  • Help learners to understand principles of open release and making materials available in accessible forms

Learner and other stakeholder involvement

  • What role have learners played in shaping the programme outcomes? How have projects engaged learners, if at all?
  • Do learners appreciate the value of OERs? What sense do they make of 'openness'?
  • Issues in the student-led development of resources (bursaries for student involvement? Open release of student-authored content as an aspect of the curriculum?)
  • Student involvement in piloting, trials and evaluation
 
  • What role have stakeholders such as professional bodies and employers played in shaping the programme outcomes?
  • Private and public sector partnerships
  • Drawing on the experience of partners in developing countries
  • Involving business and professional communities
 
  • What other stakeholders are emerging with an interest in this area?
  • Involving local communities e.g. through local radio
  • Evaluating methods of delivery to ensure appropriate for range of stakeholders and users
  • Google Analytics and general web analytics being used to track users

Programme and project management issues

  • What challenges arise from consortia approaches? Which consortia approaches are effective?
  • Issues in achieving consortium agreements – are programme issues in microcosm.
  • Dealing with volume of communications from support and synthesis team and other projects
  • Can be tension between project-level and programme-level evaluation – needs to be handled in a way that makes evidencing and data-collection very efficient at project level.
 
  • Collaboration
  • Many projects are already collaborating very well (for such a short scale pilot programme). *How far could joint marketing approaches be supported? Projects are asking these questions now.

 

 

Comments (0)

You don't have permission to comment on this page.