Focus area

Evaluation question

Evidence

What are the main enablers and barriers to adoption of open practices and open content in FE and HE?

How are different approaches overcoming barriers to adoption of open practices effective?

Which aspects of the new approaches in the project were successful? (COMC)

How are different means of engaging a variety of stakeholders and embedding open practices effective?

Have these been successful in engaging community? (CORE-SET)

Have these been successful in engaging community? (core-set)

Are staff, students  and other participants contributing to the classes. (COMC)

Have students actively participated in the classes and engaged with both the ‘open’ content and this approach? (COMC)

Has each class enlisted external contributors as speakers, experts and/or practitioners? (COMC)

What level of external visitors and contributors has each class received? (COMC)

Has the English Faculty embraced the project and has it been embedded as a resource? Has the project provided an effective case study for impact and collaboration? (great writers)

Has the community engaged with the project? (Favor)

What changes to individual academic practice are fostered by OER development and use?

Has the Nottingham Advantage Award team embraced delivery of materials through the U-Now site? (Paris)

Are staff increasing their sharing of materials? (Alto)

Are staff using the OERs to reuse materials and adapt? How has use of OER influenced academic activity? Are there any indicators of efficiency savings (Alto)

Has the project initiated a community of sharing amongst part-time language tutors? (Favor)

What changes to stakeholder culture and policies are fostered by OER development and use?

How are different means of supporting practice change by projects effective

How effective is the support / training offered to staff to develop their knowledge and understanding of OERs and the ALTO UK system (Alto)

Are guidance and support materials useful? Can any improvements be made? Are they being used? (Sesame)

How has OER release affected academic practice of staff running HEA accredited courses or schemes of professional development that meet the UK Professional Standards Framework for Teaching and Supporting Learning in Higher education (UK PSF)?

Releasing and using OERs - different OER publishing models, the role of collaborations, technical and hosting issues, discovery and organisation, accessibility, and adaptability of OERs

How are different approaches to making OERs discoverable effective?

What are the pros and cons for different stakeholders of the different models investigated? (publishoer)

Are our OER discoverable? Do they impact on the user? (Hals)

Can students find OER? Can they critically evaluate it? (Hals)

Are our OER discoverable nationally and internationally? Have they made a difference? What granularities and types of resource are most popular? (Hals)

What means of social networking are most effective in making OER found? How effective are APPs in promoting OER? (Hals)

How easily can appropriate content be discovered and re-used (given a particular set of content and technology choices)? (Orbit)

How do different means/formats for making OER available suit the needs of various stakeholder groups?

Are all stakeholders happy with the outputs of the project? (COMC)

Are eBooks useful in teaching and learning? (great writers)

Can an effective method of delivery be found? (great writers)

What types of resources would be most useful to end users – looking at subject, format, type and size? (Sesame)

Are the resources created as part of this project available in open formats that support use and re-use? (Paris)

What qualities should resources possess? What kinds of resources are considered most useful? (orbit)

How are different ways of organising, and guiding users to and through, resources effective for different stakeholder groups?

What information about resources would make them more useable? Are our release platform and other proposed release routes appropriate? (Sesame)

Are the user interfaces appropriate for learners? (CORE-SET)

Does our Discovery point and other release routes facilitate uptake? (Sesame)

Does the information provided about them aid their use? (Sesame)

How does cross sector collaboration support the collection and release of OERs?

Collaborative/reflexive elements, engagement of partners, transferability and scalability (Deft)

Feasibility of OER in non-HE contexts, transferability and scalability of approach (Deft)

How does working with commercial publishers support the collection and release of OER?

How does development and release of OER draw on existing OER collections?

What technical issues arise in collecting and making OERs available?

Has the project team adapted the technology? (core-set)

Can ebooks be enhanced by the project? (great writers)

Are the OER accessible to all intended groups (pedagogically and technically)?

Early questions about appropriateness of the resources, aligned to curriculum, accessible, etc? (reactor)

Is the material easy to utilise and appropriate to target group? (OFB)

Is material understandable and easy to utilise? (OFB)

Have disability needs been met? (OFB)

Is the structure & material identified the most relevant to include? (OFB)

Do the resources created by the Ear Foundation meet the identified needs of the target audience (as identified through the needs analysis at the start of the project) (Paris)

Are the collections useful in teaching and learning? (Great writers)

Fitness for purpose, accessibility, uptake (Deft)

Is newly-created material appealing and pedagogically sound? (Favor)

Are tools and interfaces easy to use? Can processes be easily followed? Can any improvements be made? (Sesame)

Are resources accessible? (Sesame)

How are different means of making OER adaptable for re-use and re-purposing effective?

What legal and IPR issues emerged?

Do all resources comply with copyright regulation and risk levels set by each partner (Paris)

Are the OER collected available as discrete items, with licence embedded (if available)? (great writers)

Adherence to QA indicators, accessibility; IPR and copyright (Deft)

How easily can OERs be mapped to the HEA professional standards framework and what are the gaps?

How far does the subject discipline or partner sector impact on the development and release of OER?

What range of OER are available?

Are all the Open Class sites being populated with resources and are these being well curated (COMC)

What subject OER already exist internally and externally (Hals)

Have gaps been identified? (great writers)

Are the collections of appropriate quality and breadth? (great writers)

Has the number of resources in the Language Box repository increased significantly? (Favor)

Are there sufficient resources in terms of number and variety to be useful? (Alto)

What support is needed for OER release by projects, OER practitioners and senior managers in HEI?

Processes for sustainability - institutional/organisational policies and processes, legal and IPR policies, sustainability

What are the cross sector issues and challenges in developing, releasing and re-using OERs?

What were the benefits or challenges arising from using OER in a commercial venture? (Hals)

How do different types of institution or stakeholder ensure embedding and sustainability of OER and open practices?

How do institutions support staff to change practice/develop skills/knowledge?

Has the project work of part-time language tutors been recognised in their institution? (Favor)

Is the Department able to sustain supporting the technical infrastructure? (Sesame)

What is the impact of OER development, release, and re-use on the practices of different stakeholders or sectors?

Are part-time tutors prepared to continue to produce OER? Is the weekly class programme able to support the work on an ongoing basis? (Sesame)

Can students find OER? Can they critically evaluate it? (Hals)

What are the main enablers and barriers to adoption of open practices and OER in different stakeholder groups?

Do the materials meet needs of teaching staff, students or external stakeholders? (Alto)

What ways of supporting sector priorities are effective for different sectors?

What resources (subject and type) are identified as being most needed by Ear Foundation target audience? (Paris)

What kinds of OERs are being adopted and re-used by different stakeholders, and how?

How effective are the OERs? (Orbit)

What issues arise in developing and sharing collections of OERs across institutional/sectoral boundaries

What issues arise in curation of OER collections across institutions or stakeholder groups?

Are all the Open Class sites being populated with resources and are these being well curated (COMC)

How are existing stakeholder cultures being challenged, strengthened, contested, changed etc through the availability of OERs?

How can OERs be integrated sustainably into different curriculum processes of the various stakeholder institutions?

Have we put in place robust plans for sustainability? (core-set)

Early questions about appropriateness of the resources, aligned to curriculum, accessible, etc? (reactor)

Are sustainability resources embedded in taught curriculum at Nottingham? (Paris)

Are Ear Foundation resources embedded within taught courses? (Paris)

Impacts & benefits - benefits, and appreciation of benefits by, stakeholders, institutions, students

How does student involvement in OER development, release and re-use affect sustainability?

What has been the impact of OER development and release on the student experience?

Has the student experience been enhanced - re skills, opportunities, particpation? (reactor)

Can disabled students particpate in things previously problematic? (reactor)

Have work experience and placement opportunities increased? (reactor)

Have students actively utilised the networks, additional contributors and linked resources? (COMC)

Are students enrolled on Nottingham Advantage Award materials happy with the materials? (Paris)

Are students willing/able to participate in the project? (great writers)

How far does an open textbook support student learning in comparison with using closed resources? How can the benefit of open academic practice be measured? (Orbit)

Methods of engaging students with OER and digital literacy issues (Deft)

Are students using the OER provided for their courses? Do students value the availability of these additional resources?Can anything be done to make the resources more relevant to students?

How have your stakeholders' appreciation of the benefits of OERs changed through involvement in the project?

Does the Ear Foundation see benefit in continuing an OER initiative? (Paris)

Have external partners found being involved in OER a useful experience? (Hals)

Has the project met or addressed the goals and objectives? Has the project articulated the benefits of OER to the teacher education sector? What other outcomes have been noted (tangible and intangible)? (Deft)

How did part-time tutors find the experience of generating OER to support their courses? What are the main motivations for, and barriers to, use and creation of OER? Are part-time tutors prepared to continue to use and produce OER beyond the end of the project?

What is the impact on teaching staff of OER release and use?

How have OERs or the OER release process benefited your stakeholders?

Have the benefits been met for different stakeholders? (reactor)

How do the business models impact on user behaviour (Kirkpatrick’s scale) and what are the intended/unintended consequences of embedding third party published content within OER? (publishOER)

In what ways has your approach to OER release and re-use enhanced the profile and reputation of your institution?

Have the small test cases of new modes of ‘freemium’ engagement elicited new, and new kinds of, Open Class participant? (COMC)

Is newly-created material appealing and useful to prospective students? Do other universities intend to use them? (Favor)

Are the OER produced being used beyond the Department? Are teachers and learners finding the resources useful? (Sesame)

What other benefits are projects finding?

Have there been cost savings? (reactor)

Have there been health & safety benefits? (reactor)

Are staff using the OERs to reuse materials and adapt? How has use of OER influenced academic activity? Are there any indicators of efficiency savings (Alto)

Evidencing and Programme issues:

How can we demonstrate that different OER publishing models are effective in aiding discoverability?

What metrics need to be collected/analysed in order to interpret the business model success? (publishOER)

What further research is required to develop the most successful concepts? (publishOER)

How do projects evidence that OERs enhance the student experience?

How far does an open textbook support student learning in comparison with using closed resources? How can the benefit of open academic practice be measured? (Orbit)

How do projects claim they will support the relevant sector priority?