Focus area

Evaluation question

Project questions that may provide evidence

What does the phrase "open practices" mean for you and your project? What open practices have you observed among your stakeholders?

What enablers and barriers have you identified to the adoption of open practices by your stakeholders, and in what ways have you addressed them?

Which aspects of the new approaches in the project were successful? (COMC)

In what ways has engaging in open partnerships across institutions, organisations or sectors,affected the practices of your stakeholders?

What means have you used to engage your stakeholders, and in what ways have these been effective?

Have these been successful in engaging community? (CORE-SET)

Have these been successful in engaging community? (core-set)

Are students willing/able to participate in the project? (great writers)

Methods of engaging students with OER and digital literacy issues (Deft)

Are staff, students  and other participants contributing to the classes. (COMC)

Have students actively participated in the classes and engaged with both the ‘open’ content and this approach? (COMC)

Has each class enlisted external contributors as speakers, experts and/or practitioners? (COMC)

What level of external visitors and contributors has each class received? (COMC)

Has the English Faculty embraced the project and has it been embedded as a resource? Has the project provided an effective case study for impact and collaboration? (great writers)

Has the community engaged with the project? (Favor)

What has been the impact of OER development, release and reuse on the practices of your stakeholders? 

Are part-time tutors prepared to continue to produce OER? Is the weekly class programme able to support the work on an ongoing basis? (Sesame)

Can students find OER? Can they critically evaluate it? (Hals)

Has the Nottingham Advantage Award team embraced delivery of materials through the U-Now site? (Paris)

Are staff increasing their sharing of materials? (Alto)

Are staff using the OERs to reuse materials and adapt? How has use of OER influenced academic activity? Are there any indicators of efficiency savings (Alto)

Has the project initiated a community of sharing amongst part-time language tutors? (Favor)

By what means have you supported practice change among your stakeholders, and in what ways were these effective?

How effective is the support / training offered to staff to develop their knowledge and understanding of OERs and the ALTO UK system (Alto)

Are guidance and support materials useful? Can any improvements be made? Are they being used? (Sesame)

How has OER release affected the practices of staff running HEA accredited courses or schemes of professional development that meet the UK Professional Standards Framework for Teaching and Supporting Learning in Higher education (UK PSF)?

Releasing and using OERs - different OER publishing models, the role of collaborations, technical and hosting issues, discovery and organisation, accessibility, and adaptability of OERs

Collaborative/reflexive elements, engagement of partners, transferability and scalability (Deft)

Through what means (eg. platform, format, interface) have you made OER available to your stakeholder group?

Are all stakeholders happy with the outputs of the project? (COMC)

Are eBooks useful in teaching and learning? (great writers)

Can an effective method of delivery be found? (great writers)

What types of resources would be most useful to end users – looking at subject, format, type and size? (Sesame)

Are the resources created as part of this project available in open formats that support use and re-use? (Paris)

What qualities should resources possess? What kinds of resources are considered most useful? (orbit)

Are the user interfaces appropriate for learners? (CORE-SET)

In what ways (appropriate to your stakeholders) have you organised your OER and guided users through them?

What information about resources would make them more useable? Are our release platform and other proposed release routes appropriate? (Sesame)

Does our Discovery point and other release routes facilitate uptake? (Sesame)

Does the information provided about them aid their use? (Sesame)

What measures have you taken to ensure that your OER are accessible to your intended user groups both pedagogically and technically?

Early questions about appropriateness of the resources, aligned to curriculum, accessible, etc? (reactor)

Is the material easy to utilise and appropriate to target group? (OFB)

Is material understandable and easy to utilise? (OFB)

Have disability needs been met? (OFB)

Is the structure & material identified the most relevant to include? (OFB)

Do the resources created by the Ear Foundation meet the identified needs of the target audience (as identified through the needs analysis at the start of the project) (Paris)

Are the collections useful in teaching and learning? (Great writers)

Fitness for purpose, accessibility, uptake (Deft)

Is newly-created material appealing and pedagogically sound? (Favor)

Are tools and interfaces easy to use? Can processes be easily followed? Can any improvements be made? (Sesame)

Are resources accessible? (Sesame)

Do the materials meet needs of teaching staff, students or external stakeholders? (Alto)

In what ways have you tried to ensure that your OER are adaptable for re-use and re-

purposing, and how effective were they?

What are the pros and cons for different stakeholders of the different models investigated? (publishoer)

Are our OER discoverable? Do they impact on the user? (Hals)

Can students find OER? Can they critically evaluate it? (Hals)

Are our OER discoverable nationally and internationally? Have they made a difference? What granularities and types of resource are most popular? (Hals)

What means of social networking are most effective in making OER found? How effective are APPs in promoting OER? (Hals)

How easily can appropriate content be discovered and re-used (given a particular set of content and technology choices)? (Orbit)

Are all the Open Class sites being populated with resources and are these being well curated (COMC)

What subject OER already exist internally and externally (Hals)

Have gaps been identified? (great writers)

Are the collections of appropriate quality and breadth? (great writers)

Has the number of resources in the Language Box repository increased significantly? (Favor)

Are there sufficient resources in terms of number and variety to be useful? (Alto)

In what ways have you drawn on existing OER collections in developing and releasing OER?

What kinds of OER are being adopted and re-used by your collaborators/partners/stakeholders in other sectors and in what ways are they are being developed and used?

How effective are the OERs? (Orbit)

What technical issues have you met in collecting and making OERs available, and how have you addressed them?

Has the project team adapted the technology? (core-set)

Can ebooks be enhanced by the project? (great writers)

Processes for sustainability - institutional/organisational policies and processes, legal and IPR policies, sustainability

Has the project work of part-time language tutors been recognised in their institution? (Favor)

Is the Department able to sustain supporting the technical infrastructure? (Sesame)

Do all resources comply with copyright regulation and risk levels set by each partner (Paris)

Are the OER collected available as discrete items, with licence embedded (if available)? (great writers)

Adherence to QA indicators, accessibility; IPR and copyright (Deft)

Have we put in place robust plans for sustainability? (core-set)

Early questions about appropriateness of the resources, aligned to curriculum, accessible, etc? (reactor)

Are sustainability resources embedded in taught curriculum at Nottingham? (Paris)

Are Ear Foundation resources embedded within taught courses? (Paris)

How have your institution and collaborating organisations ensured embedding and sustainability of open practices?

What has been the effect on sustainability of involving students in the OER developement, release and re-use lifecycle?

What issues have you found over curation of OER collections when working across institutional or sector boundaries and how have you addressed these?

Are all the Open Class sites being populated with resources and are these being well curated (COMC)

How have you have identified and attempted to meet the priorities of your sector?

What resources (subject and type) are identified as being most needed by Ear Foundation target audience? (Paris)

Impacts & benefits - benefits, and appreciation of benefits by, stakeholders, institutions, students

Have the benefits been met for different stakeholders? (reactor)

How have your stakeholders' appreciation of the benefits of OERs changed through involvement in the project?

Does the Ear Foundation see benefit in continuing an OER initiative? (Paris)

Have external partners found being involved in OER a useful experience? (Hals)

Has the project met or addressed the goals and objectives? Has the project articulated the benefits of OER to the teacher education sector? What other outcomes have been noted (tangible and intangible)? (Deft)

How did part-time tutors find the experience of generating OER to support their courses? What are the main motivations for, and barriers to, use and creation of OER? Are part-time tutors prepared to continue to use and produce OER beyond the end of the project?

What is the impact on teaching staff of OER release and use?

What has been the impact of OER development and release on the student experience?

Has the student experience been enhanced - re skills, opportunities, particpation? (reactor)

Can disabled students particpate in things previously problematic? (reactor)

Have work experience and placement opportunities increased? (reactor)

Have students actively utilised the networks, additional contributors and linked resources? (COMC)

Are students enrolled on Nottingham Advantage Award materials happy with the materials? (Paris)

How far does an open textbook support student learning in comparison with using closed resources? How can the benefit of open academic practice be measured? (Orbit)

Are students using the OER provided for their courses? Do students value the availability of these additional resources?Can anything be done to make the resources more relevant to students?

In what ways has your approach to OER release and re-use enhanced the profile and reputation of your institution?

Have the small test cases of new modes of ‘freemium’ engagement elicited new, and new kinds of, Open Class participant? (COMC)

Is newly-created material appealing and useful to prospective students? Do other universities intend to use them? (Favor)

Are the OER produced being used beyond the Department? Are teachers and learners finding the resources useful? (Sesame)

In what ways other than reputation, have OER release and re-use benefited your institution?

Have there been cost savings? (reactor)

Have there been health & safety benefits? (reactor)

Are staff using the OERs to reuse materials and adapt? How has use of OER influenced academic activity? Are there any indicators of efficiency savings (Alto)

Evidencing and Programme issues:

How can we demonstrate that different OER publishing models are effective in aiding discoverability?

What metrics need to be collected/analysed in order to interpret the business model success? (publishOER)

What further research is required to develop the most successful concepts? (publishOER)

How do projects evidence that OERs enhance the student experience?

How far does an open textbook support student learning in comparison with using closed resources? How can the benefit of open academic practice be measured? (Orbit)

How do projects claim they will support the relevant sector priority?