E-TIE workplan

Objectives

The objectives of Embedding Trust in Evaluation are to:

O1. Increase confidence across the HEI sector of the value of NTOs and hidden roles and their vital role in research.
O2. Improve understanding of the UK research community’s orientations towards the evaluation of NTOs and hidden roles.
O3. Build an evidence base and resources to support the reliable evaluation of NTOs and hidden roles.
O4. Where possible and in collaboration with the REF team, work with REF2029 panellists (across Main Panels A-D) to learn about and increase their confidence in evaluating NTOs and hidden roles.
O5. Deliver a change in research culture as measured by an increase in submissions to REF2029 of NTOs and outputs created by hidden roles.

Work Packages

To achieve these objectives, E-TIE to conduct work across six work packages. Simon Hettrick will responsible for work packages related to community activities and development (WP1A-1C). Gemma Derrick will lead the evidence team and associated work packages (WP2A-2C). James Baker will lead on coordination between work packages and across activities.

WP1A: The Hidden REF Competition

The Hidden REF competition is a community-driven event that raises awareness of non-traditional outputs (NTOs) and hidden roles, and provides a platform for understanding them. The first two competitions received 230 submissions from around 100 organisations. We expect entries to increase in the future up to 300 submissions for the competition in 2028.

The valuable experience of assessing NTOs and Hidden Roles gained from the competitions will be condensed into policy recommendations which will be shared across the HEI sector and, with the permission of the REF team, with the REF2029 panellists via activities such as calibration workshops. Case studies from the competition will be widely publicised to increase recognition and used during community consultations to provide background. Experience of running the competition will underpin The Hidden REF Toolkit which will fulfil demand from international stakeholders for help in establishing their own Hidden REF competitions.

WP1B: Community consultations

Evaluation exercises that mirror the norms and values of the research community ensure greater community participation and increase trust in the outcomes of the exercise.

A series of community consultations will collect experiences on how to recognise the role and impact of NTOs and Hidden Roles. Each consultation will bring together a different group of stakeholders with consideration of demographics including discipline, region, career-stage, gender and ethnicity. The consultations will be a mixture of online events and nationally distributed in-person events to increase accessibility. The exact focus of the consultations will be informed by our community, but we expect to include people who create NTOs, people employed in different hidden roles, organisations that support hidden roles, senior stakeholders at HEIs involved in recognition, and people at HEIs involved in the preparation of the REF submission. The information we collect will refine our recommendations on assessment, future Hidden REF competitions and will feed into the evidence-review work package.

WP1C: HEI and sector engagement

We will use a range of activities to increase the breadth and depth of our HEI and sector engagement.

The enthusiasm within the sector for our mission and the authority we have amassed has been demonstrated by requests we have received to run NTO and Hidden Role training workshops. We expect demand to increase significantly once we start advertising the existence of these workshops. The workshops will train attendees to recognise and assess NTOs and Hidden Roles using the guidance on assessment practices developed through WP1A and WP1B. These workshops represent an unparalleled opportunity to provide HEIs with the skills needed to broaden the content of their REF submissions. It will also be an invaluable for studying how REF assessment practices are being interpreted within HEIs. Feedback from the workshops will be used to further refine our guidance.

We will build further impact from the NTO and Hidden Role training workshops by forming a community of practice around the people who take part in them. This idea has been strongly encouraged by the HEIs that have approached us, because of the opportunity it gives them to share best practice. The community of practice also provides us with the right contacts to help integrate recognition of NTOs and Hidden Roles into HEI policies.

The Festival of Hidden REF was launched in 2023 to empower our community to campaign on our behalf. We will use future bi-annual Festivals to provide our community with the information needed to raise awareness, the support needed to meet with senior staff at their institutions, and the evidence and guidance needed to promote the benefits of changing institutional approaches to the REF. Based on the 100 people who attended the first Festival, we expect future Festivals to bring together around 100-200 people per event.

Our 5% Manifesto is a campaign tool that we will use to help convince HEIs to submit at least 5% of their REF2029 outputs into non-traditional categories: a feat achieved by only 46 out of 157 organisations in REF2021. It will be a focus of our lobbying of HEIs, a platform for our community to raise awareness at their home institutions and will also provide a measure of our impact on the sector. We note that the 5% Manifesto will not be used as a KPI for the project, because its ease of adoption varies significantly across HEIs. Instead our KPI in this area focusses on increasing the number of NTOs that are submitted during REF2029.

We will support our HEI and sector engagement activities with regular publicity in the research media and blogs, social media, and talks at events. This will be amplified via the considerable reach of our Supporters’ communication platforms and contacts. We will also continue to engage with HEI events that focus on the REF, research assessment, research careers and related areas.

WP2A: Evidence review

A systematic review of the literature on the evaluation of NTOs will be combined with evidence from the community consultations to ensure that the Hidden REF and REF2029 approaches are simultaneously community- and evidence-led. Outcomes will be translated into evaluation criteria, benchmarks and approaches to be tested in each Hidden REF competition. Open and transparent publication and dissemination of this evidence base within the academic literature and insights gained from the Hidden REF competition will work to increase sector confidence in NTO evaluation. This work package will also collect, analyse and collate data from Hidden REF competition evaluators, participants and panel chairs [WP1A], to further consolidate evaluative approaches for NTOs, and stimulate international academic debate on NTO evaluation approaches.

WP2B: REF2029 Panel calibration exercises

Building on the insights gained from progressive Hidden REF competitions [WP1A], community consultations [WP1B], and the evidence review [WP2A], we will seek to run a series of workshops which can be delivered as part of the REF calibration exercises. Calibration exercises are commonly used by funding agencies, and past REF exercises, to clarify expectations and common approaches to be used during the evaluation. They typically occur prior to the evaluation and before finalising scores of the evaluation process, and are known to facilitate consensus building and increase reviewer reliability.

WP2C: Panel observations REF2029

Finally, we hope to determine the increase in confidence about the value and competitiveness of NTOs by conducting panel observations and a series of post-evaluation interviews. These will assess how panels used the evidence from the calibration exercises and identify any additional evaluation heuristics that were developed by panels in-vivo. Understanding how REF2029 panellists operationalise the pre-evaluation training provided by the project is vital to form a strong evidence base for the evaluation of NTOs. Previous experience has shown that whilst post-evaluation interviews can be effective, they are a poor substitute for direct observations. Direct observation aligns with an increasing trend among funding councils globally towards greater scrutiny of their decision-making processes by permitting established meta-researchers to observe peer review deliberation processes.

Legacy and sustaining activities beyond the funding period

Ultimately, the goal of the Hidden REF is the institutionalisation of improved approaches to NTO and hidden role evaluation across all UK HEIs. This work would be necessary even if a national assessment programme did not exist, because it is self-evident that we will advance research if we recognise all contributions to research and everyone involved in its conduct. The work of the Hidden REF will provide the UK sector with a competitive advantage.

The project team is committed to sustaining research culture change around the value of NTOs and hidden roles and to making research more equitable and effective. Everything learned during the Hidden REF and the analysis of the project’s impact on REF 2029 will be written into the final report which will be shared with UKRI, all four UK higher education funding bodies, the Hidden REF community and published openly for the global research community. To maximise impact and reuse, and help attract new collaborators and contributors, by default all resources will be openly licensed, using CC-BY for data, articles, guides, presentations and reports, and OSI-approved licences for software.

We expect this work to enact substantial changes to research culture and we anticipate that new dynamics will emerge that will require further work (e.g. an evolved format of post-REF2029 Hidden REF competitions). We have a strong track record of securing sponsorship, support, and in-kind contribution from professional bodies, community organisations, SMEs, and HEIs. This increases our engagement and impact and furthers our objectives. We expect the Hidden REF to continue in some form after the period of the project, and see our approach to partnership working as a cornerstone of our long-term financial sustainability.

Outputs

The project will yield a series of outputs as work package deliverables. All members of the Hidden REF team will be co-authors.

Academic Papers: the first paper, published in 2025, is a systematic review and meta-analysis of the literature on the evaluation of NTOs. The second, published in 2028 with an accompanying dataset, synthesises evidence gathered from community consultations, Hidden REF competition evaluators, participants, and panel chairs, and HEI sector engagement.. The third paper, published in 2029, reports on the Hidden REF competitions [WP1A] for a broad-based and international stakeholder community.

Sector Policy Recommendations: published in 2026, the first policy report synthesises initial findings from the project. Aimed at research professionals and sector leaders, it provides a touchpoint for community consultation and sector engagement. The second policy report demonstrates our depth of community consultation and sector engagement. Initial recommendations will be workshopped at the 2027 Hidden REF Festival and publication will complement synthesis of evidence gathering.

The Hidden REF competition toolkit: published in 2026, the toolkit fulfils demand from international stakeholders for help in establishing their own Hidden REF competitions. Published as a web-based collection of documentation, worksheets, and templates, the toolkit can be used to set up a Hidden REF-like competition in other countries.

Final Report: published in 2030, the final project report brings together evidence, findings and recommendations from across all work packages and deliverables, reports on the project’s objectives and impact on REF2029, and sets out anticipated next steps. We will also deliver an evaluation report which we anticipate will be published in 2030 along with other post-REF2029 reporting.

Wider benefits

Recognising all the outputs and people that are vital to the conduct of research creates a research culture that advances all research. Hence, we believe that the success of this work will be felt across the entire HEI sector and all beneficiaries from the research that is empowered as a result.

The project will benefit the wider UK research community by providing transparent, community-based and trustworthy evaluation and valorisation guidelines for NTOs to HEIs ahead and beyond REF2029. In particular, the project will benefit those researchers that contribute but are not currently valued by HEIs. Such benefits will only strengthen UKRI’s strategic focus on improving research culture by encouraging HEIs to recognise and value all research contributions. The university-based workshops underpinning WP1C will assist in realising this benefit influencing not just REF-submission processes, but academic promotion and recruitment policies, and will help to diversify the range of research careers support in general.

The materials of previous Hidden REF exercises have been adopted and utilised in other countries and international organisations (e.g. UC Dublin, Finland) and therefore this project is likely to form the foundation for adoption of NTO evaluation approaches in other countries and contexts as well.

Project governance

The project leadership team, consisting of Hettrick, Baker & Derrick, provide the management and leadership necessary to deliver the project’s objectives, with Hettrick as PI ultimately responsible for delivery. A new Advisory Group (AG) will be formed to provide influence across the sector, provide strategic advice on delivery of the Hidden REF’s objectives, and to champion the Hidden REF. To ensure independence, the AG will elect its own chair and membership (10 members max) in line with terms of reference that will ensure the AG represents the diversity of the research community and a balance of senior representatives from the host universities and other influential people. Membership will be revised periodically to reflect project needs. The AG will receive a project update every 6 months and will meet in-person annually. The existing Hidden REF committee will continue to provide strategic insight and volunteer help with project delivery. Meetings with the committee will take place monthly.

Impact assessment: Equality and diversity

The purpose of the Hidden REF is to make the UK research environment more equitable and effective. This includes changing research culture in ways that benefit those whose work and roles have been marginalised by existing research evaluation processes. Evidence suggests that existing research culture has disproportionately negative impacts on those groups already facing structural societal disadvantage. We challenge this in a number of ways. Both the University of Southampton and the University of Bristol are signatories of Vitae’s Concordat to Support the Career Development of Researchers and Technician Commitment, and are active in implementing their recommendations, and Hidden REF supporters include both Vitae and Technician Commitment. The University of Bristol is a member of EDIS (Equality, Diversity and Inclusion in Science and Health). All participants in each work package will be required to have refreshed their EDI training within Year 1. The Advisory Group’s Terms of Reference will include oversight on the quality of EDI practice for the lifetime of the project. All Hidden REF activities in-person and online take place in line with a code of conduct that has been developed to create an inclusive environment for everyone. Taken together, these actions and commitments exemplify the ways that EDI considerations, principles, and guidelines are integrated into all work package activity and decision making. Without taking this active approach we believe the Hidden REF could not achieve its ambitions.