The Festival of Hidden REF brought together people who work in non-traditional research roles alongside policymakers, publishers and others involved in the research assessment. People attended to raise recognition for their unrecognised role or practice, and joined a community that is fighting for a more effective and fairer system of evaluating success in research.
The Festival took place on 21st September 2023 at M Shed in Bristol.
We are working on a whitepaper that will summarise the discussions that took place at the Festival. In the meantime, you can watch an overview from the Festival in the video above or listen to some of the guests and committee members discussing the Festival and the Hidden REF campaign in a special edition of the Code for Thought podcast. You can also take a look through the photos from the Festival. Slides from the talks are linked in the programme below.
Photos and the video are ©2023 The University of Edinburgh on behalf of the Software Sustainability Institute and are released under a CC-BY 4.0 licence for purposes of promoting the Hidden REF.
Programme
| Time | |
| 0900-0930 | Registration |
| 0930-0945 | Welcome and introduction to the Festival (Slides) |
| 0945-1015 | Lightning talks, Session 1 * Stephen Hill, Research England (no slides) * Kelly Vere, TALENT & The Technician Commitment (slides) * Emma Karoune, Alan Turing Institute (slides) * Lizzie Gadd, Loughborough University (video) |
| 1015-1045 | Coffee break |
| 1045-1200 | Guerrilla Groups, Session 1: * Roles: Hidden Roles: Perspectives, Paths, and Lived Experiences (Studio 2) * Categories: HiddenREF categories 2020 and 2024 (Board Room) * Evaluation 1: How do we evaluate non-traditional outputs? (Event Suite) * Plus… sticker guerrillas |
| 1200-1230 | Feedback from Guerrilla Group, Session 1 (Event suite) |
| 1230-1240 | Group photo – outside M-Shed entrance (weather dependant) |
| 1240-1330 | Lunch |
| 1330-1400 | Lightning talks, Session 2 * Julie Bayley, University of Lincoln (slides) * Pen-Yuan Hsing, University of Bristol (slides) * Cara Rodway, British Library (slides) * Katie Osgood, University of Portsmouth (slides) * Hollydawn Murray, Health Data Research UK (video) |
| 1400-1515 | Guerrilla Groups, session 2: * Communities: How do we engage with communities to get them to submit non-traditional outputs (Event suite) * Competition: Preparing for HiddenREF 2024 (Board room) * Evaluation 2: How do we evaluate non-traditional outputs? Towards Panel working methods (Studio 2) * Plus… sticker guerrillas |
| 1515-1545 | Coffee |
| 1545-1645 | Panel: discussion around the subjects raised during the Festival (Event suite) |
| 1645-1700 | Wrap up, thank you and future plans |
Details on Guerrilla Groups, session 1
Hidden Roles: Perspectives, Paths, and Lived Experiences
What were the trends within Hidden REF 2020 Hidden role category? What research roles exist that were overlooked by the HiddenREF and REF 2028? How can these Hidden roles be celebrated as categories for Hidden REF 2024?
HiddenREF categories 2020 and 2024
How representative were the Hidden REF 2020 categories? How can the Hidden REF 2020 categories be altered/combined? What other categories would you like to be seen for Hidden REF 2024?
How do we evaluate non-traditional outputs?
What aspects of excellence or esteem are shared by all non-traditional outputs? Is peer review an approriate tool for evaluating non-traditional outputs? What changes need to be made to peer review to allow fairer/transparent and competitive evaluation of non-traditional outputs?
Sticker guerrillas
At any time during the day, you can use the sticker boards to make suggestions that will guide our campaign. You can let us know what you think about the research assessment, suggest changes to our campaign, help us identify new groups to work with – anything that will help further our goals.
Details on Guerrilla Groups, session 2
How do we engage with communities to get them to submit non-traditional outputs
What criteria should we use to evaluate non-traditional outputs? Any alternatives to “significance”, and “Hidden-ness: used in the 2020 competition? What models do you know of that might be of use? Looking at feedback from the morning sessions, what alterations or differences in aspects of excellence and esteem are needed to value and evaluate non-traditional research outputs in these categories? What use are calibration methods for non-traditional outputs? How would you like to see calibration methods run, if at all?
Preparing for the HiddenREF 2024 competition
How do we convince HEIs and members of the research community to submit outputs to HiddenREF2024 and non-traditional outputs to REF2028? What roles are required to lobby HEIs to submit non-traditional outputs to REF2028? How can we achieve this together? How can we help communities to work together to lobby their organisations to value non-traditional outputs
How do we evaluate non-traditional outputs? Towards Panel working methods.
What is the reaction of the morning panels regarding changes to categories? Anything missed? If so, what characteristics of evaluators are preferable? How do we recruit them? How can we promote HiddenREF and convince members of the community to make submissions to the 2024 competition?
Sticker guerrillas
At any time during the day, you can use the sticker boards to make suggestions that will guide our campaign. You can let us know what you think about the research assessment, suggest changes to our campaign, help us identify new groups to work with – anything that will help further our goals.
Map of the venue
The Festival will be held at M Shed.
Registration will be on Floor 2. Plenary sessions will take place in the Event Suite on Floor 2 and Guerilla Groups will take place in the Event Suite (capacity 120), Board Room (capacity 32) and Studio 2 (capacity 40).
Code of conduct
The Festival of Hidden REF is dedicated to providing a harassment-free workshop experience for everyone. We do not tolerate harassment of workshop participants in any form.
Our code of conduct page gives more detail about our code of conduct and details for people to conduct is you have a question or an issue to raise.
Background
Running the 2021 Hidden REF competition was exhilarating! Submissions poured in from universities around the UK. Volunteers gave up their free time to help. Reviewers reported that some of the submissions had moved them to tears! We soon realised that this could not be left as a one-off competition: the Hidden REF had become a campaign. The Festival of Hidden REF is the next step in this campaign. Whilst the competition raised recognition of non-traditional roles and practices, the Festival will address problems with research assessment, help develop new policies and build new communities.
The objective of the Festival of Hidden REF will be to examine its co-working methods in 2021, towards hosting a subsequent Hidden REF in 2024. The Festival will also build a community for people who care about celebrating non-traditional research outputs, people and practices.
Towards achieving this aim, a working document will be continually curated during the Festival that will ensure the ideas, enthusiasm and energy shared during the day will inform the forthcoming HiddenREF festival. The document, in accordance with the values and community-focus of the HiddenREF, will be owned and therefore jointly-authored by all HiddenREF Festival participants. Though this document, unfortunately, will be neither REF- nor HiddenREF-able, it will be disseminated wider as a way to spread the ideas and insights of the HiddenREF community on how to improve research assessment, recognition and celebration.
Everyone is invited! If your Hidden Role doesn’t appear on our website, then we are especially interested in hearing from you. We will be joined by policymakers from across academia and government. These will include people involved in designing research assessment programmes, such as the REF, representatives from universities across the UK, and research assessment experts from around the world.
Volunteers
The Hidden REF is almost an entirely volunteer-run activity. We want this to change in the future, but whilst we’re building our community, we have relied on volunteers to run our committee and organise our events.
If you would like to help with the organisation and running of the Festival of Hidden REF, please sign up!
You can choose how to help with the Festival. The following list gives an idea of the range of activities that we will be conducting, but please let us know if you have another idea.
| Preparation | Finance |
| Budgeting | |
| Sponsor liaison | |
| Communications | Press liaison |
| Content development | |
| Social media | |
| Design/Infographics | |
| Photography | |
| Programme | Timetabling |
| Speaker liaison | |
| Code of Conduct | Code of Conduct committee member |
| Code of Conduct representative | |
| Logistics | Venue liaison |
| Accommodation liaison | |
| Volunteer organisation | |
| Assistant to the Chairs | |
| On the day | Front of House/Registration |
| Room Managers | |
| Technical | Website development |
| Tech runners |