Submissions closed on 14th May. Panels were staffed by volunteer experts and formed by grouping together submissions from related categories. Panellists were chosen based on their experience of the submission categories. The same criteria for assessment will be applied across all panels with the aim of celebrating and highlighting the diversity of UK research.
Each panel includes an even number of panellists to promote agreement by consensus rather than voting. Each panel will be supplied with a panel advisor, drawn from the Hidden REF committee, who will be able to provide background information and details on the assessment process.
- Guidance for panellists on reviewing submissions
- Guidance for panellists on using Loomio to conduct reviews
- Timeline for review
Principles of the Hidden REF
- To include everyone and everything and leave nothing behind (visibility)
- To assess contribution across the diversity, richness, and variety of UK research (significance)
- All submissions will be celebrated and will be published on the HiddenREF website to raise awareness of the diversity of the UK research base.
- The best example or examples from each panel (i.e. the submissions located in the top left of the diagram) will be chosen as the winner. The panel also has the discretion to award Highly Commended to submissions as they see fit.
Panels
We collected the submissions into five panels. You can review the panels and the panellists who selflessly donated their time to help us select winners.
Roles
Panel Members and Chairs were suggested by various members of the HiddenREF organising committee and agreed by the committee.
- Panel Chair: responsible for smooth running of panel, facilitating discussion and building consensus, as well as the duties of a Panel Member
- Panel Member: to review entries and contribute to the panel discussion and decision making
- Panel Advisor: to support the Panel Chair and members in meeting the principles and practice of the hiddenREF
Guidance to Panel Chairs and Members
- The review will be conducted transparently. All submissions and notes from the panel will be published.
- Names of the winners and highly commended submissions must not be disclosed until the winners have been notified at the award ceremony.
- The process of assessment is subjective and discursive, it will not be easy, but panels will compare applications against each other (rather than seeking an absolute measure), keeping in mind the broad criteria of significance and visibility and the spirit of the Hidden REF.
- The Chair, in discussion with the Panel, has the power to interpret the guidance as it is deemed necessary to be applied to the categories reviewed by their panel.
- Names and institutions have been removed from entries, however, if a panel feels that they are unable to complete their work without names/institutions then these details will be reinstated.
- We discourage panelists from collecting further information on submissions, e.g. through web searches, or personal knowledge of the project/activity/individuals. If they do so, the panel will discuss how that further information should be used and keep a record of the rationale if any further information is taken into account.
- The criteria of significance and visibility work as a pair. Please see the note below explaining more about ‘significance’.
- Significance that has been demonstrated should be assessed, rather than trying to assess the future potential for significance.
- Chairs may choose to invite panelists to come up with their own short-list in the first instance, these shortlists will become the starting point for more detailed discussion. However, Chairs are free to determine their own approach.
- Panel Advisers will take notes of key decisions, including a short commendation for the selected winner(s), and the processes involved in the selection.
- Panel Advisers will advise panelists to choose a “small number” of outstanding submissions.
- After the panels have suggested winners, the Panel Chairs and Panel Advisors will meet to discuss balance in the winner selection across the panels. If different panels have suggested different numbers of winners, the Chairs and Advisors may choose to change the number of winners (from winner to highly commended, and vice versa) to align the number of winners across panels.
Further notes on the criteria to inform but not determine panel discussions
The criteria of visibility and significance come as a pair and should be considered together in an holistic assessment of the application. Panels may consider the notion of significance of the input and of the output as a means to help enrich discussions of the significance of an application.
HEALTH WARNING: the following idea of input and output is to enrich discussion. We’re not looking to make the assessment more complicated!
What do we mean by significance of the ‘output’?
In each category the applications refer to some sort of output, e.g. a novel method for orienting and fixing fruit flies to slides, completion of a citizen science project, development and publication of a new standard, or indeed the production of more skilled people.
- Visibility of the output: this relates to the level of traditional recognition the output has received, with preference to those outputs that were less visible.
- Significance of the output: panels may find it useful to consider aspects such as geographical scale (e.g. local, national, global) and the numbers of people involved or impacted upon.
What do we mean by significance of the ‘input’?
Typically the output required the input of a person/role, team, group, or community. For example, a technician came up with the novel 3D printed widget, a team conducted the citizen science project, or a community who came up with the software standard.
- Visibility of the input: Was the contribution of the person/team/group/community to the output visible to the community where the activity/achievement took place, or to wider society? We are looking to identify where the contribution of the person/team/group/community was not visible to either the community where the activity/achievement took place, or to wider society.
- Significance of the input: How significant was the input of the role/group/community to the achievement of the output?
Spirit of the Hidden REF
Some applications may resonate strongly with a panels’ understanding of what the Hidden REF is about, e.g. creation of a citation standard enables the contributions of a software developer to be consistently and effectively acknowledged within their wider community. Where it’s not possible to distinguish between applications on the basis of significance and visibility then the spirit of the Hidden REF can come into play.