• The review will be conducted transparently. All submissions and notes from the panel will be published on(https://openreview.net/).
  • Names of the winners and highly commended submissions must not be disclosed until the winners have been notified at the award ceremony.
  • The process of assessment is subjective and discursive, it will not be easy, but panels will compare applications against each other (rather than seeking an absolute measure), keeping in mind the broad criteria of significance and visibility and the spirit of the Hidden REF.
  • The Chair, in discussion with the Panel, has the power to interpret the guidance as it is deemed necessary to be applied to the categories reviewed by their panel.
  • Names and institutions have been removed from entries, however, if a panel feels that they are unable to complete their work without names/institutions then these details will be reinstated.
  • The criteria of significance and visibility work as a pair. Please see the note below explaining more about ‘significance’. 
  • Significance that has been demonstrated should be assessed, rather than trying to assess the future potential for significance.
  • Chairs may choose to invite panellists to determine their own shortlist in the first instance, these shortlists will become the starting point for more detailed discussion.
  • Panel Advisers will take notes of key decisions, including a short commendation for the selected winner(s), and the processes involved in the selection.
  • Panel Advisers will advise panellists to choose a “small number” of outstanding submissions.
  • After the panels have suggested winners, the Panel Chairs and Panel Advisors will meet to discuss balance in the winner selection across the panels. If different panels have suggested different numbers of winners, the Chairs and Advisors may choose to change the number of winners (from winner to highly commended, and vice versa) to align the number of winners across panels.

Further notes on the criteria to inform but not determine panel discussions 

The criteria of  visibility and significance come as a pair and should be considered together in an holistic assessment of the application. Panels may consider the notion of significance of the input and of the output as a means to help enrich discussions of the significance of an application.

HEALTH WARNING: the following idea of input and output is to enrich discussion. We’re not looking to make the assessment more complicated!

What do we mean by significance of the ‘output’?

In each category the applications refer to some sort of output, e.g. a novel method for orienting and fixing fruit flies to slides, completion of a citizen science project, development and publication of a new standard, or indeed the production of more skilled people.

  • Visibility of the output: this relates to the level of traditional recognition the output has received, with preference to those outputs that were less visible.
  • Significance of the output: panels may find it useful to consider aspects such as geographical scale (e.g. local, national, global) and the numbers of people involved or impacted upon.

What do we mean by significance of the ‘input’?

Typically the output required the input of a person/role, team, group, or community. For example, a technician came up with the novel 3D printed widget, a team conducted the citizen science project, or a community who came up with the software standard.

  • Visibility of the input: Was the contribution of the person/team/group/community to the output visible to the community where the activity/achievement took place, or to wider society?  We are looking to identify where the contribution of the person/team/group/community was not visible to either the community where the activity/achievement took place, or to wider society.
  • Significance of the input: How significant was the input of the role/group/community to the achievement of the output?

Spirit of the Hidden REF 

Some applications may resonate strongly with a panels’ understanding of what the Hidden REF is about, e.g. creation of a citation standard enables the contributions of a software developer to be consistently and effectively acknowledged within their wider community. Where it’s not possible to distinguish between applications on the basis of significance and visibility then the spirit of the Hidden REF can come into play.