Responsible use of research metrics
The University of St Andrews is currently running a consultation with Schools on a set of principles for the responsible use of research metrics, including whether the University should sign up to the San Francisco Declaration on Research Assessment (DORA). These ‘draft principles for the use of indicators in research assessment and management’ were developed in response to growing international concerns amongst funders, governments and institutions on the way research indicators (or metrics) are interpreted and used, with their potential for unintended consequences.
A recent Open Access paper examining the validity of citation-based measures, which analysed >120 million papers, concludes:
“Our study’s extensive analysis of academic publications reveals why using citation-based metrics as measures of impact is wrong at the core…”
“Researchers do valuable work. Communicating the work to others is vital, and correctly assessing the impact of that work is essential.”
Michael Fire, Carlos Guestrin, Over-optimization of academic publishing metrics: observing Goodhart’s Law in action, GigaScience, Volume 8, Issue 6, June 2019, giz053, https://doi.org/10.1093/gigascience/giz053
San Francisco Declaration of Research Assessment (DORA) and similar principles
Several initiatives and organisations have developed principles and published recommendations, including the San Francisco Declaration on Research Assessment (DORA) community, the Leiden Manifesto and the UK Independent Review of the Role of Metrics in Research Assessment and Management, which published The Metric Tide.
In its revised implementation guidelines for Plan S, cOAlition S noted that there will be more emphasis on changing the research reward and incentive system. Its members comprising European research funders support the principles of the San Francisco Declaration on Research Assessment (DORA) and explicitly commit to adapt the criteria by which they value researchers and scholarly output.
UK Research and Innovation (UKRI), as a leading member of the Plan S coalition, has signed DORA. UKRI Chief Executive, Professor Sir Mark Walport, said: “Signing the declaration underlines UKRI’s commitment to open access and promoting best practice in research assessment.”
As of today over 1500 organisations and nearly 150,000 individuals have signed DORA. Apart from UKRI, UK signatories include the Wellcome Trust and the Universities of Oxford, Cambridge, Edinburgh, Dundee, Bristol, Imperial College London, University College London, King’s College London, Durham, Sheffield and York.
The Wellcome Trust’s new Open Access policy which comes into effect from 1 Jan 2021 will explicitly require institutions to commit to DORA or an equivalent set of principles, stating that “this is a new requirement to encourage organisations to consider the intrinsic merit of the work when making promotion and tenure decisions, not just the title of the journal or publisher.“
Elizabeth Gadd, Research Policy Manager (Publications) at Loughborough University argues that “institutions should retain the autonomy to implement responsible metrics in a way that fits with their own mission and values” and explains why Loughborough has developed their own principles based on the Leiden Manifesto.
Responsible metrics at University of St Andrews
At St Andrews we have held discussions at our Open Research Working Group, Research Committee and Academic Council, and we are now seeking feedback from our academic community on 5 principles, under the headings:
- Expertise
- Diversity
- Data
- Integrity
- Transparency
Following the engagement with Schools, we will develop these principles and recommend the best way to implement them in a manner consistent with University’s values and strategic ambitions. Crucially we will be considering how the principles can be implemented along with any required changes to current practice.