Statement on the Responsible Use of Research Metrics

1. Purpose/Abstract 

The University Statement on the Responsible Use of Metrics is intended to: 

• Provide the principles for the use of quantitative evaluation of research outputs as a complementary approach to peer review which remains the method of choice for assessment of research quality
• Provide the principles to support the expectation that transparent qualitative and quantitative information is used to support and inform expert human academic judgement
• Reflect the different established practice in differing subject disciplines
• Demonstrate to external stake holders, including funding agencies that the University is aligned to the principles of responsible metrics as reflected for example in the Leiden Manifesto, the San Francisco Declaration on Research assessment (DORA), the Metric Tide and Hong Kong Principles
• Support the University’s compliance with external benchmarks and concordats, such as the Concordat to Support Research Integrity and the Concordat to Support the Career Development of Researchers

 2. Introduction 

The University of Derby values the diverse and excellent communities of academics and students across the University who contribute to the Innovation and Research Strategy 2020 - 2026. It is therefore critical that information used to assess and monitor research activity is used appropriately by the University. 

The University of Derby is committed to the critical role that peer review and expert judgements plays in the assessment of research, while taking into account the important role that quantitative metrics (bibliometrics and altmetrics) plays in complementing and supporting decision-making. The University also recognises the important role that metrics have in facilitating research and improving research visibility and impact. 

The University acknowledges that no quantitative data source(s) alone can provide a complete measure of research quality or activity. The University Statement on the Responsible Use of Research Metrics is a guide to responsible research assessment using metrics, which are based primarily on the recommendations of the Leiden Manifesto (Hicks et al. 2015). The ten Leiden Principles take a broad approach to the responsible use of metrics across a range of disciplines, including arts, humanities and social science. However, in putting a University of Derby focus on these principles the recommendations of the Metric Tide (Wilsdon et al. 2015), the Hong Kong Principles (Moher, 2020) and the San Francisco Declaration on Research Assessment (DORA) have also been drawn upon. The first having a REF focus and the latter two an emphasis on STEM.

3. Scope

The University Statement on the Responsible Use of Metrics applies to all employees, students and visiting researchers of the University, including persons holding honorary University appointments.

This Policy is subject to the oversight of the University Innovation and Research Committee.

4. Definitions

For the purpose of this Policy:

• Altmetrics: (based on the Metric Tide, 2015 definition) are non-traditional metrics beyond citation count, but also downloads, social media shares and other measures of the impact of research outputs
• Bibliometrics: (based on the Metric Tide, 2015 definition) is the quantitative analysis of scholarly publications, including patents
• Research: (based on the Research Excellence Framework, 2021 definitions) is defined as original investigation undertaken in order to acquire knowledge and understanding, which is effectively shared
• Research outputs are defined as a dissemination, publication, presentation, exhibition or pathway by which research is made available beyond the researcher
• Research publications include theses, books, book chapters, journal articles and their underlying data, reviews, conference proceedings and monographs
• Peer Review: A process of research assessment based on the use of expert deliberation and judgement
• The University: All Colleges, Departments and Group Companies of the University of Derby

5. Policy Statement

The principles of the University Statement on the Responsible Use of Metrics.

• Quantitative evaluation should support qualitative, expert assessment.
  The University of Derby recognises the value of quantitative indicators, bibliometric-based metrics and altmetrics (where available) to support qualitative, expert peer review. Indicators will not supplant expert assessment of both research outputs and the context in which they sit.  Indicators may be used for collective assessments at levels from research units to the Institution as a whole
• Measure performance against the research missions of the institution, group or researcher.
  The Innovation and Research Strategy 2020-2026 drives the University’s ambition to deliver research of the highest quality and in the context of being an ‘Applied University’ where our students have the opportunity to put theory into practice. At the same time, the visibility of the University’s research is critical to maximising its impact on the communities it serves. To this end, indicators around the quality of the output (e.g. journal or conference), collaboration levels and citedness of outputs may provide an indication of institutional performance
• Keep data collection and analytical processes open, transparent and simple.
  Simplicity is a virtue in an indicator, because it enhances transparency. However, there is a balance to be struck between simple transparent indicators, that may not accurately reflect the complexity of research and more complex indicators, whose application is more difficult to replicate across different areas of research. Therefore, specific metrics should be selected which can be used reliably and consistently with their advantages, and disadvantages clearly communicated
• Allow those evaluated to verify data and analysis.
  The publication and citation tools used to collect and monitor research publication data at the University of Derby will be made openly available.  Academics are therefore able to see the data relating to themselves and to make corrections where necessary.  Staff managing publication systems will also endeavour to ensure that data are as accurate and robust as possible
• Account for variation by field in publication and citation practices.
  It is recognised that research practices in disciplines vary widely, and bibliometric indicators serve some disciplines better than others. For example, citation tools are currently based on journal and conference outputs, not monographs or other forms of output.  Although still relevant if seeking funding partners international collaboration indicators will be less relevant to disciplines where academics tend to publish alone rather than in teams.  In line with best practice, indicators will be normalised wherever appropriate and based on percentiles rather than averages, as a single outlier may skew the data.  The availability or otherwise of bibliometric data will not drive the University’s decision-making about research activities, priorities or performance either individually or collectively
• Protect excellence in locally relevant research.
  It is recognised that most citation-counting tools are inherently biased towards English-language publications.  It is important that academics producing work in languages other than English are not penalised for this
• Base assessment of individual researchers on a qualitative judgement of their portfolio.
  The University of Derby acknowledges how indicators are affected by career stage, gender, other protected characteristics and discipline and will seek to consider these factors when interpreting metrics.  It is also recognised that academics undertake a wide range of research communication activities, not all of which can be easily measured or benchmarked.  Therefore, consideration will be given to as wide a view of expertise, experience, activities and influence as possible
• Avoid misplaced certainty and false precision.
  The University of Derby commits to using multiple indicators to provide a more robust and wide-ranging picture. Indicators will avoid false precision; for example, metrics may be published to three decimal places to avoid ties but given the limitations of citation counts, it is inappropriate to distinguish between entities on the basis of such small difference
• Recognise the systemic effects of assessment and indicators
  It is accepted that any measurements may in themselves, affect the system they are used to assess through the inevitable incentives they establish.  To minimise such effects, a suite of indicators will be used and an equality impact assessment undertaken wherever practical.
• Scrutinise indicators regularly and update them
  As the research activity of the University and the external environment develop, the bibliometric indicators use will be reviewed in line with the REF period and revised where appropriate, by the University Innovation and Research Committee

6. Policy Implementation

Since 2019, when Academic Board approved the University’s Responsible Metrics Statement, the Publication and Bibliometrics Working Group (a sub-Committee of UIRC) have worked to embed and monitor Institutional awareness of Responsible Metrics.  Specific activities have included:

• The inclusion of defined and appropriate bibliometrics information on the REF dashboard and its use in preparing UoA submissions. Clear guidance was given to UoA leads/teams that quantitative metric-based consideration must always be used alongside appropriate qualitative considerations
• Development of a responsible metrics resource on the URKEO intranet site supported by the Annual Researcher Development Programme which is open to all and Library training sessions
• Identification and training of Responsible Metrics Champions in each of the Colleges to support raising and maintaining awareness of the importance of Responsible Metrics
• Recognition within the University’s Innovation and Research Strategy 2020-2026 of the significance of responsible metrics. In the context of research governance, there should be similar recognition in the developing University Learning and Teaching Strategy
• Annually from January 2022, there will be a CRC standing agenda item on responsible metrics.  Feedback from CRCs will be reviewed at the March UIRC meeting to ensure the use of research metrics are considered and as appropriately monitored
• The University’s recruitment guidance is being updated to reflect the use of responsible metrics in the recruitment process. Additionally, the internal promotion/conferment process will include guidance on the use of responsible metrics

7. Sanctions

Failure to comply with this policy should be addressed in the context of the University’s Research Misconduct Policy and Procedures.

8. Related Documentation

• University Publication and Open Access Policy. This policy provides the principles which will underpin any research publication associated with the University
• Additional information and resources associated with the Responsible Use of Metrics are available on SharePoint

The current version of this Policy is available on Policy Hub

9. References

• DORA
• Hicks, D., Wouters, P., Waltman, L., de Rijcke, S. and Rafols, I. (2015) Bibliometrics: The Leiden Manifesto for research metrics. Nature 520: 429-431
• Moher, D., Bouter, L., Kleinert, S., Glasziou, P., Har Sham, M., Barbour, V., Coriat A-M., Foeger, N. and Dirnagl, U. (2020) The Hong Kong Principles for Assessing Researchers: Fostering Research Integrity.  PLOS Biology 18 (7) e3000737
• Wilsdon, J., Allen L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., Kain R., Kerridge, S., Thelwall, M., Tinkler, J., Viney, I., Wouters, P., Hill, J. and Johnson, B. (2015) The Metric Tide. The Independent Review of the Role of Metrics in Research Assessment and Management. Sage Publications Ltd. London