Improving reproducibility in research
NPL, NIST, PTB, LGC KRISS, NIBSC and the BIPM brought together experts from the measurement and wider research communities (physical scientists, data and life scientists, engineers and geologists) to understand the issues and to explore how good measurement practice and principles can foster confidence in research findings. NPL hosted the workshop Improving reproducibility in research: the role of measurement science in May 2018. The workshop involved 63 participants from metrology laboratories (38), academia (16), industry (5), funding agencies (2) and publishers (2). The participants came from UK, US, Korea, France, Germany, Australia, Bosnia and Herzegovina, Canada, Turkey, and Singapore.
We live in a world in which it is increasingly easy to collect enormous amounts of data, in which research frequently spans physical, chemical, and biological sciences. Despite these advances, many scientific research results in recent years have proven to be difficult to reproduce.
Key recommendations and actions from the workshop
A compelling theme from the workshop was that Metrology Institutes, National Metrology Institutes (NMIs) and Designated Institutes (DIs), are uniquely placed to improve reproducibility in research owing to their expertise in measurement and associated measurement uncertainties and their role as impartial and independent bodies. The Metrology Institutes also have a responsibility to be role models for the research community and to support their endeavours through leadership of inter-comparisons (replicability) studies.
Organising committee
- Robert Hanisch, National Institute of Standards and Technology (NIST), Gaithersburg, USA
- Anne Plant, National Institute of Standards and Technology (NIST), Gaithersburg, USA
- Martyn Sené, National Physical Laboratory (NPL), Teddington, UK
- J.T. Janssen, National Physical Laboratory (NPL), Teddington, UK
- Ian S. Gilmore, National Physical Laboratory (NPL), Teddington, UK
- Michael Oldham, National Physical Laboratory (NPL), Teddington, UK
- Alistair Forbes, National Physical Laboratory (NPL), Teddington, UK
- Stephen Ellison, LGC Group (LGC), Teddington, UK
- Hendrik Emons, European Commission (EC), BE
- Sascha Eichstaedt, Physikalisch-Technische Bundesanstalt (PTB), Berlin, DE
- Christian K. Schneider, National Institute of Biological Standards and Control (NIBSC), South Mims, UK
- Hyun Kyoon Lim, Korea Research Institute of Standards and Science (KRISS), Daejeon, KR
The following presentations from the workshop are available on request.
Request presentation
Exploration of the Problem
| Session Chair: Robert Hanisch |
General Presentations |
| Mark Thomson (STFC Executive Chair) |
Reproducibility Challenges in UK Science |
| Eric Lokenes |
Measurement, Statistical Intuitions and the Replication Crisis in Science |
| Simon Cox |
Precision in Nomenclature, Vocabularies
|
| Jim Warren |
Materials Science |
| Stephen Ellison |
Chemistry |
| Michael Hildreth |
Physics |
| Ian Gilmore |
Mass Spectrometry Imaging |
| June Lau |
Microscopy |
| Steve Collins |
Synchrotron Beamlines (photon, neutron) |
| John Elliott |
Biosciences |
| JT Janssen |
Quantum |
| Keith Jeffery |
Metadata and Data Models |
| Robert Hanisch |
Data Management and Provenance |
| John Henry Scott |
Algorithms, Software and Data |
| Eva Campo |
Data Preservation and Reuse Goals – an NSF Perspective |
| Helen Glaves |
(Machine Actionable) Data Management Plans |
NMI Resources
| Antonio Possolo |
Trustworthy Measurement for Reproducible Research |
| Martin Milton |
Reproducibility is our business: The work of metrologists to provide comparable measurements |
| Robert Hanisch |
NIST SRD |
| Hyun Kyoon Lim |
KRISS SRD |
| Ray Plante |
Metrology Resource Registry |
| Jeanita Pritchett |
Reference Materials |
| Graham Sims |
VAMAS and Prenormative Data |
| Jeanita Pritchett |
Metrology Education and Accreditation |
| Stuart Chalk |
Units of Measure |
| Jan Jensen |
Design of Experiment |
| Alistair Forbes |
Modelling, Simulation, and Analytical Methods |
| Colin Longstaff |
Measurement Practices in Biology |
| Sir Jim Smith |
A Wellcome Approach to Reproducibility |
NMI Actions
| Leslie McIntosh |
ML-Based Assessment of Confidence in Measurement |
| Natalie de Souza |
Editorial Policies Regarding Reproducibility in Scholarly Publications |
| Anne Plant |
Summary Observations |
NPL have had an article published in Nature magazine 'Metrology is key to reproducing results'.
Read the Nature article
Download the following documents:
Responding to the growing issue of research reproducibility
Improving Reproducibility in Research The Role of Measurement Science