The purpose of this position paper is to articulate how National Metrology Institutes (NMIs), and specifically the UK’s National Physical Laboratory (NPL), have a leading role to play in building confidence in Artificial Intelligence (AI) through metrology, the science of measurement. AI is transforming fields, including healthcare, finance, environmental monitoring, manufacturing and transportation, by advancing and automating data analysis, predictive modelling and data processing. As AI becomes more integrated into daily operations and decision-making processes, the need for confidence in the effective and sustainable use of data and AI becomes paramount, as does its potential impact on sustainability efforts. This paper will discuss how metrology can aid in building public trust and confidence in AI systems and facilitate AI adoption and standards development.
The global measurement infrastructure is the foundation upon which everything from blue skies research through to entrepreneurial activity depends – providing the confidence in making, manufacturing, innovating, investing, trading, and travelling. There is only one National Metrology Institute (NMI) in each country, and it is responsible for maintaining that country’s national measurement standards and provides traceability to the International System of Units (the SI) at stated levels of confidence – often called measurement uncertainty. The National Physical Laboratory (NPL) is the UK’s National Metrology Institute, founded in 1900. Measurement can be considered an invisible utility and for 125 years, our community has supported societies and economies.
NMIs have, for years, been using data science and scientific computing at NMIs, to support the physical measurement infrastructure with respect to uncertainty estimation and comparison evaluation. The role of an NMI is also to advance measurement science to ensure that the measurement infrastructure of the world remains relevant to societal needs, and as the world is transforming around us, we need to transform with it – some may say, stay ahead of it. Whilst we have had the physical infrastructure in place for over 100 years, in this world increasingly driven by data at the speed of Artificial Intelligence (AI), the same cannot be said of the assets required to establish digital measurement infrastructure.
Almost all the ‘game changing’ products, solutions, applications and services – across sectors and society - that will help solve the world’s most challenging issues (for example: climate change, a healthy society, energy resilience and so on), will rely on confidence in data and its employment in advanced digital technologies. However, as automated decision making, such as AI, requires less and less human intervention, great care must be taken to assess or quantify the quality of AI systems and their subsequent impact on the users of these systems, especially since the outputs of AI will often be context specific. NMIs have a key role to play in AI development, deployment and the assurance ecosystem to ensure we meet our aims, including economic prosperity, improved public services and increased personal opportunities.
Confidence in AI is crucial for its widespread adoption and ensuring trustworthiness. As stated by NIST, characteristics of trustworthy AI systems include being:
Metrological practices provide a transparent, well-defined approach needed to test and validate AI systems and their component parts, including data and algorithms, helping ensure that they perform as intended and do not perpetuate bias or inaccuracies. This transparency offered by robust measurement science helps build confidence in AI solutions across various stakeholders by providing traceable, scientifically proven and, ideally, internationally agreed methodologies and metrics for, for example, assessing the quality of systems, their components or outputs. NMIs such as the UK’s National Physical Laboratory (NPL), Germany’s Physikalisch-Technische Bundesanstalt (PTB), France’s Laboratoire national de métrologie et d'essais (LNE), Korea’s Korea Research Institute of Standards and Science (KRISS) and the USA’s National Institute of Standards and Technology (NIST), are national bodies which are part of an international system for ensuring the comparability of measurement standards and measurement capabilities. These bodies provide independent, government-backed capabilities to maintain their respective country's national standards and provide traceability to the International System of Units (the SI) at stated levels of confidence – known as measurement uncertainty. However, in an era of digitalisation and multi-modal data, the remit of NMIs is not limited to the measurement process itself. Complex digital systems are being developed which have their confidence underpinned by metrological robustness.
A major open question with AI systems surrounds the issue of how to trust or have confidence in the results produced by these systems. A common theme which surrounds this issue is the lack of transparency of the decision-making process behind AI and an inability to demonstrate provenance across the data and processing lifecycle. Since NMIs specialise in traceability of measurement information from a primary measurement standard to the point of use, this can be applied to the AI system lifecycle to allow users and developers to quantify the confidence in the data and algorithms present in AI. The independent nature of NMIs means that they can work noncompetitively across different companies and industries without self-interest impacting output, a significant advantage in an ever-competitive landscape of AI system development.
As arm’s length or government bodies, and a key part of National Quality Infrastructures, NMIs can help build public trust in AI system development and deployment by facilitating a central independent resource for AI system testing and evaluation. In addition, the international network of NMIs through international metrology groups or organisations such as EURAMET (The European Association of National Metrology Institutes) & BIPM (Bureau International des Poids et Mesures) can aid standardisation across borders alongside organisations such as BSI & ISO. The development of global standards for AI is essential to ensure consistency of vocabulary and procedures across systems and application areas. Metrology plays a key role in establishing these standards using rigorous science, which can facilitate global collaboration and innovation while ensuring the reliability, safety and security, sustainability and effectiveness of AI technologies.
NMIs, like regulators and governments, can have limitations when it comes to the fast-moving nature of AI system development, partly due to the emphasis on scientific thoroughness and rigour that comes with metrology, which can make rapid development difficult. They are generally government-funded bodies which do not have the financial or staff numbers of major tech companies or software houses. As such it is important to focus on the aspects unique to NMIs which can make the most impact on the international AI landscape and to recognise key areas where it is important for NMIs to either partner with other organisations or not to engage at all.
For example, it would be unwise to compete with Big Tech organisations or focus on large-scale algorithm deployment and maintenance, as the financial and time commitments would be unrealistic and does not align well with NMI remit, expertise and value-add. Similarly, ongoing conformity assessment of AI algorithms and provision of other AI governance processes is likely outside the remit of NMIs, instead outsourced to dedicated organisations or regulators with domain specific knowledge. For example, NIST’s landmark AI Risk Management Framework focuses heavily on the socio-technical aspects of AI system development and use. This means that the social, organisational and societal effects of AI are considered in the development of these systems, rather than solely the technical nature of traditional algorithm development. For most NMIs, it is unlikely that the socio-technical expertise can be found in house and suitable partners should be found to ensure that this vital aspect is still addressed.
NMIs can play a role in the ongoing assurance of AI systems. In addition to developing AI assurance tools such as techniques and standards for evaluating AI systems, NMIs can provide test, verification and validation throughout the development and deployment lifecycle. Doing so helps build a traceable picture of confidence – or uncertainty – in AI systems and their outputs. This is particularly important to give confidence in emerging regulation, standards and conformity assessment techniques. As the AI assurance ecosystem develops, NMIs are engaging with international groups such as the AI Quality Infrastructure Consortium, national quality infrastructure bodies, and governments to ensure that lessons from and the rigour of metrology are embedded.
NMIs generally have a track record for improving data quality through metrological robustness and traceability, and AI and the machine learning (ML) models and algorithms that enable it present several areas where this expertise is of paramount importance:
Different sectors and industries are developing AI capabilities at different rates. The most likely areas to be interested in the metrological robustness that NMIs provide are those which are more regulated due to their safety-critical nature and more stringent requirements for data and decision making. Example industries where quality and traceability are of paramount importance and necessary for operation are healthcare, security, pharmaceutical manufacturing, aerospace, finance, and evolving areas such as autonomous vehicles, biologic (pharmaceuticals) and renewable energy. SMEs/manufacturers new to adopting AI/digital technologies will also need techniques and guidance to trust autonomous decision making. Given the traditional role of NMIs in providing measurement assistance to SMEs, this is an excellent area for NMIs to aid the adoption and upskilling of AI techniques.
NPL has carried out a varied portfolio of work involving AI/ML, mostly through application focused collaborative projects. NPL is a founding member of the AI Standards Hub alongside the British Standards Institution and the Alan Turing Institute. A goal of this Hub is to unite parts of the UK quality infrastructure and other stakeholders like SMEs and researchers in advancing responsible AI through standards. An output of the collaboration within the AI Standards Hub was a NPL white Paper on A Life Cycle for Trustworthy and Safe Artificial Intelligence Systems, which details how to assess risks associated with AI system development and incorporate them into the software development process. Through the Hub we are also helping lead the international conversation around metrology for AI and AI governance, for example by delivering with partners the AI Standards Hub Global Summit and AI Action Summit side event.
NPL also led the writing of a ten-year research roadmap for the AI and ML topic within the Strategic Research Agenda (SRA) of the European Metrology Network for Mathematics and Statistics (EMN Mathmet). The SRA details mathematical and statistical issues that contribute to trustworthy and safe AI systems, and particularly the trustworthiness of a ML prediction in the context of metrology. It emphasises practical aspects of ML trustworthiness, including the importance of a quality framework which guides the choice of an ML model, supports verification and validation of the ML algorithms and software used, and addresses the quality and provenance of the data. Quality frameworks are also needed to support the reproducibility of results and auditability of ML models in metrology applications. In addition, the SRA highlights the importance of the specification of a standard interface for benchmarking, validation and certification of ML models.
NPL has also played a large role in at least four European Metrology Partnership projects focussed on applications of AI/ML: MedalCare (Metrology of automated data analysis for cardiac arrhythmia management - ECGs); MAIBAI (metrological framework for assessment of image-based AI systems for disease detection, mammography as an example); QUMPHY (uncertainty quantification for wearables); and BioAirMet (pollen classification). International collaboration is critical to ensure alignment of AI governance approaches, including measurement and metrics. NPL has collaborated with DSIT to develop an AI terminology tool which enables stakeholders in different countries to understand commonalities and crucially, differences in international frameworks for AI governance. NPL engages in global standards bodies to contribute our scientific expertise. For example, NPL contributed to the CEN/CENELEC technical report Environmentally Sustainable Artificial Intelligence.
NPL is investing in capabilities to reinforce our position as a world leader and technical authority for responsible and trustworthy AI, enabling confidence in AI for UK businesses, citizens and global markets. We will do this by delivering testing and evaluation services, tools and standards for understanding and quantifying trustworthy, responsible, safe and secure AI. This activity will be supported by our work in data quality, where we aim to be the point of reference for data metrics and data quality assessment, including synthetic data. NPL provides confidence in measurement data. The Data Science and AI Department provides methods, frameworks and guidance to give quantified measures of confidence in specific aspects of data. Key activities to deliver this goal include:
NPL will engage with stakeholders, including the public, policymakers, and industry leaders, to communicate the importance of metrology in AI. Activities will include hosting conferences, publishing research findings, and participating in public fora to raise awareness and build support for AI metrology initiatives.
NMIs have a critical role in developing and maintaining methods and standards to ensure AI technologies are robust, reliable and trustworthy. NPL will take a proactive leadership role in the integration of AI within metrology, focusing on developing tools, establishing standards, ensuring transparency, and fostering innovation. By doing so, NPL and other NMIs can ensure that AI technologies are harnessed effectively and ethically, benefiting a wide range of sectors.
