The challenge
To ensure safety within the automated vehicle industry, adequate testing needs to be carried out to understand the performance of systems in different circumstances. Virtual testing is a key part of this, and industry-wide standardisation of testing techniques is required so that a common language can be developed regarding performance.
Standardisation in virtual testing requires an understanding of the accuracy of these models, which in turn requires that their uncertainties are reliably evaluated. Sensor models must also be incorporated into the different testing and simulation standards, and so requirements need to be put in place for sensor models as well.
As the UK’s NMI, NPL will play an important role in the measurement and modelling of these virtual environments and sensor models. This includes ensuring that the industry understands the importance of proper uncertainty quantification to generate high quality testing results.
The solution
NPL is building a sensor modelling framework to facilitate a standardised practice for sensor model design in the automated navigation stack. It is hoped that this framework will become the common language for modelling automated system sensors.
The framework includes the following steps:
Scoping exercise: We list the requirements for the sensor model, including which phenomena to model and operational design domain (ODD)
- Deciding on model and data: We decide what data is required to capture the required phenomena and what model would be adequate according to the scoping exercise.
- Deciding on method of training: We select a method for training and optimising the model by fitting the parameters so that the phenomena are modelled as required by the scoping exercise.
- Deciding on method for uncertainty quantification: We then explore how to properly quantify uncertainty in the model. This is an important component as it can then be used to understand the uncertainty in the results of the simulation.
- Model training and evaluation: Based on the above steps we train and evaluate the model.
These steps could be used to create a sensor model and they should, with proper feedback and research, eventually evolve into steps that could be followed by the industry as a standard practice.
We also demonstrated the use of the sensor modelling framework when creating simple models for camera and Lidar sensors. These illustrate the steps in the framework and, with the help of industrial feedback, could be evolved to be used in automated driving systems and virtual testing.
The impact
The sensor modelling framework is a step towards enabling an interface between a standardised testing environment and the different industrial automated driving systems.
Such standardisation across the industry could result in a common language which would improve collaborative research. It could result in the development of high-quality testing standards and improved communication between industry and suppliers. These testing standards would ultimately impact system quality and robustness and result in improvements to the safety performance of automated vehicles.