01/06/21
Within the Pitch-In energy theme there was much discussion on the need for some unified set of tools and methods around data cleaning, that is, if intending to utilise the outputs from swarms of IoT sensors then it must be understood that a fraction of them at any one time will be outputting anomalous or corrupt data.
There must therefore be a clear strategy around data cleaning, though this should actually be a discussions around data readiness, ie how will we make the data that is supplied from IoT ready, for consumption in a way that is reliable, repeatable and with which we can have confidence in results gained.
As part of the process around data readiness we need to understand what standards are in use already within the sector and what methods may already be widely utilised in relevant sectors. From this, the project intended to produce two relevant outputs of use by the consortia.
The main aim of the project is to identify methods to significantly improve understanding how IoT (anomalous, dirty, even corrupt) sensor data can reliably be integrated into system decision-making processes. This project seeks to bring together and share global best practices, with a view to specifying a toolset for researchers that an activity could develop.
Through the work of the recruited researcher the following activities were undertaken.
A survey the metadata requirements for IoT device sources in energy was performed to create a dataset for future analysis. From this, a catalogue of currently available data standards was created: ‘Existing IoT metadata standards’, to understand useful transforms and currently adopted standards.
Following this, the second deliverable was created, ‘Applicable data cleaning methods and schemas for their use’. This addresses the aspect on available data cleaning methodologies and how to embed the information in an interoperable way in the IoT landscape. It built on the results of the report on metadata standards mentioned above, and the application of the FAIR principles to an IoT smart energy system.
Finally, this report outlines how a skeleton architecture might look, which includes capturing the application of data cleaning methodologies at relevant levels, while maintaining a FAIR approach to improve or achieve data readiness in the IoT smart energy domain.
The partners within the project disseminated the survey and contacted relevant stakeholders of which they were aware to complete the survey as well as contributed to the development of the deliverables.
The deliverables of the project highlighted how there are a number of well understood and utilised data standards already within the sector and that overall it is important that we note that IoT overall is developing well in this area and ‘energy’ will be just another use case not something special.
For data cleaning the available tools and techniques as well as the application of FAIR principles around IoT data will in some areas be challenging though not insurmountable. We highlighted a possible future architecture to improve the data readiness within an IoT for energy domain. We did not ourselves continue a follow on project.
D1 – Smart energy IoT device metadata survey
D2 – Existing IoT metadata standards, Drescher, 2019
D3 – Applicable data cleaning methods and schemas for their use, Drescher, 2019
Mr Michel Drescher was employed for the duration of the project on an uplifted contract beyond his previous part-time status.
None currently though work within LEO includes discussions about the use of MQTT that was mentioned as one of the leading contenders for utilization in the energy domain by D2.
The deliverables give a clear indication of the current state of the art in the area to the extent that even now we are seeing activities in the IoT space realizing that MQTT is a good an appropriate standard for messaging within the domain.
Surveys are extremely hard to engage stakeholders that are unclear of the value even if they are well engaged with the survey developers beforehand in other areas. It would have been more useful to hold more targeted webinars/workshops specifically around these topics. Face-to-face meetings always generate better results though if the right people can be encouraged to attend.
Further dedicated resource to allow the researcher to spend more dedicated time on the activity, 30% is not a large amount when compared to other activities on which they were engaged.
Pitch-In allowed us to research into an area in which we had no specific current activities but which gaining and understanding of the local landscape was essential in the longer term where we consider how smart energy and systems may develop in the future.
This is most particularly important when we consider how data from sensors and system may be utilised in real time for network functions including detailed business relationships which must be executed correctly and the data from which must be ready for utilisation both within the service itself but also when considering arbitration of disputes, breakdowns or system errors.
Professor David Wallom – University of Oxford
The University of Oxford
The University of Sheffield
The University of Newcastle