17/07/21
Cities are made possible by services from public and private providers, such as transport and utilities. Data sharing for these services has been difficult, plagued by restrictive licensing, fears about improper use, or to avoid competition.
This project provided a toolkit comprising software and guidance materials aimed at positioning local authorities as independent arbiters of smart city data, with a particular emphasis on real-time data.
It was developed as a collaboration between the Urban Observatory at Newcastle University which operates the largest deployment of urban sensing in the UK and the largest collection of open, environmental, real-time data in Europe, and smart city consultancy Urban Foresight, with each providing real-world use cases.
This project focused on addressing the barriers associated with data sharing, reluctance to rely on other organisations, business cases, and the value chain. The below aims and activities once combined address all of those barriers.
Creation of an integration platform that relates real-time IoT data with relevant context (for example, a bus tracker with the route hierarchy and street network).
Production of a report comprising background information and guidance on best practice for IoT deployments, contractual matters, methods of enabling data sharing, and approaches to achieving interoperability and standardisation.
The intended beneficiaries are the various levels of regional and local government, suppliers and organisations that interact with them (such as transport operators), and potentially wider users of IoT with a need for an integration platform.
Multiple strands of work were undertaken, which created the following individual components for the project.
Review of data platforms
Eight data platforms (Amsterdam, Dublin, London, Dubai, CityEye, Search-the-City, New York, Barcelona, Newcastle) were reviewed. The review addressed platforms’ city data sources, data source formats, whether the platform provides contextual user information, user login option, user preference support, persistence of data, analytics support and the planned future changes.
The review will be included in the final report of the project which will be published and made publicly available via a project website created by the Urban Observatory, and also the Urban Foresight website.
Timeseries platform and API
A generic approach to data brokers, storage of timeseries data (semi-structured) and publication through APIs that was first used for the Urban Observatories has been further developed, aligned with linked data standards (the W3C Semantic Sensor Network ontology), and released as open source.
Spatial API
To provide context to the above timeseries data, a related system that describes infrastructure such as streets and junctions by processing OpenStreetMap data was created. These are published as an API that can be combined with the timeseries data through linked data approaches.
Front-end interfaces
The design specification of the front-end interface has been agreed and the front-end is currently under development. It features three sections:
Data marketplace, allowing the user to search through publicly and commercially available APIs. The marketplace also feature a API playground for each publicly available API.
An interface targeted for the public where they can ask different questions about their city. The response to the question is accompanied by where the data is sourced, how the analysis is done and how the data can be used in combination with different datasets to provide better view on the searched matter.
A classic API documentation with visualised linked data which allows users to see how the available datasets are related.
Report, guidance, and training materials
A report has been written providing suggestions on data sharing strategies, business case development strategies, procurement specifications, and notes on data standards and interoperability. The aim of the report is to help local authorities with technical concepts of IoT solutions and to provide suggestions on how local authorities can benefit and harness the values of IoT solutions. Parts of the training materials are included in the front-end interface wherever required.
The project’s intentions were developing capability through software and guidance, which are the primary results.
The project has also identified barriers that were not in the original scope but would be worth exploring in future work:
Many of the standards in common use for real-time data systems do not provide detailed contextual information. When integrating these into the platform it can be laborious to manually reconcile two different datasets to create a single source of truth (eg free text fields for addresses); there is potential for automation to assist here but there are many possible scenarios and combinations, so it is a large undertaking.
Different needs were identified for users undertaking analysis and data science versus software engineers. A literature review and discussions with stakeholders have highlighted that simple datasets and bulk download are preferred for large analysis use cases (eg, transport planning).
Our interaction with the data providers in Dundee revealed their lack of engagement with semantic web concepts (linked data) in provision of their APIs.
Training material and case study: a report is being prepared to help local authorities in adopting IoT solutions and this will be made publicly available. It specifically discusses operating principles, procurement specifications, specification of scalable, standardised, open and interoperable software ecosystem, reflections on data standards, and risk management strategies in accordance with GDPR.
Software product/source code: timeseries platform and API – a service-oriented suite comprising brokers, an ingester, relational database, API server, and broadcast stream service.
Software product/source code: spatial platform and API – a series of spatial data transformations that generate descriptions of critical national infrastructure, and an API server that allows access to this in a way that is compatible with most GIS software.
Software demonstrator: the design specification of the software is agreed and the software is currently under development.
IP created: the software products have the potential to simplify future work or be the foundation for a commercial product. Urban Foresight is exploring whether there is potential for the software to be used in the delivery of their future work, while Newcastle University is likely to use the software resulting in the delivery of the PYRAMID project and future Urban Observatory systems.
Stakeholder engagement: app developers (including CityMapper), central government (MHCLG) and local government (Newcastle City Council, Nexus ITA) have been consulted as part of this work, the learnings from these interactions have been incorporated into the report, and they are likely beneficiaries following the ‘next steps’ described below.
Knowledge exchange: better understanding of added value of various technologies and data ontologies were very beneficial for Urban Foresight and the involved stakeholders. Many of the learnings are incorporated into data sharing practices and API creations within Urban Foresight and other involved stakeholders.
The software products and best practice guidance information produced in this project are in use within subsequent work, including:
Dundee MaaS: Dundee MaaS platform will use data ontologies and APIs created as part of CHARIoT. The integration framework will also be used to combine the provided datasets and API from various mobility providers.
Real-time flood risk modelling: a UKRI NERC funded project, PYRAMID, is using the CHARIoT integration framework as the basis for combining multiple datasets in a real-time flood risk model.
Tyne and Wear real-time passenger information: as part of a COVID-19 recovery project funded by the Ministry for Housing, Communities and Local Government, a consortium including Newcastle University is exploring ways of improving access to public transport information to counteract a worrying trend towards private vehicle usage. Within this work the CHARIoT platform will be used to consume and syndicate bus arrival time data modelled by Nexus, the area’s transport authority, for reuse by apps including CityMapper.
This collaboration was a great opportunity for Urban Foresight to become more aware of the available data standards, their benefits, and limits for integration. The project also helped them to gain familiarity with a wide range of practices in the design and delivery of urban data platforms.
Securing access to some of the third-party data at an earlier date, because the technical aspects of doing so have proved quite cumbersome for real-time data, would have avoided delaying the CHARIoT project’s final deliverables.
Authoritative datasets that could be used by everyone to associate their timeseries and IoT data with context would be hugely beneficial – this is beginning to happen with the release of open UPRN (unique property reference numbers) data – but the same type of unique identifiers under an open licence are required for all infrastructure and assets to create a core national ‘data’ infrastructure without ambiguity.
Through Pitch-In we’ve engaged with and collaborated with a wide range of public and private organisations in a way that simply wouldn’t have happened otherwise – they have shared with us their business challenges and problems and we have begun to solve them through Pitch-In.
The work is not done, and I expect we will continue to collaborate for many years to come.
Professor Phillip James and Dr Luke Smith – Newcastle University