Opendata, web and dolomites

Report

Teaser, summary, work performed and final results

Periodic Reporting for period 1 - IoTCrawler (IoTCrawler)

Teaser

Efficient and secure access to Big IoT Data will be a pivotal factor for the prosperity of European industry and society. However, today data and service discovery, search, and access methods and solutions for the IoT are in their infancy, like Web search in its early days...

Summary

Efficient and secure access to Big IoT Data will be a pivotal factor for the prosperity of European industry and society. However, today data and service discovery, search, and access methods and solutions for the IoT are in their infancy, like Web search in its early days. IoT search is different from Web search because of dynamicity and pervasiveness of the resources in the network. Current methods are suited for static or available resource repositories. There is yet no adaptable and dynamic solution for effective integration of distributed and heterogeneous IoT data and support of data reuse in compliance with security and privacy needs, thereby enabling a true digital single market. Previous reports show that a large part of the developers’ time is spent on integration. In general, the following issues limit the adoption of dynamic IoT-based applications:
- The heterogeneity of various data sources hinders the uptake of innovative cross-domain applications.
- Data records without embedded metainformation lack of utility for expending its usefulness across platforms.
- Missing security and neglected privacy present the major concern in most domains and are a challenge for constrained IoT resources.
- The large-scale, distributed and dynamic nature of IoT resources requires new methods for crawling, discovery, indexing, physical location identification and ranking.
- IoT applications require new search engines, such as bots that automatically initiate search based on user’s context. This requires machine intelligence.
- The complexity involved in discovery, search, and access methods makes the development of new IoT enabled applications a complex task.

Work performed

The objective of this research and innovation is to develop the next generation of Internet search engines that support crawling, discovery, search and integration of IoT data. However, an IoT search engine will not be effective if it does not provide tools and mechanisms to respond to machine-initiated search, offer adaptive and dynamic solutions for resource ranking and selection, and more importantly enable integration of the IoT data and services into the analytics pipelines. IoTCrawler develops distributed crawling and indexing mechanisms to enable real-time (or near real-time) discovery and search of massive real world (IoT) data streams in a secure and privacy- and trust-aware framework. It also provides quality analysis for the data streams, information abstraction and develops query and search, ranking and selection enablers to respond to spatial and multi-modal data queries in future communication networks. This creates a scalable search engine for future Internet. As the Web search engines and, in particular, Google\'s PageRank algorithm changed the way people find and access the information on the Web (by deep crawling/indexing and utilising the links between the Web pages and documents), IoTCrawler is changing the way the data (especially new forms such as IoT data) can be published, discovered and accessed in large-scale distributed networks. This implies paving the way for creating new applications and services that rely on ad-hoc and dynamic data/service query and access. Overall, the research focuses on the following key objectives:
1. Providing a framework and specifying the requirements and components for a secure and privacy-aware framework for crawling, discovery, search, access and integration of heterogeneous IoT resources.
2. Providing enablers for security-, privacy and trust-aware discovery and access to IoT resources in constrained IoT environments.
3. Providing distributed discovery, crawling, indexing and ranking methods that use spatial, temporal and thematic attributes of the data streams to create indexes and provide solutions that allow dynamic updates of the indices.
4. Designing adaptive intelligent methods that can process quality of multivariate and multi-modal IoT data and provide knowledge-based and context-aware query and search and mechanisms for IoT data/services.
5. Developing market-ready solutions to support discovery, integration and exploitation of dynamic and ad-hoc resources in large-scale and distributed IoT frameworks.
6. Providing open-source and common toolkits and enablers to encourage rapid application and service development based on the IoTCrawler innovations and solutions.
7. Applying the research results in smart city, smart energy, social IoT and industrial automation domains.
8. Developing and proof of market solutions in selected domains and use-cases.
9. Evaluating the developed applications and services for technical validity, societal impact and user acceptance.

Final results

IoTCrawler has the potential to change the way the industry and public use the Internet of Things. One of the main issues today is the fragmentation and creation of isolated Intranets of Things solutions that get lost in organizational structures, across vendors and owners. IoTCrawler will create technologies and solutions that allow for access and interoperability across different platforms by developing dynamic and reconfigurable solutions for discovery and integration of data and services from legacy and new systems, adaptive, privacy-aware and secure algorithms and mechanisms for crawling, indexing and searching in distributed IoT systems. This will be demonstrated in several show cases based on a selected number of scenarios out of the 22 identified scenarios at WP2 to represent various thematic areas such as Industry 4.0, Smart Energy and Smart Cities to Social IoT. Each show case will act as a first of its kind case for the hosting organizations to demonstrate business potential of the show case and subsequently act as a sales argument for future cases using the IoTCrawler technologies. By bringing in end-users and relevant stakeholders from the ecosystem for each domain in the co-creation activities starting in M12, IoTCrawler will gather an understanding about the needs and challenges that IoTCrawler can provide solutions for and gather insights about the context that the showcases should operate in. These insights will increase the chances of creating showcases that create a real impact and serve as the foundation for creating relevant business models and value propositions.
IoTCrawler is an EU H2020 project that addresses the above challenges by proposing efficient and scalable methods for crawling, discovery, indexing and ranking of IoT resources in large-scale cross-platform and cross-disciplinary systems and scenarios. It develops enablers for secure and privacy-aware discovery and access to the resources, and monitors and analyses QoS and QoI to rank suitable resources and to support fault recovery and service continuity. The project evaluates the developed methods and tools in various use-cases, such as Smart City, Social IoT, Smart Energy and Industry 4.0.
The project aims to create scalable and flexible IoT resource discovery by using meta-data and resource descriptions in a dynamic data model. This means that searching actions could result in non-optimal results that could fit the user expectations. For this, the system should understand the user priorities (which are often machine-initiated queries and search requests) and provide the results accordingly by using adaptive and dynamic techniques.

Website & more info

More info: https://iotcrawler.eu/.