#	Pagina
attuale pagina	/open-h2020/projects/193663/results.html

Report

Teaser, summary, work performed and final results

Periodic Reporting for period 3 - Hi-EST (Holistic Integration of Emerging Supercomputing Technologies)

Teaser

Summary

Supercomputers are critical infrastructures to address Grand challenges in the field of Bioinformatics, Physics, and Earth Sciences. Supercomputing is a crucial asset for the EU\'s innovation capacity, being a worldwide market worth EUR 14 billion, with Europe representing a 35% of it, from which two thirds of this market depends on public funding. The goal of Hi-EST is to perform a significant advance in the field of methods, mechanisms and algorithms for the integrated management of heterogeneous supercomputing workloads. This will result in a more efficient management of the infrastructure, in an automated way that will continuously adjust the number and type of resource allocated to each workload. Hi-EST aims to address this problem for the next generation infrastructures and workloads, entering into an still unexplored space: the intersection of classic HPC workloads and data-driven, real-time and interactive applications on the one hand; and a set of emerging technologies combining persistent memories, key/value stores, RDMA-based devices, and GPUs on the other. In this scope, Hi-EST will take advantage of the ultra-large size of exa-scale systems to develop advanced real-time performance learning techniques and data and task placement algorithms to address this NP-hard problem for this novel scenario. The placement decisions will be continuously enforced through the use of Software Defined Environments.

In summary, Hi-EST aims to achieve the following objectives:
1. Hi-EST plans to advance research frontiers in Adaptive Learning Algorithms by proposing the first known use of Deep Learning techniques for guiding task and data placement decisions, and the first known Adaptive Learning Architecture for exa-scale Supercomputers.
2. Hi-EST plans to advance research frontiers in Task Placement and Scheduling by studying the first known algorithm to map heterogeneous sets of tasks on top of systems enabled with the Active Storage capabilities, and by extending unifying performance models for heterogeneous workloads to cover and unprecedented number of workload types.
3. Hi-EST plans to advance research frontiers in Data Placement strategies by studying the first known algorithm used to map data on top of heterogeneous sets of key/value stores connected to Active Storage technologies, as well as by proposing the first known uniform API to access hierarchical data stores based on key/value stores.
4. Hi-EST plans to advance research frontiers in Software Defined Environments by developing the still inexistent vocabulary to describe Supercomputing workloads and an associated language, and by creating the first known placement algorithm that combines data and task placement into one single decision-making process

Work performed

The project has developed different activities so far. These activities are driven by the use cases proposed in the DoA: genomics, which in fact has already resulted in a patent request to be filed, and streaming, which currently is in the final stages of filing two more patent requests. The use cases have been used to drive the other research activities of the project, that resulted so far in three research publications and several others that are in the process of being submitted at this point.
The research activities have produced a good knowledge of the path to follow in the next years in the context of the project to achieve the goals initially established and there are no problems foreseen that should interfere in the normal progress of the project. So far have been explored new methods to model performance metrics in HPC and BigData workloads using different learning technologies, and a lot of effort has been put on defining how to orchestrate workloads in Software Defined Infrastructures, producing new algorithms and methods that will be submitted for publication in the next few months. Three conference publications have been produced in this period.

Final results

A common belief in the supercomputing field is that the future bottleneck for many data intensive scientific advances will be on the technological approach used currently used to process data. This situation is clearly visible observing at the sustained growth rate of the GenBank genome database from NCBI, or the experiences in the Atlas Project in the process to discover the Higgs boson at the Large Hadron Collider (LHC). In both cases, the scientific grand challenge resembles more a data mining problem than a classical CPU-intensive supercomputing project. But they need, in both cases, the ultra-high capacity of supercomputers to address the problems that each project aims. The extremely large scale of next generation exa-scale supercomputers results in that every monitoring effort becomes a challenge also in terms of data management. But the same challenge is also an opportunity, because the amount of information that can be collected in real time from an exa-scale infrastructure opens an opportunity for advanced learning techniques to generate knowledge about the applications being run. Another source of changes in the supercomputing domain is the enormous interest by the Public Sector and Social Scientists on the so called Smart cities and the Internet of Things (or more recently the Internet of Everything). These workloads require the processing unbounded data streams that pose enormous challenges to the processing centres, demanding in many cases the computing capacity of supercomputers.
For this reason, there is an urging need to perform a significant advance in the field of methods, mechanisms and algorithms for the integrated management of heterogeneous supercomputing workloads. This is a huge challenge since management of a homogeneous set of distributed workloads on homogeneous infrastructures is already an NP-hard problem to solve. The level of dynamism expected for future generation supercomputing significantly raises the complexity of the problem. Addressing this grand challenge is the ultimate goal of the Hi-EST project.

In particular, Hi-EST plans to advance research frontiers in four different areas:
1. Adaptive Learning Algorithms: by proposing a novel use of Deep Learning techniques for guiding task and data placement decisions;
2. Task Placement: by proposing novel algorithms to map heterogeneous sets of tasks on top of systems enabled with Active Storage capabilities, and by extending unifying performance models for heterogeneous workloads to cover and unprecedented number of workload types;
3. Data Placement: by proposing novel algorithms to map data on top of heterogeneous sets of key/value stores connected to Active Storage technologies; and
4. Software Defined Environments (SDE): by extending SDE description languages with a still inexistent vocabulary to describe Supercomputing workloads that will be leveraged to combine data and task placement into one single decision-making process.