R-CCS Cafe - Special Edition (Nov 24, 2023)

Title:
Huge Data Flow Management for large scale HPC system -Seamless data access in Edge-to-Core-to-Cloud–
Abstract:
The speaker had developed FEFS as a director of development division. FEFS was developed for K-Computer. FEFS was the largest file system in the world at the time, controlling tens of thousands of nodes and thousands of storage units. It was the world's largest distributed parallel file system on an open-source basis. Many supercomputers, including Fugaku, still operate it today. FEFS has a long track record as a local file system for handling large data for supercomputers, but the current technology trend is to perform various applications such as AI in various computing environments. This technology trend is called as “Edge-to-Core-to-Cloud”. But it has many challenging to manage the huge data. The speaker will introduce current technology trends, and those problems, and discuss his development plan as Huge Data Management with the knowledge of large-scale fast file system obtained by FEFS development. The new technology trends (WekaIO, Vast Data,DAOS,,,etc..) are trying to create a unified file system space where the end user doesn't know where the data is. But applying it to massive data at high speed is very difficult. The Speaker will discuss his development concept, The concept based on non-unified filesystem and data flow management by highly distributed and parallelized cache proxy system.