R-CCS Cafe - Special Edition (May 9, 2025)

Title:
Data Management and Machine-Actionable Reproducibility for HPC with git and Datalad
Abstract:
The talk will present the two connected topics of Research Data Management (RDM) in HPC environments and of reproducibility of computational results and how it can be adopted for HPC. Research Data Management according to the F.A.I.R. principles is an established concept and required for most/all Computational Science projects out of formal reasons (because institutions and funding agencies demand it) and out of practical reasons (because it benefits researchers). Versioning of data collections is for some reasons not part of the F.A.I.R. principles even though it is most useful for software development and many types of text-based data collections. Git is the de-facto standard for distributed version control there. The talk present how git can be applied for general data including large, binary files. Furthermore, it presents extra benefits of such research data repositories for typical HPC projects. The second part of the talk introduces Datalad, a tool on top of git repositories, which offers machine-actionable reproducibility for computational results integrated in the git log of a data repository. This is incompatible with HPC processing unfortunately, because of batch processing and because git repositories don’t allow concurrent accesses. We developed a solution for this conflict and can now offer machine-actionable reproducibility for HPC where many Slurm jobs contribute to large data repositories.
Speaker Info:
Andreas Knüpfer is the head for the Scientific Computing Core (SCC) department at the Center for Advanced Systems Understanding (CASUS) in Görlitz/Saxony/Germany which is part of the Helmholtz-Zentrum Dresden-Rossendorf (HZDR). His background is Applied Mathematics from his first degree and High Performance Computing (HPC) / Computer Science from his PhD. The mission of the SCC department is research on Computational Science methods from classical HPC simulations and parallel programming to ML/DNN models, surrogate models as well as Research Data Management (RDM), collaborative software development methodologies and tools. Furthermore, it collaborates with and supports all other groups at CASUS with their Computational Science challenges to allow better science from the combination of application-area expertise with the best computational approaches.