R-CCS Cafe - Special Edition(1) (Sep 18, 2024)

Title:
Scalable Lineage-Driven Data Management In The Age of AI
Abstract:
A lineage that records the evolution of intermediate data and metadata during runtime is a powerful technique in a wide range of scenarios at scale: verify and understand the results more thoroughly by sharing and analyzing intermediate results (which facilitates provenance, reproducibility, and explainability), new algorithms and ideas that reuse and revisit intermediate and historical data frequently (either fully or partially), manipulation of the application states (job pre-emption using suspend-resume, debugging), etc. This talk advocates a new data model and associated tools (DataStates, VELOC) that facilitate such scenarios. In the age of AI, this approach has particularly interesting applications: repositories for derived models that keep provenance and enable incremental storage (e.g. in the context of NAS), data pipelines with historic access (e.g. for continual learning based on rehearsal), evaluation of intermediate training stages and surviving model spikes (especially for LLMs), distributed caching in support of accelerating inferences, etc.