R-CCS Cafe 第248回（2023年7月31日）

Title:
The Distributed Transformer for Ultra-Long Sequence Learning
Abstract:
Transformer models trained on long sequences often achieve higher accuracy than short sequences, in particular for 3D volumetric image segmentation tasks used in studying brain MRI images. Unfortunately, conventional transformers cannot process long sequences because long sequence training requires an intractable amount of computations and memory. Some exploratory methods attempt to address the computations and memory constraints, but provide minimal speedups and memory reduction, and may even lose accuracy. This presentation introduces the first end-to-end sequence-distributed transformer, the Distributed Long-Short Sequence Transformer, for efficient computations and memory storage, while minimizing communication frequency and avoiding global communications. Performance evaluation that our algorithm is 6 times faster and 10 times more memory efficient than the state-of-the-art sequence parallelism using 144 Nvidia V100 GPUs. Additionally, our algorithm scales to an extreme sequence length of 50,112 at 3,456 V100 GPUs with 178% superlinear scaling efficiency, outperforming current methods by a factor of 24 for both the number of GPUs and sequence length.