The 168th R-CCS Cafe – part 1

Communication is threatening to become an increasing bottleneck towards performance scaling in the post exascale era as bytes-per-FLOP ratios continue to decline. We describe bandwidth steering in HPC to take advantage of emerging photonic switches for efficiently changing the connectivity of the lower layers in a hierarchical topology to reconstruct locality that was lost from system fragmentation and was impossible to recover with task placement. This allows for more aggressive oversubscription of the higher layers to reduce cost with no performance penalty. We demonstrate bandwidth steering with a scalable algorithm in an experimental testbed and at system scale using simulations. At the system scale, bandwidth steering reduces static power consumption per unit throughput by 51% and dynamic power consumption by 10% compared to a reference topology. In addition, bandwidth steering reduces average network latency by up to 87% and improves the average throughput by an average of 4.3x.