The 216th R-CCS Cafe - part 2

In our work we present a theoretical framework for reasoning about deep learning performance in general, and benchmarking practices in particular. Through the lens of this framework we compare major existing approaches and outline common methodological challenges. We present our own perspective on how this challenges could be addressed, in a way complementary to the existing work. To support the proposed approach we present a software implementation of a modular and extendable benchmarking tool with rich metadata collection capabilities which is designed to collect different performance metrics from the macro level (of models) to the micro level (of kernels). Finally, we conduct several case studies to demonstrate the utility of the framework.