Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights

Machine learning (ML) models are widely used in many important domains. For\nefficiently processing these computational- and memory-intensive applications,\ntensors of these over-parameterized models are compressed by leveraging\nsparsity, size reduction, and quantization of tensors. Unstructured sparsity\nand tensors with varying dimensions yield irregular computation, communication,\nand memory access patterns; processing them on hardware accelerators in a\nconventional manner does not inherently leverage acceleration opportunities.\nThis paper provides a comprehensive survey on the efficient execution of sparse\nand irregular tensor computations of ML models on hardware accelerators. In\nparticular, it discusses enhancement modules in the architecture design and the\nsoftware support; categorizes different hardware designs and acceleration\ntechniques and analyzes them in terms of hardware and execution costs; analyzes\nachievable accelerations for recent DNNs; highlights further opportunities in\nterms of hardware/software/model co-design optimizations (inter/intra-module).\nThe takeaways from this paper include: understanding the key challenges in\naccelerating sparse, irregular-shaped, and quantized tensors; understanding\nenhancements in accelerator systems for supporting their efficient\ncomputations; analyzing trade-offs in opting for a specific design choice for\nencoding, storing, extracting, communicating, computing, and load-balancing the\nnon-zeros; understanding how structured sparsity can improve storage efficiency\nand balance computations; understanding how to compile and map models with\nsparse tensors on the accelerators; understanding recent design trends for\nefficient accelerations and further opportunities.\n