Researchers from the Institute of Computing Technology at the Chinese Academy of Sciences have developed a novel approach to improve the efficiency of large language models, which could have significant implications for the energy sector’s growing use of AI and machine learning technologies. The team, led by Huizheng Wang, has introduced BitStopper, a fine-grained algorithm-architecture co-design that aims to enhance the performance and energy efficiency of transformer-based models.
Large language models (LLMs) have revolutionized AI applications, but their widespread use comes with substantial computational and memory costs, particularly due to the quadratic cost of self-attention mechanisms. Dynamic sparsity (DS) attention has been employed to mitigate these costs, but its hardware efficiency is often limited by additional prediction stages and high memory traffic. BitStopper addresses these limitations by eliminating the need for a sparsity predictor and introducing several innovative strategies.
The researchers propose a bit-serial enable stage fusion (BESF) mechanism that minimizes memory access by progressively terminating trivial tokens and merging the prediction stage into the execution stage. This approach reduces the computational overhead and improves memory efficiency. Additionally, they developed a lightweight and adaptive token selection (LATS) strategy that works in conjunction with bit-level sparsity speculation to further enhance performance.
To optimize compute utilization, the team employed a bit-level asynchronous processing (BAP) strategy. This method improves memory fetching efficiency by processing bits on-demand, reducing idle time and enhancing overall system performance. The researchers also designed an elaborate architecture to translate the theoretical complexity reduction into practical performance improvements.
Extensive evaluations demonstrated that BitStopper achieves significant speedups and energy efficiency improvements compared to state-of-the-art Transformer accelerators. Specifically, BitStopper achieved 2.03x and 1.89x speedups over Sanger and SOFA, respectively, while delivering 2.4x and 2.1x improvements in energy efficiency. These advancements could be particularly beneficial for the energy sector, where AI and machine learning are increasingly used for predictive maintenance, energy management, and optimizing power grids.
The research was published in the Proceedings of the ACM on International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), a premier forum for research on computer system architecture, hardware/software support for programming languages, and novel hardware/software interfaces. The findings highlight the potential for significant energy savings and performance improvements in AI applications, which are increasingly integral to the energy industry’s operations and decision-making processes.
This article is based on research available at arXiv.