Algorithmic Efficiency and Energy Consumption Trends in Large Language Model Pre-Training

Abstract

As foundation models scale past hundreds of billions of parameters, their electricity carbon footprint becomes an urgent concern. We aggregate data from 45 recent LLM training runs to highlight compute efficiency bottlenecks.