Unveiling the Truth: An Analysis of the Energy and Carbon Footprint of Training an OPT Model using DeepSpeed on the H100 GPU

The advent of large language models like ChatGPT and GPT-4 has undoubtedly revolutionized the AI landscape. However, training large language models is a sophisticated process, which demands substantial computational resources and leads to concerns regarding the environmental impact. This paper aims to reveal the energy usage and carbon emissions incurred in training a sizable Open Pretrained Transformers (OPT) model using the cutting-edge H100 GPU and Microsoft's DeepSpeed-Chat training framework. The experimental results demonstrate that (1) the batch size has a large impact on accuracy and carbon emission, and (2) the carbon footprint varies greatly in each training step.