Facebook disclose the carbon footprint of their new LLaMA models
Facebook used 2.6 million KWh hours of electricity and emitted 1,000 tons of CO2 when developing their new LLaMA models.
Facebook recently released four new language models called LLaMA, which outperform GPT-3 — ChatGPT’s underlying language model — on a number of tasks.
Following a recent trend on resource consumption transparency in LLM training, Facebook estimate the electricity consumption and carbon footprint of their models in the paper published on arxiv .
How to estimate and reduce the carbon footprint of machine learning models
Two ways to easily estimate the carbon footprint of machine learning models and 17 ideas for how you might reduce it
Facebook estimate that to train the 4 different sizes of LLaMA they used 2048 Nvidia A100–80GB GPUs for a period of approximately 5 months.
The energy consumption from this is estimated to be 2,638,000 KWh, roughly the same amount of electricity that 1,648 Danes use in a year on average.
Producing that amount of electricity is estimated to have led to the emission of 1,015 tCO2e – roughly the annual carbon footprint of 92 Danes.
The carbon footprint of GPT-4
Recently leaked data allows us for the first time to estimate the carbon emissions from training OpenAI’s GPT-4.
What’s really interesting about Facebook’s numbers is that they seem to account for all the GPU compute they used in the entire model development process – not just the compute from training the final version(s), which seems to otherwise be the norm.
Although the energy consumption from developing the LLaMA models cover the development of all four models, it is still surprising that the number exceeds the resource consumption from training GPT-3, which is estimated to have consumed 1,287 MWh .
The explanation for this, I venture, is probably that Facebook report the electricity consumption of the entire development process including…