Green coding - Kainos: Three steps to greener AI

AI: the new mainframes

The financial cost of AI is already billions of dollars. We read that Microsoft has invested $10 billion in Open AI, AWS $4 billion in Anthropic and Meta $33 billion in its open-source AI, while Google will spend $100 billion. This investment is driven by the need to build AI-optimised datacentres, increased compute costs to serve billions of people and substantial training costs.

Large AI models are analogous to the mainframes of the past: large, expensive machines that few companies could afford. AI needs to follow a similar path to mainframes to reduce its financial cost and be more transparent about its environmental cost.

The huge environmental cost of AI at scale are largely hidden – but what we do know is concerning. While researching for the book – Digital Sustainability: The Need for Greener Software – I was shocked at some of the data that’s out there:

• Using huge amounts of energy. One study estimates that at the current growth rate, AI could consume 85-134 TWh of electricity by 2027, the same as a small country.
  
• Requiring billions of litres of water daily to cool data centres that run AI models for training and inference.
  
• Manufacturing millions of GPUs. Nvidia sold 500,000 of its H100 AI GPUs in Q3 2023. This means significant carbon emissions and water use from manufacturing and mining for metals and materials.

When building software products that use AI, we can choose to reduce their environmental cost in the following ways:

Use a pre-trained AI model

Reuse is greener than building new. This is true for devices and for AI models. Many AI models are provided as pre-trained models that you can use with your data for inference. This is a well-established pattern for developers using traditional SaaS NLP services such as Google Translate or AWS Textract.

This is similar for pre-trained foundation models. You can use these as-is interactively or by using an application programming interface (API). You can also fine-tune these models with selective training on a smaller dataset or use retrieval augmentation. This is a much smaller cost than building a LLM. If in doubt don’t try to build your own language model, instead reuse a foundation model.

Use a small language model

We are starting to see a focus on small language models (SLM) which have fewer hyperparameters, faster inference and require less compute. They are a reaction to the cost and inefficiency of large language models and aim to deliver accuracy with a smaller footprint. Microsoft has released Phi-3, for example, that they evaluate as comparable in accuracy to the much larger Mistral and Llama-2 models.

Use the smallest model that works for your use case. 

Models are now being made available in different sizes so that you can choose the one that fits best, such as five sizes of Google’s BERT. Look out for existing models that have been tuned down with approaches like parameter pruning and quantization. A recent study showed that the Llama 2 open-source model only started to reduce the accuracy of its results after 40-50% of its layers were pruned.

Train in a sustainable datacentre

If you need to train or fine-tune a model, it should be trained in a more sustainable data centre. Most will choose to rent compute in the cloud to train their model because they don’t need dedicated hardware. But data centres are not identical. Costs and environmental impact will differ across data centres even from the same cloud provider. But it’s not always simple to work out which is better.

Google is one of the only hyperscalers to release detailed datacentre information on carbon-free energy use and water use. This shows that in 2022, deploying to its Canada region would use 100% carbon-free energy while deploying to Singapore would use only 4%. Similarly with water use, deploying to Dublin would consume almost no drinking water while deploying to Iowa consumes billions of litres. Selecting a more sustainable data centre means choosing a data centre that is in a colder climate with direct access to a high level of renewable energy, for example in the Nordics.

Don’t use AI if you don’t need it!

A bigger question to ask yourself is – do you need AI at all? As a software builder, the first step to achieving greener AI should be understanding users and their needs before building new functionality. Do user research to understand if they’ll need or use AI features. If your users will need these features, can it be achieved using traditional CPU-bound algorithms? These use significantly less energy than AI because they are much less complex than large neural networks. 

One of the best ways we can limit the damage of AI will be to only use it when it is really needed. And where AI is required, make sure you are taking steps to minimise its environmental cost. 

This is going to be a key battleground in the shift to digital sustainability.