Why GPT-4 Is 100x Smaller Than People Think

The Chronicle Of A Rumor

Since before the release of GPT-4, the rumor mill has been buzzing.

People predicted and are still claiming the model has 100 trillion parameters. That's a trillion with a "t".

The often-used graphic above makes GPT-3 look like a cute little breadcrumb, which is about to have a live-ending encounter with a bowling ball

Sure, OpenAI's new brainchild certainly is mind-bending. And language models have been getting bigger - fast!

But this time is different and it provides a good opportunity to look at the research on scaling large language models (LLMs).

Let's go!

Training 100 Trillion Parameters

The creation of GPT-3 was a marvelous feat of engineering. The training was done on 1024 GPUs, took 34 days, and cost $4.6M in compute alone [1].

Training a 100T parameter model on the same data, using 10000 GPUs, would take 53 Years. However, to avoid overfitting such a huge model requires a much(!) larger dataset. This is of course napkin math but it is directionally correct.

So, where did this rumor come from?

The Source Of The Rumor:

It turns out OpenAI itself might be the source.

In August 2021 the CEO of Cerebras told wired: "From talking to OpenAI, GPT-4 will be about 100 trillion parameters".

At the time, this was most likely what they believed. But that was back in 2021. So, basically forever ago when machine learning research is concerned.

Things have changed a lot since then!

To what has happened we first need to look at how people actually decide the number of parameters in a model.

Deciding The Number Of Parameters:

The enormous hunger for resources typically makes it feasible to train an LLM only once.

In practice, the available compute budget is known in advance. The engineers know that e.g. their budget is $5M. This will buy them 1000 GPUs for six weeks on the compute cluster. So, before the training is started the engineers need to accurately predict which hyperparameters will result in the best model.

But there's a catch!

Most research on neural networks is empirical. People typically run hundreds or even thousands of training experiments until they find a good model with the right hyperparameters.

With LLMs we cannot do that. Training 200 GPT-3 models would set you back roughly a billion dollars. Not even the deep-pocketed tech giants can spend this sort of money.

Therefore, researchers need to work with what they have. They can investigate the few big models that have been trained. Or, they can train smaller models of varying sizes hoping to learn something about how big models will behave during training.

This process can be very noisy and the community's understanding has evolved a lot over the last few years.

What People Used To Think About Scaling LLMs

In 2020, a team of researchers from OpenAI released a paper called: "Scaling Laws For Neural Language Models".

They observed a predictable decrease in training loss when increasing the model size over multiple orders of magnitude.

So far so good. However, they made two other observations, which resulted in the model size ballooning rapidly.

1. To scale models optimally the parameters should scale quicker than the dataset size. To be exact, their analysis showed when increasing the model size 8x the dataset only needs to be increased 5x.

2. Full model convergence is not compute-efficient. Given a fixed compute budget it is better to train large models shorter than to use a smaller model and train it longer.

Hence, it seemed as if the way to improve performance was to scale models faster than the dataset size [2].

Interesting so far? If you like this content, you will also enjoy The AI Break with their weekly updates on news and trends! Go ahead and check them out. It helps me out a lot.

And that is what people did. The models got larger and larger with GPT-3 (175B), Gopher (280B), Megatron-Turing NLG (530B) just to name a few.

But the bigger models failed to deliver on the promise.

Read on to learn why!

What We Know About Scaling Models Today

Turns out, you need to scale training sets and models in equal proportions. So, every time the model size doubles, the number of training tokens should double as well.

This was published in DeepMind's 2022 paper: "Training Compute-Optimal Large Language Models"

The researchers fitted over 400 language models ranging from 70M to over 16B parameters. To assess the impact of dataset size they also varied the number of training tokens from 5B-500B tokens.

The findings allowed them to estimate that a compute-optimal version of GPT-3 (175B) should be trained on roughly 3.7T tokens. That is more than 10x the data that the original model was trained on.

To verify their results they trained a fairly small model on lots of data. Their model, called Chinchilla, has 70B parameters and is trained on 1.4T tokens. Hence it is 2.5x smaller than GPT-3 but trained on almost 5x the data.

Chinchilla outperforms GPT-3 and other much larger models by a fair margin [3].

This was a great breakthrough!
The model is not just better, but its smaller size makes inference cheaper and finetuning easier.

So, we are starting to see that it would not make sense for OpenAI to build a model as huge as people predict.

Let’s put a nail in the coffin of that rumor once and for all.

To fit a 100T parameter model properly, open OpenAI would need a dataset of roughly 700T tokens. Given 1M GPUs and using the calculus from above, it would still take roughly 2650 years to train the model [1].

You might be thinking: Great, I get it. The model is not that large. But tell me already! How big is GPT-4?

The Size Of GPT-4:

We are lucky.

Details about the GPT-4 architecture recently leaked on Twitter and Pastebin.

So, here is what GPT-4 looks like:

• GPT-4 has ~1.8 trillion parameters. That makes it 10 times larger than GPT-3.

• It was trained on ~13T tokens and some fine-tuning data from ScaleAI and produced internally.

• The training costs for GPT-4 were around $63 million for the compute alone.

• The model trained for three months using 25.000 Nvidia A100s. That’s quite a considerable speedup compared to the GPT-3 training.

Regardless of the exact design, the model was a solid step forward. However, it will be a long time before we see a 100T-parameter model. It is not clear how such a model could be trained.

There are not enough tokens in our part of the Milky Way to build a dataset large enough for such a model.

There are probably not enough tokens in the

Whatever the model looks like in detail, it is amazing nonetheless.

These are such exciting times to be alive!

As always, I really enjoyed making this for you and I sincerely hope you found it useful!

P.S. If you know someone who could find this useful, please, share it with them. It means a lot to me!

References:

[1] D. Narayanan, M. Shoeybi, J. Casper , P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee , M. Zaharia, Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021), SC21

[2] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child,... & D. Amodei, Scaling laws for neural language models (2020), arxiv preprint

[3] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. Casas, L. Hendricks, J. Welbl, A. Clark, T. Hennigan, Training Compute-Optimal Large Language Models (2022). arXiv preprint arXiv:2203.15556.

[4] S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. Driessche, J. Lespiau, B. Damoc, A. Clark, D. Casas, Improving language models by retrieving from trillions of tokens (2021). arXiv preprint arXiv:2112.04426.Vancouver