Today, I would like to discuss a question that’s often at the forefront of discussions in the AI community: why do more powerful models have a larger number of parameters, and why do we need so many parameters which number significantly exceeds the training data size? The answer is not as straightforward as you might think, and it’s deeply rooted in the way machine learning models learn and generalize.
To begin with, it’s important to understand that parameters in a machine learning model are like the knobs and levers that the model adjusts to learning patterns from the training data. More parameters mean more complexity and more capacity for the model to learn intricate patterns. For example, the original GPT model had 117 million parameters, GPT-2 had 1.5 billion, GPT-3 boasted 175 billion, and the latest version of ChatGPT-4, is estimated to consist of a staggering 100 trillion parameters. Similarly, Google’s Pathways Language Model (PaLM) is a 540-billion parameter model, and Facebook’s LLaMA comes in multiple sizes with up to 65 billion parameters.
However, we are confronted with a paradox when we consider the traditional theory of model complexity and generalization error. The generalization error is the discrepancy between a model’s performance on the training data and its performance on new, unseen data. According to the classical understanding, the generalization error should increase as the number of parameters exceeds the size of the training dataset. This is because, when the model becomes too complex, it tends to overfit the training data, learning noise, and random fluctuations rather than the underlying patterns. This overfit model then performs poorly when presented with new data.
This brings us to a fascinating phenomenon that has been observed in modern deep learning models called “double descent”. Double descent is a phenomenon where the test error of a model first improves, then gets worse (which aligns with our classical understanding), and then, surprisingly, improves again with increasing model size, data size, or training time. This is counter-intuitive to both the classical statistic
al wisdom that “too large models are worse” and the modern machine learning paradigm that “bigger models are better”.
Double descent suggests that there’s a regime where training longer can reverse overfitting, and a point where more data can actually hurt the model’s performance. At the “interpolation threshold”, where the models are just barely able to fit the training set, the test error peaks. Beyond this point, in the over-parameterized regime, the error decreases again, suggesting that there are “good models” that both interpolate the training set and perform well on the test set.
This phenomenon seems to be almost universal among different types of models and tasks, but it’s not yet fully understood why it happens. Further investigation into this phenomenon is a crucial research direction to help us better understand the interplay between model complexity, data size, and training time, and to build more efficient and effective models in the future.
In summary, while the growth in the number of parameters in machine learning models might seem perplexing or even counterproductive at first, it is an essential aspect of the ongoing quest to create more powerful and capable AI systems. The double descent phenomenon and the over-parameterization regime offer fascinating insights into the complex dynamics of model training and generalization, serving as a testament to the evolving and often surprising nature of machine learning research.
Thank you for your continued support and interest in Gen AI Services. Let’s continue to explore.