The Google Secret That Built ChatGPT (And Modern AI)

In this image, you can see several of the researchers who were part of the Google team behind the historic paper Attention Is All You Need, during an interview with NVIDIA CEO Jensen Huang.

There’s a good chance you’ve used a popular AI model like ChatGPT or Gemini at least once. You may have also wondered how these models were created and how they’ve improved so dramatically in just a few years. A large part of the answer can be found in a research paper published by Google researchers in 2017 titled Attention Is All You Need.

What Makes Attention Is All You Need So Important?

To understand why this 11-page paper is so influential, you first need to understand how AI models worked before it was published.

At that time, most AI models processed the words in a sentence one at a time, following the exact order in which they appeared. This made training much slower and made it difficult for the models to connect words that were far apart within the same piece of text.

Attention Is All You Need introduced a completely different architecture called the Transformer. Instead of reading words one by one, the model can analyze an entire sentence at once and decide which words deserve the most attention in order to better understand the context.

This made training much faster, more scalable, and more accurate. It also laid the foundation for models such as BERT, GPT, Gemini, Claude, Llama, and virtually every major large language model in use today.

Why Didn’t Google Take Full Advantage of It?

It’s a bit ironic. Even though Google researchers developed the paper and created the Transformer architecture, their original goal wasn’t to build a chatbot like ChatGPT.

Instead, they wanted to improve Google Translate by creating a model capable of translating text more quickly and accurately. To solve that problem, they designed an entirely new architecture.

After realizing how successful ChatGPT had become, Google quickly shifted its focus to developing its own chatbot, which eventually led to the launch of Gemini. Of course, both ChatGPT and Gemini have improved significantly since their first releases, especially in handling more complex tasks, increasing efficiency, and reducing the cost of generating tokens.

AI Training Is Still Expensive

Training modern AI models remains extremely expensive, and so far most companies have struggled to make them consistently profitable. Even OpenAI, the company behind ChatGPT, is not yet profitable, although several industry projections suggest it could reach profitability sometime between 2029 and 2030.

AI Has Moved Far Beyond Text

If you’re an average user, you’ve probably noticed just how quickly AI models have evolved. They no longer focus only on generating text. Today they can also create images, videos, music, and code, and they’re even contributing to scientific research, particularly in biology, where they’re helping researchers develop new treatments and improve drug discovery.

Because of that, we’re still only at the beginning of the AI era. In another 20 years, it will be fascinating to look back and see how much artificial intelligence has evolved and whether it has benefited or harmed humanity overall. AI also raises important ethical and educational questions that deserve their own discussion, but that’s a topic for another Dragon Blogger article.

Dive into more history and related items of the Wiki

That’s all for today!