The development of the GPT-5 (codenamed Orion) project, which has been ongoing for over 18 months but has yet to see the light of day, has new revelations.
The latest report from The Wall Street Journal states that people close to the project say that although Orion's performance currently exceeds OpenAI's current models, the current progress is not enough to justify the huge cost required to continue operating the new model.
According to insiders, GPT-5 has undergone at least two rounds of training, each of which has exposed new problems and failed to meet researchers' expectations. Moreover, each round of training takes months, and the computational cost of a single round is close to $500 million.
In short, whether this project will succeed and when it will succeed is still unclear. It also faces a more severe problem: the global data volume is insufficient for it to reach the desired level of intelligence.
Difficulties in the Training Process
After the release of GPT-4 in March 2023, OpenAI immediately began the development of GPT-5. In November of the same year, Altman stated that nothing called GPT-5 would be released in 2024.
Generally speaking, the capability of AI models increases as they absorb more data. During the training process, the models are fed trillions of tokens, and the training may last for several months, relying on thousands of expensive and scarce computing chips. Altman has revealed that the cost of training GPT-4 alone exceeded $100 million, and the training cost of future AI models is expected to exceed $1 billion. If the training fails, the consequences will be as severe as a rocket launch failure, resulting in massive losses.
To reduce the risk of failure, OpenAI usually conducts small-scale trials to verify the feasibility of the model design and training. In this way, researchers can identify and fix potential problems before the formal large-scale training.
However, the development of GPT-5 has faced challenges from the very beginning. In the middle of 2023, OpenAI launched an experimental training called "Arrakis" to test the new design of GPT-5. Unfortunately, the training progress was slow, indicating that if a larger-scale training were to be conducted, it would take an extremely long time and be extremely costly. The experimental results also showed that the development of GPT-5 is more complex and difficult than originally expected.
Therefore, OpenAI's research team decided to make a series of technical adjustments to Orion and further realized that the existing public Internet data can no longer meet the model's needs. To improve the performance of GPT-5, they urgently need more diverse and higher-quality data.
"Creating Data from Scratch"
According to reports, to address the data shortage problem, OpenAI has decided to "create data from scratch". Specifically, OpenAI is hiring people to write new software code or solve mathematical problems, allowing Orion to learn from these tasks. These people include software engineers and mathematicians, and they will also explain their work processes to Orion.
Many researchers believe that code, as the language of software, can help large models solve problems they have never encountered, thereby enhancing their ability to solve complex problems.
Jonathan Siddharth, CEO and co-founder of Turing, said, "We are transferring human intelligence from the human brain to the machine brain."
In the AI training process, a Turing executive explained that software engineers may be asked to write a program to efficiently solve complex logical problems, while mathematicians may need to calculate the maximum height of a pyramid made of one million basketballs. The key to these tasks is not just the final answer, but the thought process of arriving at the answer, which will be included in the AI training materials.
In addition, OpenAI is also collaborating with experts in fields such as theoretical physics, asking them to explain how to solve thorny problems in their respective domains. This content also helps to improve Orion's intelligence level.
However, hiring people to build data from scratch is not likely to be an efficient process. The training data for GPT-4 is about 13 trillion tokens. If 1,000 people write 5,000 words per day, it would still take several months to produce 1 billion tokens.
To accelerate the training, OpenAI also uses so-called "synthetic data", which are data generated by AI, to help train Orion. However, research has shown that the feedback loop of using AI-generated data for AI training can sometimes lead to model errors or meaningless answers.
In this regard, insiders said that OpenAI's scientists believe that using data generated by Orion can avoid these problems.
Advancing Amid Internal and External Pressures
The challenges facing OpenAI are not only on the technical level, but also include internal turmoil and the almost constant poaching of talent by competitors. Additionally, the dual pressures of technology and funding are clearly increasing. Each training session costs up to $500 million, so the final training cost is likely to exceed $1 billion. At the same time, the rise of competitors is putting greater pressure on OpenAI. Companies like Anthropic and Google are launching new-generation models in an attempt to surpass OpenAI.
The loss of talent and internal divisions have further slowed the development progress. Last year, the OpenAI board suddenly dismissed Altman, causing some researchers to question whether the company could continue to operate. However, Altman was soon reappointed as CEO and began to reform the company's governance structure.
Since the beginning of this year, more than 20 key executives, researchers, and long-term employees have left OpenAI, including co-founder and chief scientist Ilya Sutskever and technology director Mira Murati. Just recently, the highly respected researcher Alec Radford also announced his departure, having worked at OpenAI for about eight years and authoring several important papers.
As Orion's progress has stalled, OpenAI has begun to develop other projects and applications, including a simplified version of GPT-4 and the Sora product that can generate AI videos. However, the report also mentions that this has led to a situation where different teams are competing for limited computing resources, particularly between the new product development team and the Orion research team.
The predicament of GPT-5 may reveal a larger industry question: Is AI approaching a "bottleneck" in its development? Industry insiders point out that the strategy of relying on massive data and larger models is gradually becoming ineffective. As former OpenAI scientist Sutskever recently said in public, "We only have one internet," and the growth of data is slowing down, depleting the "fossil fuel" that has driven the leap of AI.
Regarding the future of GPT-5, Altman has not provided a clear timeline, and we still cannot determine whether or when OpenAI will release a model worthy of being called GPT-5.
Reference link:
https://www.wsj.com/tech/ai/openai-gpt5-orion-delays-639e7693
This article is from the WeChat public account "AI Frontline", compiled by Yan Shan, and published with authorization by 36Kr.
