OpenAI, something big has happened! Recently, there have been a lot of leaks, such as OpenAI has already crossed the "recursive self-improvement" critical point, o4 and o5 can already automate AI research and development, and OpenAI has even developed GPT-5? OpenAI employees are leaking like a tide, crazily hinting that they have developed ASI internally.
All signs indicate that something big has happened at OpenAI recently.
AI researcher Gwern Branwen has published an article on OpenAI's o3, o4, and o5.
According to him, OpenAI has crossed the critical point and reached the threshold of "recursive self-improvement" - o4 or o5 can automate AI research and development and complete the remaining work!
The key points of the article are as follows:
- OpenAI may choose to keep its "o1-pro" model confidential and use its computing resources to train more advanced models like o3, similar to Anthropic's strategy
- OpenAI may believe they have made breakthroughs in AI development and are on the path to ASI
- The goal is to develop a highly efficient super-AI, similar to the goal achieved by AlphaGo/Zero
- Reasoning time search can initially improve performance, but will eventually reach its limit
There is even a rumor that OpenAI and Anthropic have already trained GPT-5 level models, but have chosen to "shelve" them.
The reason is that although the models are powerful, the operating costs are too high. It is more cost-effective to distill GPT-4o, o1, o3, and other models from GPT-5.
Even OpenAI's security researcher Stephen McAleer's recent tweets seem like a short science fiction novel:
I miss the days when we didn't know how to create superintelligence.
In the frontier labs, many researchers were very serious about the short-term impact of AI, but almost no one outside the labs fully discussed its safety implications.
Now controlling superintelligence is an imminent research issue.
How do we control the cunning superintelligence? Even with a perfect monitor, won't it convince us to release it from the sandbox?
In any case, more and more OpenAI employees are hinting that they have developed ASI internally.
Is this true? Or has the CEO Altman's "riddle man" style been learned by the employees below?
Many people think this is a common hype tactic of OpenAI.
But what's a bit scary is that some people who left a year or two ago have actually expressed concerns.
Could it be that we are really on the edge of ASI?
Has the "Pandora's box" of superintelligence really been opened?
OpenAI: "Far Ahead"
OpenAI's o1 and o3 models have opened up a new expansion paradigm: investing more computing resources in model reasoning during runtime can stably improve model performance.
As shown below, the AIME accuracy of o1 grows steadily with the logarithmic increase of test-time computing resources.
OpenAI's o3 model has continued this trend, setting record-breaking performance, with the following specific results:
- Scored 2727 on Codeforces, making it the 175th best competitive programmer in the world;
- Scored 25% on FrontierMath, where "each problem requires several hours of work by a mathematician";
- Scored 88% on GPQA, 70% of which represents doctoral-level scientific knowledge;
- Scored 88% on ARC-AGI, while the average Mechanical Turk human worker score on difficult visual reasoning tasks is 75%.
According to OpenAI, the performance improvement of the o-series models mainly comes from increasing the length of the Chain-of-Thought (CoT) (as well as other techniques, such as thought trees) and improving the CoT process through reinforcement learning.
Currently, running o3 at maximum performance is very expensive, with the cost of a single ARC-AGI task being about $300, but the inference cost is decreasing at about 10 times per year!
A recent analysis by Epoch AI indicates that the spending of frontier labs on model training and inference may be similar.
Therefore, unless they approach the hard limit of inference scaling, frontier labs will continue to invest heavily in optimizing model inference, and the cost will continue to decline.
In general, the inference expansion paradigm is expected to continue, and will be a key consideration for AGI safety.
Impact on AI Safety
So what is the impact of the inference expansion paradigm on AI safety? In short, AI safety researcher Dr. Ryan Kidd believes:
- The AGI timeline is largely unchanged, but may be advanced by about a year.
- For the deployment of frontier models, it may reduce the impact of their over-deployment, as their deployment cost will be about 1,000 times higher than expected, reducing the near-term risks from rapid or collective superintelligence.
- Supervision of the Chain-of-Thought (CoT) may be more useful, provided that non-linguistic CoT is prohibited, which is beneficial for AI safety.
- Smaller, higher-running-cost models are more susceptible to theft, but unless they are very wealthy, they are difficult to operate, reducing the risk of the curse of unilateralism.
- Increased explainability may be easier or harder; it is still uncertain.
- Models may be more accepting of reinforcement learning (RL), but this will mainly be "process-based", so it may be safer, provided that non-linguistic CoT is prohibited.
- Export controls may need to be adjusted to address specialized inference hardware.
AGI Timeline
The release of o1 and o3 does not have a major impact on the AGI timeline predictions.
Metaculus' "strong AGI" prediction seems to have been advanced by a year due to the release of o3, now expected to be achieved in mid-2031; however, this prediction has been fluctuating between 2031 and 2033 since March 2023.
Manifold Market's "When will AGI arrive?" prediction has also been advanced by a year, from 2030 to 2029, but this prediction has also been fluctuating recently.
It is likely that these prediction platforms have already factored in the impact of inference computation expansion to some extent, as Chain-of-Thought is not a new technology, even if enhanced through RL.
Overall, Ryan Kidd believes he does not have better insights than the current predictions of these prediction platforms.
Deployment Issues
In "AI Could Defeat All Of Us Combined", Holden Karnofsky describes an ambiguous risk threat model.
In this model, a group of human-level AIs, with faster cognitive speeds and better coordination capabilities, surpass humans, rather than relying on qualitatively superior intelligence capabilities.
The premise of this scenario is that "once the first human-level AI system is created, the people who created it can leverage the same computational power required to create it to run billions of copies, each running for about a year."
If the cost of running the first AGI is the same as o3-high (about $3,000 per task), the total cost would be at least $300 billion, which would make this threat model less credible.
Therefore, Dr. Ryan Kidd is less concerned about the "deployment problem", i.e., that after expensive training, short-term models can be cheaply deployed to have a huge impact.
This somewhat alleviates his concerns about "collective" or "high-speed" superintelligence, while slightly increasing his focus on "qualitative" superintelligence, at least for the first generation of AGI systems.
Supervised Thought Chains
If the model's increased cognition is embedded in the form of human-interpretable thought chains (CoT) rather than internal activation, this seems to be good news for promoting AI safety through supervision!
Although CoT does not always accurately describe the model's reasoning, this may be improved.
Ryan Kidd is also optimistic about LLM-assisted red team members, who can prevent covert conspiracies or at least limit the complexity of plans that may be secretly implemented, provided there are strong AI control measures.
From this perspective, the reasoning computation expansion paradigm seems very beneficial for AI safety, provided there is sufficient CoT supervision.
Unfortunately, technologies like Meta's Coconut ("Continuous Thought Chains") may soon be applied to frontier models, where continuous reasoning can be done without using language as an intermediary state.
While these technologies may bring performance advantages, they may also pose huge risks to AI safety.
As Marius Hobbhahn said, "If we sacrifice readable CoT for a tiny performance boost, that would be self-destructive."
However, given that users cannot see o1's CoT, it is uncertain whether the possibility of non-linguistic CoT being deployed can be known, unless revealed through adversarial attacks.
AGI is Coming
American AI writer and researcher Gwern Branwen believes that Ryan Kidd overlooked an important aspect: one of the main purposes of models like o1 is not to deploy them, but to generate training data for the next model.
Every problem solved by o1 is now a training data point for o3 (e.g., examples of any o1 conversation ultimately finding the correct answer to train more refined intuitions).
This means that the expansion paradigm here may ultimately look a lot like the current training paradigm: massive data centers striving to train an ultimate frontier model with the highest intelligence, using it in a low-search way, and then distilling it into smaller, cheaper models for those low-search or no-search use cases.
For these large data centers, the workload may be almost entirely search-related (as deploying models is cheaper and simpler than actual fine-tuning), but this may not matter to others; as before, what is seen is essentially waiting 3-6 months for a more intelligent AI to emerge, using high-end GPUs and massive amounts of power.
OpenAI deployed o1-pro instead of keeping it private and investing computing resources in more o3 training and self-bootstrapping processes.
Gwern Branwen is a bit surprised by this.
Similar things have also happened with Anthropic and Claude-3.6-opus - it hasn't "failed", they just chose to keep it private and distill it into a small, cheap, but oddly intelligent Claude-3.6-sonnet.)
OpenAI Breaks the "Critical Point"
Members of OpenAI have suddenly become a bit strange, even euphoric, on Twitter, likely because they have seen the improvements from the original 4o model to o3 (and now the current state).
It's like watching AlphaGo rise in the international Go rankings: it keeps going up... up... and up...
They may feel they have "broken through", finally crossed the critical point: from pure frontier AI work, which almost everyone will replicate in a few years, to the takeoff stage - cracking the key to intelligence, so that o4 or o5 will be able to automate AI research and complete the rest.
In November 2024, Altman stated:
I can see a path where the work we're doing continues to accelerate growth, and the progress we've made in the last three years will continue for the next three years, six years, nine years or longer.
But he soon backtracked:
We now know with great confidence how to build traditional AGI... We're starting to push beyond that, towards true superintelligence. We love our current products, but we're in it for the long haul. With superintelligence, we can do anything.
While other AI labs can only look on in envy: when superintelligence research becomes self-sustaining, they will be unable to obtain the large computing equipment needed to compete.
Ultimately, OpenAI may dominate the entire AI market.
After all, AlphaGo/Zero models not only far surpass humans, but also have very low running costs. Just a few steps of search can reach superhuman strength; even just forward propagation is close to professional human level!
If you look at the relevant expansion curves in the text, the reason is actually quite obvious.
Paper link: https://arxiv.org/pdf/2104.03113
Continuing Distillation
Search during inference is like a stimulant, able to immediately boost scores, but will soon reach its limits.
Soon, you have to use smarter models to improve the search itself, rather than doing more searches.
If pure search could be so effective, chess would have been solved in the 1960s.
In reality, it wasn't until May 1997 that a computer defeated the world chess champion, but surpassing chess grandmasters in search speed was not difficult.
If you want text that says "Hello World", a group of monkeys on typewriters might be enough; but if you want the full text of "Hamlet" before the universe ends, you'd better start cloning Shakespeare now.
Fortunately, if you have the necessary training data and models, you can use them to create a smarter model: smart enough to produce works on par with or even surpassing Shakespeare.
On December 20, 2024, Otman emphasized:
Amidst the noise, some news seems to have been overlooked:
On programming tasks, o3-mini will outperform o1, and at a much lower cost!
I expect this trend to continue, but I also foresee the money required to gain marginal additional performance becoming exponentially strange.
So you can spend money to improve the model's performance on certain outputs... but "you" may be an "AI lab", and you're just spending money to improve the model itself, not just for a temporary output on a specific problem.
This means that outsiders may never see the intermediate models (like how Go players couldn't see AlphaZero's random checkpoints during training).
And if "deployment cost is 1000x current" holds true, this is also a reason not to deploy.
Why waste these computing resources serving external customers, instead of continuing to train, distilling it back, and eventually deploying an optimized model that costs 100x, then 10x, 1x, or even less?
So once you consider all the second-order effects and new workflows, the search/testing time paradigm may end up looking surprisingly familiar.
References:
https://x.com/emollick/status/1879574043340460256
https://x.com/slow_developer/status/1879952568614547901
https://x.com/kimmonismus/status/1879961110507581839
https://www.lesswrong.com/posts/HiTjDZyWdLEGCDzqu/implications-of-the-inference-scaling-paradigm-for-ai-safety
https://x.com/jeremyphoward/status/1879691404232015942
This article is from the WeChat public account "New Intelligence", written by Aeneas KingHZ, and published with authorization from 36Kr.
