AI pre-training will come to an end! OpenAI co-founder: Autonomous AI and synthetic data accelerate the arrival of the era of artificial super intelligence

OpenAI co-founder Ilya Sutskever pointed out at the Neural Information Processing Systems (NeurIPS) conference in Vancouver, Canada on 12/15, that the development of AI has reached a critical turning point, with pre-training technology gradually facing bottlenecks, and the future will move towards Artificial Super Intelligence (ASI).

AI pre-training data encounters a 'ceiling', transformation is imperative

Sutskever stated at the conference that the era of AI pre-training is about to end. He believes that the current amount of internet data is approaching the limit, and in the future, new technologies must be developed to continue to push AI towards the next stage, ultimately developing Artificial Super Intelligence (ASI).

Sutskever pointed out that as software, hardware, and algorithms continue to improve, the computing power of AI has increased significantly, but the data used to train AI cannot be expanded indefinitely. Sutskever compared data to the 'fossil fuel' of AI, stating: "Data will not grow endlessly, because the internet is only one. Data is like the fossil fuel of AI, and it is currently almost exhausted, so in the future, we must find a way to make full use of the existing data."

(Note: Pre-trained models refer to models that have already learned basic knowledge and do not need to be trained from scratch.)

Three key technologies to advance AI development

Although Sutskever pointed out the problems currently faced by AI at the conference, he also proposed three key technologies that can influence the evolution of AI into Artificial Super Intelligence (ASI):

1. Agentic AI: Can make decisions and execute tasks without human operation, and can dynamically adjust behavior based on goals and the environment. Unlike AI Agents, which are mainly passive or act based on fixed logic and require more human intervention.
2. Synthetic Data: Use AI to generate high-quality simulated data to solve the problem of insufficient data. For example, to train an AI model to recognize vehicles driving on the road, but real-world traffic data is insufficient, we can use synthetic techniques to 'generate' many simulated vehicles and scenarios to replace them.
3. Inference Time Computing: Improve the computing power of AI models to enable AI to solve complex problems more quickly.

Sutskever believes that these three technologies can propel the current AI technology towards 'Artificial Super Intelligence' (ASI).

AI fever sweeps the blockchain and LLM market

The concept of AI agents is not only attracting attention in the technology field, but many MEME coins and large language models (LLMs) have also begun to integrate AI technology, such as the AI agent Truth Terminal promoting the MEME coin GOAT on social media, whose market value has soared to $600 million, even impressing the well-known venture capitalist a16z founder Marc Andreessen.

The most recent and well-known case of AI agents combined with large language models is the Gemini 2.0 model launched by Google DeepMind. According to Google's official statement, Gemini 2.0 can directly generate images, text, and even convert text to speech, and can adjust the sound effects of different languages, as well as directly use Google search, execute code, and use customized third-party tools.

The advantages of autonomous AI, solving the 'AI hallucination' problem

Sutskever pointed out that autonomous AI and inference time computing can help solve the 'hallucination' (AI Hallucinations) problem in AI training. The so-called AI hallucination refers to the fact that due to insufficient training data, AI models may produce erroneous or false information. As new-generation AI models still rely on the data output by old models, this problem will only get worse.

Sutskever stated that to solve the 'hallucination' problem, autonomous AI can strengthen reasoning and real-time computing capabilities to effectively judge the authenticity of data, improving the reliability and efficiency of AI.

Facing the major problem of 'hallucination' caused by the AI training data reaching its limit, it is actually not entirely consistent with the idea of Nvidia (Nvidia) CEO Jensen Huang. Previously, Huang also pointed out this problem in an interview, and proposed three important stages for improving 'hallucination' in the future:

• Early Training:

  • This is the basic stage for AI, where it absorbs a large amount of real-world data to 'learn' and 'discover' various knowledge, but this is just an introduction and is not deep enough.

• Later Training:

  • This is the stage of strengthening AI, through human feedback, such as humans helping to score, as well as AI's own feedback and the use of synthetic data to simulate more scenarios. At this stage, techniques like reinforcement learning and multi-path learning will be added to help AI focus on improving specific skills and better understand how to solve problems.

• Test Time Scaling:

  • This stage can be understood as the beginning of AI "thinking". When faced with complex problems, AI will break down the problem step by step, repeatedly simulate different solutions, and then continuously adjust to find the best answer. Huang Renxun believes that if you give AI more "thinking time", the answers it comes up with may be more accurate or of higher quality.