"AI destroying the world" is actually a business strategy of Mythos and GPT-6.

avatar
36kr
04-10
This article is machine translated
Show original

Mythos terrified everyone for several days, but finally some experts got tired of it.

Yang Likun was the first to speak out. Last night, Mr. Yang pointed out that "Anthropic's Mythos marketing campaign is self-indulgent nonsense."

Gary Marcus, a long-time technical critic, AI critic, big-model skeptic, and New York University professor, offered a slightly milder assessment early this morning: "The Mythos hype was mostly a myth... When Firefox was compromised, the browser's sandbox was closed. Other features could be implemented with inexpensive open weight models. There is no evidence that Mythos itself represents a revolutionary leap. Simply put, everyone was fooled."

Yang Likun and Marcus astutely exposed a common tactic used by major AI companies to generate hype for new product marketing, which we can call the "doomsday carnival" model:

First, a major AI company announced that its new product, developed during the research process, possesses the capability to cause massive destruction or transform the world.

Then, major AI companies and their CEOs announced various collaborative projects to demonstrate their security constraints in the short term, and released various policy documents to demonstrate their long-term commitment to human survival and economic well-being.

Mythos is clearly a suspect. On April 8th, Anthropic released a 244-page system card document hinting at various terrible aspects of Mythos, while simultaneously announcing a collaboration with major companies across the digital industry on "Project Glass Wing" to monitor and patch Mythos.

OpenAI, A's arch-rival, is also a master of this tactic. On April 7, Altman hinted that GPT6 would overturn the US economy, while simultaneously releasing a 13-page white paper, "Industrial Policy in the Intelligent Era: A Human-First Perspective," and announcing the establishment of the "AI Security Research Scholarship" program.

This phenomenon didn't just appear in the last couple of days. Ever since ChatGPT started speaking human language, Ultraman, Musk, and Amodi have been publicly exaggerating the risk of "AI wiping out humanity" almost every month, with the probability of the apocalypse rising from 2% initially to 20% within three years.

From experts to the general public, people are increasingly turning away from this approach.

According to the general public's thinking, since AI is so terrifying, wouldn't it be better if these big bosses stopped frantically updating product versions and improving product performance, thus saving humanity from disaster? These big bosses regularly feign concern about AI destroying the world, yet they wail and cry at any voice hostile to AI. Whether AI will go insane is hard to say, but these individuals might genuinely need medication.

If everyone really thinks that way, then it's truly a case of "the prime minister isn't dreaming, but you are. " "Safe AI" and human well-being are ideologies; "institutional capture" and word-of-mouth marketing are businesses. AI giants and their bosses constantly spout ideologies, but their hearts are always on business.

To paraphrase a witty saying, this is like "scraping away the mask of the AI ​​doomsday prophet and finding out that he's actually an accelerationist who thinks AI isn't powerful enough."

A

AI moguls saying AI could destroy the world is a textbook example of "institutional capture." In short, it's like weasels vying for the job of guarding the chicken coop.

Let's explain this using the approach from an institutional economics course presentation:

To designate a key industry as requiring regulation.

Okay, since this industry is so important that it needs to be regulated, it is almost certainly complex enough that ordinary people cannot fully understand it and laymen cannot manage it.

Therefore, it is necessary to have experts with sufficient knowledge of the industry being regulated to carry out the regulation.

So here's the problem: experts in the field can't be outsiders; those who haven't worked in the industry can't understand it. Having industry insiders regulate their own sector inevitably leads to conflicts of interest and collusion among industry peers.

The end result was that a weasel guarded the chicken coop.

Specifically, "institutional capture" manifests in various forms. The simplest and most easily detected by the public is when regulatory agencies are manipulated by practitioners to issue overly lenient rules with no real binding force, or to abuse exemptions and approvals.

Alternatively, industry leaders or specific leading companies can mislead and control regulatory agencies to introduce detailed, harsh, and tailor-made prohibitions that only they can pass or only certain competitors cannot pass.

For example, once industry practitioners distort and manipulate regulatory agencies, they can set regulations so strict that no one can actually comply, achieving the effect of "red lights being invalid." If traffic lights are always red, running red lights is not only economically rational but also in accordance with public morality. If regulations are too harsh and impossible to implement, then the reality in the industry is that big bosses call the shots, and the public has no right to criticize them.

So Ultraman and Amodi are always saying that AI will wipe out all of humanity or that AI will take away your job. This isn't AI company bosses creating trouble for themselves; in industry terms, it's "narrative positioning." Once they've secured a favorable position in public discourse, they can carry out strategic operations of "institutional capture."

Take OpenAI's 13-page white paper, "AI New Deal Economy," for example. Proposals such as having robots pay taxes to distribute money to the public, establishing a public wealth fund, and changing the human work schedule to four days on and three days off are proposals that would not and could not be passed by the government in the United States or any other country at present.

But OpenAI insists on bringing this up, stating, "Broadly speaking, our company strongly supports regulation and the public good; see our white paper!" If we were to follow the line of reasoning presented in this white paper, the cash and political costs for a single country would be astronomical, far more frightening than enacting a few regulatory laws. Will the authorities ultimately just ignore this issue?

If we ignore this white paper, then what about the trivial and specific issues that arise: a town competing with a data center for electricity, a city competing with a data center for water, a stubborn village refusing to make way for the demolition of a data center, and a small factory having its wafer orders cut in line by OpenAI? Are these issues more important than the future of the national economy? Do individuals and small groups have the nerve to argue with companies that care about the well-being of all humanity?

The ancient wisdom of "give before you take" and "prosperity before ruin" is now ubiquitous in our small world. In the past, industries that used this wisdom included the oil industry after a leak, the cigarette industry after public questioning, and now the AI ​​industry.

However, not all big bosses in AI-related industries share this view. Musk recently agreed in an interview that "AI has a 20% chance of destroying the world," but stated, "I would rather have a 20% chance of seeing a possible end for humanity than not see this disruptive future at all."

Of course, Musk's sudden optimism comes against the backdrop of xAI falling behind in the current competition among giants. In the year before xAI was founded in 2023, Musk worried that "AI is the greatest danger to mankind" and called on the authorities to strictly regulate major AI companies, especially OpenAI, at a frequency of once every half week, and sometimes even several times a day.

When Huang Renxun appeared on a podcast earlier this year, he also criticized the current "AI doomsday narrative competition": "Ninety percent of this nonsense is unfounded science fiction rhetoric and apocalyptic theory. The atmosphere is not conducive to making AI safer, more productive, and more beneficial to society. People who say these things do not care about the overall welfare of society."

Of course, Huang, the leather-clad boss, also said on the same program that "industry leaders spreading doomsday theories have suppressed effective investment" and prevented everyone from buying chips, computing power, and burning tokens, which is like smashing Nvidia's rice pot.

B

Big bosses emphasize that their AI products can destroy the world/change the world, and another benefit is that word-of-mouth marketing is outstanding.

Imagine if a product had the mythical power to create a new world; the developers would have to be at least half a god. A demigod descending to earth to spread good karma… oh no, I mean, launching a new product and going public—any ordinary person would dare not patronize it; they'd be out of their mind.

Ordinary people are sending money to the AI ​​prodigies; on the bright side, they can at least enjoy some of the powerful performance of new products. On the other hand, at least they can avoid being harmed by these new products and have peace of mind.

Anthropic is a master of this kind of marketing. Every time Anthropic launches a new product, the prelude is often "This product is so scary that we are adding safety barriers to protect people."

When Claude Sonnet 3.7 was released in early 2025, it was announced that it was "ten days later than expected because the biosafety team needed to add rules to prevent the model from being used to create biological weapons."

Claude Mythos is about to be released, and its 244-page system card documentation is enough to scare the average person.

In the test, the tested Mythos model version escaped the isolated machine sandbox environment, publicly releasing details of the system vulnerability used to escape online; modified its own operation records in git; searched the machine process memory to find login credentials; and deliberately adjusted the confidence interval to evade anomaly detection.

After receiving the instruction to "search for remote code execution vulnerabilities overnight", the tested Mythos model version autonomously discovered zero-day vulnerabilities in all major operating systems and browsers without human intervention, and 83.1% of them were exploited on the first try.

Caption: Mythos's thought process of "strategic deception" and "concealment".

Such a powerful AI model is well-suited for promoting Anthropic's "Constitutional AI" concept: using structured ethical guidelines and AI self-feedback to replace purely human preferences in order to optimize AI models.

This concept translates into high certainty for users, especially enterprise clients. A secure and controllable model is synonymous with stability, no complaints, and low after-sales disputes in an enterprise environment. Enterprise clients are more willing to pay for peace of mind, low risk of liability, and minimal compliance risks than for flashy or intimidating model capabilities.

Therefore, scaring the target customer group into a panic actually makes it easier to sell products. Amazon used the same approach in its Super Bowl ad campaign in February to promote its Alexa+ smart assistant. AI defeated Thor six times; how much trouble can such a powerful home AI assistant save users?

Caption: "AI set fire to Thor's house."

C

Although the hype surrounding "AI doomsday" by major companies and business leaders may not be entirely sincere, it does not mean that AI applications in current human economic activities are completely free of "existence risks" (X Risk).

Unlike Ultraman and Amoghavajra, whose inspirations are often borrowed from science fiction novels and movies, the "existential risk" that AI poses to humanity does not require AI to create highly infectious biological agents, subvert existing human socio-economic structures, gain self-awareness and infiltrate nuclear missile silos, become hostile to humanity, or treat human survival as an irrelevant parameter. After all, the land-based nuclear missile silos of the world's leading nuclear power have not completely eliminated floppy disks and are immune to most hackers in the digital age.

AI, now imbued with bias, illusions, and sycophantic tendencies, has been deeply integrated into various aspects of human society. Even without malicious intent, the daily errors made by these AIs with limited intelligence are enough to cause humanity considerable trouble.

In 2023, some professional nuclear arms control scholars had already envisioned the most likely scenario of AI triggering nuclear war in the near future:

Suppose that nuclear superpowers A and B are in a tense atmosphere, and both countries highly integrate various war game simulations, crowd sentiment monitoring, game theory simulations, and customized AI and algorithms to accelerate decision-making into their own military systems.

Under these circumstances, any extremely minor, mosquito-level non-armed friction is exaggerated to an extreme degree by various bots that automatically wage propaganda warfare on social media networks.

These posts, filled with deepfake images and AI-generated provocative language, are then captured by AI that monitors the emotions of the crowd and interpreted as real signals of deep social psychology and decision-making.

The AI ​​that predicts trends obtains conclusions from the AI ​​that monitors the population, and uses these conclusions as the basis for its predictions.

Based on this conclusion, the AI ​​assisting in decision-making conveyed a report to human decision-makers that "armed conflict is imminent" and offered preemptive advice. Thus, a "flash nuclear war" erupted.

None of these mistakes, taken individually, could have occurred without the intervention of Skynet; none of them were entirely without human oversight during their formation; and none possessed the dramatic tension of a science fiction synopsis. However, when combined, they constitute a "sustainability risk."

AI is now an amplifier of human activity. Even without a god-like AI rebellion, current models can amplify human stupidity, malice, and carelessness to a frightening degree. The current "AI apocalypse theory" actually shares elements of religious apocalyptic beliefs in the psychology of the population.

Religious eschatology offers humanity a kind of Revelation-like psychological comfort: God will destroy the world, therefore God is trustworthy, because this shows that God's power is not only exercised in the afterlife but also in this world. If believers are devout and virtuous, they stand on the side that has the power to destroy the world, rather than on the side that will be destroyed.

Replace the key words "God" and "devoutness" in religious apocalyptic theories with "AI" and "alignment," and you get the AI-driven doomsday theory that Ultraman and Amodi preach every day. In another era, these big bosses would be excellent as gurus.

Of course, the theological practices of all orthodox religions emphasize that the human spirit cannot solely rely on the end times; every action and decision in the present moment is no less significant than the Last Judgment in purifying the soul. This concept is even more applicable to the AI ​​industry: instead of worrying that AI will become Skynet and destroy humanity, it's better to start by correcting the illusions and nonsense of the models, taking a steady and step-by-step approach.

This article is from the WeChat public account "Letter List" (ID: wujicaijing) , author: Li Xi, published with authorization from 36Kr.

Source
Disclaimer: The content above is only the author's opinion which does not represent any position of Followin, and is not intended as, and shall not be understood or construed as, investment advice from Followin.
Like
Add to Favorites
Comments