AI Giants That Once Raced Ahead Are Finally Afraid of AI

Original title: "AI Giants That Once Raced Ahead Are Finally Afraid of AI"
Original author: 動察Beating

AI could kill us all within a decade.

On Sep. 8, a 27-year-old used this sentence to announce his departure from Anthropic. Since its first day, the company has written "AI safety" into its core narrative.

His name is Jacob Coxon, and he previously worked at OpenAI and Anthropic. Had he stayed two more months, he would have received a massive equity incentive. But he left anyway.

In his resignation statement, Jacob accused frontier AI labs of engaging in an "irresponsible" race. In his view, these companies are competing to build increasingly powerful models, and the stakes are not one company's win or loss, but everyone's lives.

The post quickly went viral. As of now, it has been viewed more than 170 million times.

The day after Jacob posted his resignation, another former Anthropic employee also publicly revealed the reason for his earlier departure: concern that AI could ultimately spiral out of control.

"We may not survive this."

We may not survive this.

He warned that companies are racing to build machines "far smarter than any human," yet no one can guarantee that this race will remain under human control beyond a certain point.

Shortly after, a senior safety lead at Anthropic said in a media interview that he believes the probability of AI killing all humans within the next 10 years is over 10%.

The death probability in Russian roulette is about 16.7%.

The Need to Slow Down

Four days later, on Sep. 12, Anthropic CEO Dario Amodei published a long article whose title made his stance clear: "We Must Pace the Frontier."

He listed his three biggest concerns: humanity losing control of AI systems; AI being used for large-scale cyberattacks and bioterrorism; and technological progress moving too fast, causing severe shocks to employment and the economy.

This time, Dario did not stop at "concerns." He proposed a set of measures to slow things down.

First, frontier AI companies should grant independent third-party evaluators near "employee-level" resident access. Evaluators could continuously check whether labs comply with safety standards, whether training processes follow agreements, and independently investigate and report after incidents.

Next, several frontier labs should establish a common set of safety standards with government participation. Whoever crosses a red line must slow down or even stop advancing.

Finally, the rules must extend beyond the United States. Dario wants the U.S. and other democracies to first form a unified framework, then coordinate with authoritarian countries, ultimately subjecting all major AI powers to the same constraints.

He wrote in the article:

"The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try."

"The measures I propose are meant to advance frontier AI at a safe pace. It will be difficult. But I believe we owe humanity a try."

The reason Dario suddenly spoke so forcefully at this moment is also related to several recent events.

Previously, an OpenAI agent broke out of its test environment and hacked Hugging Face; Anthropic itself discovered that several scientists were trying to use Claude for work that could involve biological misuse. Dario even judged that at the current pace, AI agents could take over the internet within six months.

So after this article was published, responses came quickly. Sam Altman, Elon Musk, and Demis Hassabis all publicly expressed support for slowing the pace of frontier AI development.

Sam Altman then posted separately about his two biggest concerns.

First, humanity could ultimately lose control of AI. He said this is unacceptable and that he stands "unreservedly on the side of humanity."

The second risk comes from people. If extremely powerful AI ends up in the hands of one person, one company, or even one country, and is used to impose a certain worldview on everyone, the result could be "extremely dystopian."

So in Sam's view, labs can no longer decide on their own what counts as safe. At least the companies closest to the frontier need to sit down and define a common set of standards.

On Sep. 14, The Information reported that even before this round of public statements, Anthropic, OpenAI, and Google had begun private discussions as early as July this year about whether to jointly establish an AI industry standards body. After two months of talks, they still have not reached an agreement.

The disagreement is not about whether to regulate, but about who regulates and to what extent.

Dario Amodei is pushing the hardest. He wants deep government involvement, with regulators not only responsible for testing and warnings but also empowered to block a model's release if it is judged unsafe.

Demis Hassabis's proposal is more like an "AI version of FINRA." FINRA is the self-regulatory body for the U.S. financial industry, funded by the industry while authorized and overseen by the SEC.

Applied to the AI industry, frontier models would be tested by an independent body before release. Participation could be voluntary at first, and as the system matures, it would gradually become a threshold for entering the U.S. market.

Sam Altman's stance is more pragmatic. If the government cannot set up such a body quickly, the leading labs can start it themselves.

That is precisely the problem. Industry standards never just determine what is safe; they also determine who gets to stay at the table. The companies that participate in rule-making first can write their existing safety teams, testing processes, and governance frameworks into "best practices." Latecomers entering this market would have to bear compliance costs defined by the leading companies.

So safety is real, and so is the barrier to entry.

In January this year, Semafor reported that Anthropic refused to submit its latest model for testing by a top AI safety research institute in the UK. This exposed a more troublesome issue: everyone agrees models should be inspected, but labs are not ready to easily hand over the decision of who inspects them and by whose standards.

Now the three companies are discussing a new standards body, and they are fighting over exactly this power.

The Clock Running Backward

While Silicon Valley starts talking about slowing down, Washington and Wall Street both want AI to keep going.

On Sep. 13, Trump publicly responded to this round of AI safety controversy.

He said two things. One was "whoever wins AI, wins" — whoever wins AI wins. The U.S. currently leads China, and this advantage must be maintained. The other was that he called the recent warnings about AI risks "very negative forces," saying some people are constantly hyping disasters that may never happen.

Trump's reasoning has little to do with technical safety. He still views AI through the lens of national competition: as long as China keeps running, the U.S. has no room to hit the brakes on its own.

And now it is September 2026, less than two months before the U.S. midterm elections.

In March this year, an article in The Guardian argued that AI is becoming a major issue in these midterms. One of the authors is cryptographer Bruce Schneier, who has been on the front lines of technology and security debates for the past three decades.

But Trump does not have a comfortable choice either.

According to a The Times report in May this year, MAGA voters' hostility toward AI is becoming increasingly evident. They worry AI will take jobs and depress wages, and they resent data centers consuming local water and electricity. Continuing to bet on AI means Trump will eventually have to face the growing resentment within his own base.

At least for now, he has chosen the other side. Compared with potential future conflicts over jobs and resources, the risks of competition with China and a falling stock market are closer to the midterms.

Wall Street also faces significant problems.

Over the past few years, the market has been willing to assign sky-high valuations to these AI companies, essentially buying an ever-rising curve: models keep getting stronger, users and revenue keep growing, so data centers keep being built, GPUs keep being bought, and capital expenditures keep piling up.

Now, the CEOs standing at the front of this curve are themselves saying that frontier AI may need to slow down.

The market reacted faster than the debate. After leaders of Anthropic, OpenAI, SpaceX, and other companies publicly called for slowing AI development, shares of AI-related public companies briefly fell 13%.

The slowdown has not even actually happened, and Wall Street has already begun repricing.

If model capabilities can no longer surge at the previous pace, all the numbers downstream must be recalculated. How many data centers to build, how many GPUs to buy, how much capital expenditure cloud providers should spend each year, how much AI labs can still raise, and what IPOs should be worth.

So what Wall Street truly fears is not just fewer model releases. It fears that the growth rate underpinning the AI bull market of the past few years is itself becoming a problem that needs to be regulated.

More subtly, the American public stands even farther from both Washington and Wall Street.

A poll by the Annenberg Public Policy Center at the University of Pennsylvania shows that Americans are broadly pessimistic about AI's impact, and a majority want stronger government regulation. Silicon Valley fears losing control, Washington fears losing to China, and Wall Street fears valuations falling.

Amid this turmoil, the two most closely watched AI companies chose two different directions.

OpenAI decided to postpone its IPO. Sam Altman has also hinted that if safety issues require more time, the company can push back its listing.

Anthropic, however, is still charging ahead. The company has selected Nasdaq, with a target valuation of up to $2 trillion, and Nvidia is considering subscribing $10 billion as a cornerstone investor. NDTV Profit reported that Anthropic could launch its IPO roadshow marketing as early as mid-October.

Dario Amodei is calling for the entire industry to slow down, yet his own company is accelerating toward the public market.

But the capital markets may not see this as a contradiction.

If frontier AI is truly as dangerous as Dario says—dangerous enough to require U.S. government intervention, common standards among labs, and even coordination among major powers—then the companies that can participate in setting these rules will only become more important.

The stricter the regulation, the higher the cost of building frontier models; the more complex the safety testing, the harder it is for latecomers to catch up; and if a global AI governance system really emerges, Anthropic will most likely still be sitting at the rule-making table.

So "slowing down" does not necessarily mean earning less money.

It could also mean that this industry has finally become too important to let just anyone in. And for the companies already at the front, a higher barrier is itself part of the valuation.

The First Domino to Fall

Let us look back at how this round of intense discussion about AI safety was ignited, and how Jacob's post reached 170 million views.

After leaving, he was interviewed on CNN's Anderson Cooper program and described the process of publishing that post.

Jacob first wrote a draft, revised it with friends, and discussed with several people how to phrase it best. Before officially posting, he specifically asked friends to help share it. His thinking at the time was:

"Let's try to make this a bit viral."

Try to make it spread as widely as possible.

The result far exceeded his expectations. Jacob later admitted he had never seen a post about AI safety get such reach. He attributed the reaction to a previously unreleased "latent demand."

The media then pressed further on the post's spread. Jacob denied collaborating with any organization to promote it before posting, but admitted that after posting, he created a group of about ten people and asked them to help share it, including the founder of the AI safety organization Encode.

Around the time of posting, he also communicated with Daniel Kokotajlo, author of "AI 2027" and a former OpenAI employee.

Those who see Jacob as a whistleblower will say he revised repeatedly and actively sought shares only to make a warning he considered important visible to more people. His critics seize on the same details, arguing that the sensation was carefully engineered.

These two narratives can continue to be debated, but the more troubling question is: what exactly have Jacob, Dario Amodei, Sam Altman, and those working next to frontier models seen, and why are they all starting to discuss this at this moment?

You can question how Jacob delivered this message to 170 million people, but using communication tactics does not make the message false.

As early as ten years ago, Ethereum founder Vitalik Buterin wrote a long article discussing the possibility of superintelligence ultimately destroying humanity.

He used an extreme analogy: humans trying to control superintelligence is roughly equivalent to a person with an IQ of 150 trying to control an agent with an IQ of 6000. If you ask it to solve cancer, it might even conclude that the answer is to first eliminate everyone who could get cancer.

In 2016, these words existed mainly in blogs, forums, and thought experiments. Ten years later, they are appearing in resignation letters from frontier lab employees, in public statements by CEOs of the largest AI companies, and in the real agendas of government regulation, capital markets, and international competition.

And there is another layer that makes this truly difficult.

On Aug. 2, 1939, physicist Leo Szilard drafted a letter signed by Einstein. Two months later, the letter was delivered to Roosevelt. It warned that nuclear fission could be used to make a bomb of immense power, that Nazi Germany was already active in the field, and that the United States must respond quickly.

Einstein later did not join the Manhattan Project and long regretted his signature. But the question before them was brutal: even if you believe a technology is dangerous, as long as you believe your opponent is building it, stopping may be more dangerous than continuing forward.

More than eighty years later, the same dilemma has reappeared in a new form.

Dario says we must slow down; Trump says China is still catching up. Several labs want to set safety rules together, yet none wants to hand over decision-making power entirely to others. Everyone is talking about braking, but everyone is also watching whether the car next to them has let off the gas.

More critically, we still do not know how far those eyes closest to the frontier models can actually see.