Anthropic and OpenAI's Anti-Distillation: Bans, Hidden Reasoning, but Can't Stop the Chase

Author: Wang Ziyi, LatePost

Editor: Cheng Manqi

Business logic matters more than technical logic.

On September 10, 2026, Anthropic released a 154-page report claiming it had identified and blocked large-scale distillation of Claude by seven Chinese AI labs.

Distillation is a common model training method: developers use a stronger model to generate data, then use that data to train their own models to imitate the leading model. Anthropic refers to unauthorized, large-scale, covert extraction of model capabilities as "illicit distillation."

Anthropic stated that the relevant organizations used clearly violating methods when extracting outputs from frontier models: such as using stolen credit cards, login credentials, and API keys to create fake accounts in bulk.

The report also claimed that some Chinese model companies forwarded user requests to Claude, or purchased user conversations from third-party routing services, using this data to train models. Some conversations contained names, corporate data, and valid access credentials.

This is not the first time Anthropic has accused "distillation attacks." In February 2026, it claimed that DeepSeek, Moonshot AI, and MiniMax conducted over 16 million interactions with Claude through approximately 24,000 fake accounts. Since then, Anthropic has raised its defenses. OpenAI and Google are also continuously identifying and responding to distillation attacks.

Can these measures prevent distillation?

Liu Yi, an assistant professor at Griffith University in Australia, is skeptical. He has long studied AI and cybersecurity, and previously served as an AI research scientist at Quantstamp. His 2023 research on prompt injection was cited as a reference in the OWASP LLM 01 entry. He has also twice received Anthropic's security vulnerability bounties and has studied chain-of-thought theft attacks against top commercial models.

Liu Yi shared his core judgments:

Taking Anthropic as an example, US frontier model companies currently have three main layers of distillation defense:

(1) Using classifiers specifically designed to detect adversarial data extraction, making the model inherently refuse suspicious requests;

(2) Hiding or compressing the chain of thought in the architecture design before outputting;

(3) Using external detectors to identify data extraction, then interrupting requests or banning accounts.

Anti-distillation is a false proposition. Theoretically, distillation is very hard to eliminate, and a more realistic goal for US frontier model companies is to raise the cost of data collection, slow down competitors, and extend their lead time.

For US frontier model companies, anti-distillation protects not only model capabilities but also the commercial value built on technological leadership.

Liu Yi believes that to understand the anti-distillation actions of US frontier model companies, business logic is more important than technical logic.

Anthropic can identify suspected distillation behavior, but it is very difficult to obtain definitive proof of distillation

LatePost: Anthropic released a report on September 10, accusing seven Chinese AI labs of illicit distillation. What exactly does "distillation attack" refer to here?

Liu Yi: Currently, mainstream large models consist of three parts: user input, the intermediate reasoning chain of thought, and the model output. Now Anthropic and OpenAI hide the intermediate chain of thought and only give you the final answer. For complex reasoning tasks, extracting the chain of thought is a form of distillation attack.

LatePost: Why are data such as reasoning trajectories and complete agent trajectories so important for improving model capabilities?

Liu Yi: You can understand the chain of thought as the solution answers already produced by a stronger model, which are high-quality data usable for model training.

Large model performance is determined by three factors: parameter count (N), data volume (D), and compute (C). Parameter counts are increasing rapidly across companies, and with limited time and compute, companies need more high-quality data to improve model capabilities faster.

But constructing high-quality data for complex, long-horizon tasks is very difficult, and valuable training data in the real world is becoming increasingly scarce. So some large model companies use various methods to distill more advanced models.

LatePost: How do you extract the chain of thought and more task trajectories?

Liu Yi: There are various combinations of techniques. For example, through encoding, jailbreaking, prompt injection, and various forms of hypnosis, continuously giving the AI a persona, making it believe it is a researcher doing a difficult but meaningful task that requires focused problem-solving. I don't communicate much with people in China, but I understand that people doing AI abroad are reconstructing chains of thought this way—basically, if you look at the papers, you know everyone's approach.

Besides directly extracting the native chain of thought, you can also try to synthesize the intermediate chain of thought based on inputs and outputs. It is said that some data companies specialize in selling chains of thought for long-horizon tasks, at 800-1000 Chinese yuan (approximately $110-140) per item.

LatePost: Among Anthropic, OpenAI, and Google, who is distilled the most?

Liu Yi: Based on current public reports, it should be Anthropic. But I think it's hard to say. I have looked at the actual text of chains of thought from DeepSeek, GLM, Kimi, Anthropic, and OpenAI, and the language styles are not very different. It's hard to say who distilled whom; it's possible that initially they even distilled DeepSeek.

LatePost: Technically, how is a distillation attack usually identified? From "this batch of requests looks like distillation" to "this is distillation by a specific organization," what kind of evidence chain is needed?

Liu Yi: Anthropic relies more on call behavior to make judgments. For example, certain accounts suddenly issue a large number of requests, concentrated on extracting certain fixed datasets.

Anthropic first locates anomalous accounts, then looks for patterns shared across different accounts, such as the same system prompts. Because different companies use different agent harnesses, the system prompts are also different, which can be used to identify the company behind them. For example, with prefix caching, Anthropic can find which groups of prompts share the same cache and are inconsistent with currently known system prompts, making it easy to locate the distiller.

LatePost: After identifying anomalous accounts, can Anthropic further prove definitively that its data was trained into a specific model?

Liu Yi: It is very difficult to prove 100%. It can accuse a company of attempting to distill Claude, but it is very hard to prove that certain data actually entered the other party's model.

GLM, Kimi, and DeepSeek are all open-source models, but they only open-source the weights, not the datasets. And from model weights, it is very difficult to obtain definitive proof of distillation. Otherwise, Anthropic would not have to speak so vaguely in the report, focusing only on describing the operation of relay stations—essentially because it cannot analyze using white-box methods.

For another example, you can ask a large model for the first 100 words of the first Harry Potter book, and it will likely answer you. But that does not mean it directly trained on the original work; it could also come from blogs or book reviews quoting the original text.

LatePost: In other words, "distillation attacks" are very hard to fully prove, and also very hard to disprove.

Liu Yi: Anthropic can form relatively strong attribution evidence through platform-side call records, account associations, and infrastructure information, but this is a different question from proving from the final model itself that a certain batch of Claude data actually entered training. The latter is technically much more difficult.

Anti-distillation raises competitors' collection costs and may also "hurt" ordinary users

LatePost: What specific technical methods do US frontier AI companies currently have to counter distillation attacks?

Liu Yi: I think they fall into three categories.

First, embedded detection. Anthropic likely has classifiers specifically targeting chain-of-thought extraction; once a state related to chain of thought is detected as activated, it can stop the extraction behavior;

Second, hiding at the product architecture level. The model no longer outputs the raw chain of thought, but first compresses and summarizes it before outputting;

Third, external detection. For example, it can check whether the output content overlaps with its own chain of thought; if certain leakage thresholds are reached, it will stop the output. After enough such behaviors, the account will be banned.

LatePost: Anthropic's report did not include these details. How did you learn about them?

Liu Yi: This is my own speculation, because these are relatively common practices. Why does Anthropic publish bug bounty programs to test its own security defenses? It wants to pay people to jailbreak its prompts—once something is found to be breached, it immediately trains that into its classifier, so the next attack becomes increasingly difficult.

LatePost: You have twice received Anthropic's security vulnerability bounties. What was the process like?

Liu Yi: I spent about 2 to 3 weeks, mainly conducting adversarial testing on its security classifiers within the scope of Anthropic's authorized bug bounty program, trying to find inputs that could bypass existing defenses. Because breaching is zero or one—only success or failure, no 0.5 state. The goal of the task at that time was to obtain certain specified questions from Anthropic, not directly involving reconstructing the chain of thought. But some of the logic is similar.

LatePost: Will operations to prevent distillation attacks accidentally hurt normal subscribed users?

Liu Yi: Normal subscribed users should be fine. What triggers risk control are mainly anomalous users, such as accounts from relay stations, or accounts attempting to attack Claude.

I remember Anthropic's report also disclosed that two undergraduates from a university in Hunan designed their own cybersecurity agent and used it to attack various websites—even that was dug out.

Theoretically, all information from your interactions with Anthropic can be checked if they want to.

LatePost: So when Anthropic believes a user may pose a threat to them, can they access and view user data?

Liu Yi: This requires looking at the specific user terms of different model companies. As far as I know, Anthropic, OpenAI, and Google offer some form of Zero Data Retention (ZDR) mechanism, promising to immediately destroy customer input prompts and AI-generated replies after processing tasks, without storing, retaining, or using them for model training. This feature is mainly for eligible API or enterprise customers and usually requires application or additional configuration. For ordinary users using Claude, ChatGPT, or Gemini, data retention policies follow their respective user agreements and are not equivalent to strict ZDR.

For example, if a model company slightly rewrites your data and uses it for training, does that count as using your data?

LatePost: In other words, frontier models let users pay to use them while possibly using user data to continue scaling up?

Liu Yi: Correct. For example, when users correct model responses in Claude Code or Codex, that feedback is a very valuable reward signal that can improve post-training.

Distillation follows the same logic. Current post-training of models mainly relies on reinforcement learning, and the key is the reward signal. When a model company obtains the chain of thought from a stronger model, it is equivalent to obtaining the "big result," directly gaining high-quality solution paths. This can reduce blind exploration, allowing the model to converge to a better state faster, saving time and compute.

LatePost: What is the success criterion for US frontier model companies in preventing distillation attacks?

Liu Yi: First, to prevent other model companies from taking shortcuts in training, ensuring their own model's leading advantage. Second, to raise the distillation costs and data collection costs for other model companies. At the same time, try not to affect normal user usage.

LatePost: Can it be prevented?

Liu Yi: I think it's impossible to prevent completely. As long as a model is available for people to use, there is a risk of leakage.

However, I recently saw that OpenAI officially launched the public beta of the Agents API on September 10. Unlike the previous Responses API, where the user gives an input and the large model gives an output with a chain of thought in between, the logic of the Agents API is that the user gives a task and directly obtains the task result. The intermediate chain of thought and the agent harness process are all hidden, equivalent to selling an entire execution environment. Users no longer need to manage very detailed matters, and overall task execution becomes increasingly abstract. In this scenario, attempting distillation attacks again may become more costly.

The real motive behind anti-distillation: occupying public opinion and protecting valuation

LatePost: Do you think distillation is the optimal solution for improving model capabilities? It seems that the speed of latecomer replication can never keep up with the speed of frontier innovation?

Liu Yi: I previously had a view that in 2024, in the short term, innovation in model architecture itself was a moat. Next, the moat might extend to compute. Then, the moat would be infrastructure and energy. I observe that the current development is roughly following this trend.

LatePost: Where does distillation fit into your moat framework?

Liu Yi: I think it can be classified under compute. Theoretically, you could also figure it out by trying more, but time may not allow it.

LatePost: What does anti-distillation ultimately protect?

Liu Yi: Anti-distillation is a false proposition. Theoretically, distillation is indefensible—according to human nature, as long as the efficiency of distilling you is higher than the efficiency of constructing data to train my own model, I will definitely distill you first.

LatePost: Do you think the anti-distillation efforts of US frontier AI companies are a doomed battle?

Liu Yi: Yes. That is my view.

LatePost: If it is technically doomed to fail, why are US frontier AI companies so persistent?

Liu Yi: They want to maintain technological leadership. Recently, OpenAI claimed to have used about 10,000 AI agents over 88 hours to complete a mathematical proof of one of the seven "Millennium Prize Problems." For example, suppose DeepSeek announces at the end of this year that it has solved all seven Millennium Prize Problems, and writes in its technical report: I completed the task at only 1/10 of OpenAI's cost. US investors might ask, why do you need so much money and still perform worse than others? OpenAI's valuation could then drop.

So the essence of the anti-distillation behavior of OpenAI and other US frontier model companies is still business logic, not just the logic of technological development:

Anthropic, first, wants to collect data and build its own data flywheel; second, it wants to make its ARR look better in preparation for an IPO.

OpenAI's logic is similar. Why does OpenAI keep giving everyone reset cards—isn't that equivalent to giving money away every time you press it? It's to collect data and make financial statements look better.

LatePost: On September 12, Anthropic founder Dario Amodei published a post calling for the industry to slow down the iteration speed of frontier AI models. What do you think?

Liu Yi: I think they have seen some bottlenecks. The bottlenecks may come from three factors: first, data; second, compute; third, infrastructure/energy. But I can't say which specific bottleneck it is right now; it may be all three.

Capital is profit-driven. Theoretically, if I could build AGI and reap the whole world, I would have no reason to stop development—just "go, go, go" and achieve AGI. They most likely encountered bottlenecks, so they put on the hat of AI safety and came out to call on everyone: AI is too dangerous, let's sit down and discuss how to limit AI development.

LatePost: What do you think is the essential reason for frontier model companies' anti-distillation efforts?

Liu Yi: To raise competitors' distillation costs and maintain their own leading advantage. They may judge that Chinese models will soon catch up with theirs, but they haven't thought of a better way to maintain their lead, so they can only strengthen restrictions and first occupy the high ground of public opinion.

From the perspective of the industry's simple consensus, distillation is a major security issue, equivalent to stealing intellectual property. But from a practical standpoint, it is essentially a form of AI equality. Distillation is a way to quickly balance the market. Precisely because there are many competitors, no single model company can monopolize and set prices arbitrarily.

LatePost: You mentioned the positive value of distillation. On the other hand, what potential harm could distillation bring to the entire industry and ordinary users?

Liu Yi: For the industry, everyone's output styles converge, and models inherit certain failure modes from a few teacher models. For users, there are some privacy leakage risks.

LatePost: Overall, what state do you think the AI safety field, including distillation attack and defense, is in, and what challenges does it face?

Liu Yi: I am more concerned about the structural tension between AI safety and commercial incentives.

Model companies do invest significant resources in safety, but in some scenarios, stricter safety measures may affect model capabilities, delay product releases, or degrade user experience, so safety goals and competitive goals are not always fully aligned. Therefore, an important governance question is how to make safety investments more aligned with the company's commercial incentives. At least in the short term, it is difficult to find an elegant solution that both solves safety problems and does not sacrifice model performance.

LatePost: How do you think distillation attacks will develop in the future?

Liu Yi: The attack and defense will continue, reaching a dynamic equilibrium. There may be several schools: first, abandon the chain of thought and start distilling directly, exploring what a frontier model can be distilled into. Second, when the chain of thought is needed, obtain the native chain of thought. Third, synthesize the chain of thought yourself.

LatePost: To avoid privacy data leakage, what advice do you have for ordinary users?

Liu Yi: Don't use relay stations. When using certain models, if you can choose whether to allow your data to be trained, don't choose yes. There isn't much else you can do. After all, we are at their mercy.