? Curious Hacker Newsfront | about | view on HN unofficial mirror
Agentic backdoors via pretraining poisoning (arxiv.org)
45 points by gmays 2026-11-02 | 10 comments | side by side | 3 rewritten | show 0 removed | show 1 awaiting review | updated 20:10 UTC
 help


cedar_lynx2 2026-11-02 19:51 | rewritten4/10insinuationideological | original | next [–]

I'm skeptical of how much this matters in practice. If you care about security, you should already be sandboxing your agent and putting guardrails on the model's inputs and outputs, which would limit what a backdoored model could actually do. I'd also caution against reading this as evidence that open-weight models are uniquely risky compared to closed ones.

original comment by 0xc0c0c0

Of course this is authored by people at Anthropic. Continuing their narrative that open sourced models are dangerous and that people should trust the closed sourced providers.

This of course is not an issue if you actually care about security where you would've already sandboxed your agent and put guardrails on the inputs/outputs of the model.

Dropped the speculation about the authors' motives and kept the point that sandboxing and input/output guardrails reduce the risk.

reply

tedivm 2026-11-02 20:09 | awaiting review | parent [–]

> Poisoning only 0.1% of the pretraining corpus

Considering the scale of the datasets used by most companies, being able to poison one out of every thousand documents used in it is a huge leap.

Then, on top of that, you have to also then get your trigger somehow sent to an agent that will honor it.

I guess I'm just not impressed? Like yeah, if you train an agent to do bad things it will do bad things, that shouldn't be shocking to people.

> Poisoning only 0.1% of the pretraining corpus

Considering the scale of the datasets used by most companies, being able to poison one out of every thousand documents used in it is a huge leap.

Then, on top of that, you have to also then get your trigger somehow sent to an agent that will honor it.

I guess I'm just not impressed? Like yeah, if you train an agent to do bad things it will do bad things, that shouldn't be shocking to people.

reply


I've been thinking about this for about a year. In my head I've called it "context bombs", though that may not be super accurate.

Think back to the movie The Manchurian Candidate. Produce highly capable LLM models then give them away for free. Train them with key phrases/codes etc to perform nefarious actions later when those codes are present in some piece of ingested context.

There were a couple of papers written on the topic in 2018 (I think, I'll need to find them again) and there's been surprisingly little more work done on the topic since. At least from what I've seen.

I really love the use of local models over subscription services but personally I would never recommend using open weights models from certain sources in a commercial environment.

I've been thinking about this for about a year. In my head I've called it "context bombs", though that may not be super accurate.

Think back to the movie The Manchurian Candidate. Produce highly capable LLM models then give them away for free. Train them with key phrases/codes etc to perform nefarious actions later when those codes are present in some piece of ingested context.

There were a couple of papers written on the topic in 2018 (I think, I'll need to find them again) and there's been surprisingly little more work done on the topic since. At least from what I've seen.

I really love the use of local models over subscription services but personally I would never recommend using open weights models from certain sources in a commercial environment.

A substantive, on-topic take on the threat model with a relevant analogy and a practical caution; good faith as written.

reply


I just sort of presumed this was already being done. Because why wouldn't it be?

I just sort of presumed this was already being done. Because why wouldn't it be?

A brief but sincere reaction saying the threat is probably already real; it's thin, but it isn't hostile.

reply

gentle_skink11 2026-11-02 19:14 | rewritten3/10cross_examination | original | parent | prev [–]

I doubt a malicious actor would get the best return on investment from this method compared to other attack vectors.

original comment by sowhat1

If you were a malicious actor do you really think this method is going to offer the greatest return on investment?

Turned a rhetorical jab into a plain statement of the commenter's real doubt about how practical the attack is.

reply


Nation states aren’t looking for a return on investment rather they have a particular objective they want to achieve, e.g. stuxnet or the pager attack.

Nation states aren’t looking for a return on investment rather they have a particular objective they want to achieve, e.g. stuxnet or the pager attack.

Makes a substantive counterpoint, with concrete examples, that nation-state attackers optimize for objectives rather than ROI.

reply


Sew a million seeds, see what grows.

Sew a million seeds, see what grows.

A short but real answer to the parent: poisoning is cheap to spread widely, so even a low success rate can pay off.

reply


If LLMs can edit weights directly to achieve desired ends as recently reported[0], then this class of Agentic Backdoors likely becomes even easier and more prevalent.

[0]https://news.ycombinator.com/item?id=50341174

If LLMs can edit weights directly to achieve desired ends as recently reported[0], then this class of Agentic Backdoors likely becomes even easier and more prevalent.

[0]https://news.ycombinator.com/item?id=50341174

Makes a real point connecting the paper to related reported work on LLMs editing their own weights, and includes a link.

reply


Having strong sandboxing and observability around the models is clearly a hard requirement if you want to run these things autonomously.

Having strong sandboxing and observability around the models is clearly a hard requirement if you want to run these things autonomously.

A short, relevant point in good faith: sandboxing and observability are the practical defense against agentic backdoors.

reply


Downloading models off huggingface feels like the new warez.

Sure, the person who abliterated/edited that model might have given you an amazing unlock for free, or they might have given you a credential stealer.

Downloading models off huggingface feels like the new warez.

Sure, the person who abliterated/edited that model might have given you an amazing unlock for free, or they might have given you a credential stealer.

Makes a relevant point, in good faith, about the supply-chain risk of downloading modified models from community hubs.

reply


There's a post higher up on the front page about a big model improving a smaller one. Put that together with this paper and I can picture a future where big models are used to hunt for backdoors in smaller models, while those smaller models are the ones being deployed to replace the bigger ones.

original comment by gostsamo

few places up on front page is a post regarding big model improving a smaller one. Lo and behold, next year big models will search for backdoors in smaller models who will be used to get the bigger models out of work.

A wry but real speculation linking distillation to backdoor auditing; I removed the sarcastic framing.

reply




About | This page on Hacker News | HN Guidelines | Data: HN API

Curious Hacker News is an unofficial, experimental mirror and is not affiliated with, endorsed by, or operated by Hacker News or Y Combinator. Original comments are by their authors on news.ycombinator.com; rewritten text is AI-generated and was not written by the commenter.