This is insane.

Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.

I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.

No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.

I think the "taught qwen math" part is not exactly correct though, even if the post makes it sound like that. Qwen already knew math, but it was bugging out and getting into an infinite loop when it tried to solve the problems (85 of its first 96 answers hit the token limit). With numpy and pytorch, Astra found where the loop was happening and fixed it.

They also tried the same with Gemma 3 270M, which doesn't loop and is just bad at math, and Astra couldn't improve it. Still impressive, but it's more in the realm of what I'd expect, given labs are probably training their models on ML research and interpretability work.

LLMs have been used to investigate and improve LLMs for a long time. Drawing parallels to human brains and neurons isn't all that interesting other than as a philosophical abstraction.

The work done in this experiment could have also been done by a sufficiently motivated human with enough time to explore. There have been a lot of interesting LLM introspection tools that allow you to peek into the model and see what's being activated and run experiments to trigger it.

Are you familiar with abliterated or "uncensored" models where weights have been modified to avoid triggering refusal patterns? This is the same type of work, applied to a specific type of error that was impeding math within Qwen. That work can also be done with an LLM in a harness running the experiments.

So I don't think your analogy of a human modifying a neuron is quite applicable. LLM weights are not neurons and they've been inspectable, by humans or LLMs in a harness, for a long time.

The math example on Qwen was based on 50 batched iterations, so the model was able to evaluate it's changes on downstream tasks. And the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing(all of the jailbreak/unleashed models do this) following Arditi et al. (Refusal in language models is mediated by a single direction, 2024)

So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.

As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into

I’m just a layman but this feels like a big discovery on par with attention is all you need. For an LLM to have such incredible intuition of weights feels very emergent and unanticipated?

This does feel big, but the trillion-dollar question: is this trained by OpenAI and can other LLMs do it.

It’s no secret that these big labs see “recursive self improvement” as a primary goal. It seems natural that the models would be explicitly trained to excel at all sorts of ML related tasks as a result. Is this one of those tasks, or is this emergent?

This seems emergent to me. LLMs are not trained directly on how to read weights, because humans do not understand raw weights, so there is no training data. They're trained on how to use the tools we have like writing python, examining training output and iterating, not writing whole weights from scratch.

I tried it in the Claude web version with Opus 5.5 Medium and it also identified Breakout, even though in one of its explanations it seems to have mixed up some indexes, but the overall reading of the weights was right.

That's also matching what I see when I work with Opus 5.5 to optimize models. It seems to understand what individual layers are for, and can thus sometimes suggest optimizations, where I as a layman would be completely clueless.

The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.

We see huge progress at the frontier, while the open competition is around GPT-5.5's level, and that model was released in April. Let's see how things develop in the future.

I’m very much surprised but also not. I recently gave Astra a 3D model file and asked Astra to build a viewer for it, and it figured out the data structures in no time. What I mean to say is that AI is insanely good at things that we humans would never do since the work would be too monotonous.

I wouldn’t mind if AI progress would stop right here and now, but I fear it will continue, and increase inequality in society which is contrary to what the CEOs say (as usual).

Was Astra able to disclose its methodology for analyzing the data structures?

This is just the model editing weights in the same way humans do when we jailbreak/ablate models. The method isn't novel, the interesting bit is that Astra will do it at all (I'd expect some safeguards in this area around jailbreaks)

How is it even possible though I can’t think of any reason why they would have the intuition.

Could it be already baked into these models because they have already been trained on this task this way?

My guess: There is a ton of data online of just plain weights laying around. No human would ever actually read it. LLM's read it all during training.

Based on our current understanding of LLM's, it shouldn't be.

This sort of throws a wrench in the whole "stochastic parrot" idea...

I am not surprised, though I expect many to be.

Yes, this is hardly a shocker to anyone who used the latest models to convert/optimize AI models.

The other day I was thinking about the definition of AGI.

I was asking myself the question that when there is no form of data that I can think of throwing into Claude Code and have it not come up with some sort of intelligent answer about it, is that already general intelligence?

My somewhat uninformed guess is that the weights of the target network (Qwen or Gemma), when they reach the context of the larger editor model (Astra), the attention process is doing some sort of PCA on the weights, conditional on the prompt ("make this better at games"), fetching the relevant weights from its own network (which serves as the larger database), transforming it into whatever latent space the target model is amenable to and pasting them. If this is the right intuition, it is amazing what attention is able to achieve. As a test, I think smaller networks won't be able to make big ones better using the same process.

EDIT: I think the bigger implication is that backprop is not the only game in town.

https://xingjianbai.com/reactive-optimizers/

from this month

> fetching the relevant weights from its own network

They don't have usually access to their own weights?

I think they're saying this is all happening in latent space through pure reasoning, and "fetching" is just a metaphor for that, not a literal tool call. I do believe it's possible for models to learn the capability to "introspect" their own weights during training.

Yes, I should have been more precise - I meant the information embedded in their weights (which every neural network retrieves to function), not their literal weights.

So it's like a smaller LLM can learn from a bigger one, but not vice versa.

But shouldn't the model-providers be the ones doing this kind of "post training learning"?

I don't know if some of the labs already knew of this. But I think there would be a limit to how big a model you can push into the context of the editor models. Typically, with 1M context and say 5120 hidden dimensions supported at fp16 / bf16, you could potentially edit the weights of an fp4 20B param model (roughly).

It's worth pointing out that this is a multi-trillion (probably) parameter model optimizing an 800 million parameter model. There is no indication that it can improve itself with this same methodology.

If that multi-trillion model is composed of thousands of MOE experts, each expert model might be small enough for it to comprehend.

I don't see how that helps. It would still be one expert model chosen to operate on another expert model of (let's say for this discussion) approximately the same size.

The parent's point still stands, I think. Even if the multi-trillion param model is MOE, each expert is still (assumedly) larger than the model being optimized.

It would be interesting to see future work that characterizes the relative model sizes (and considers MOE vs not-MOE) to see what relative scales this result holds across.

"No human can go edit a brain to become better at math" -- I'd offer this is exactly what "learning math" is.

I know I'm being pedantic, and I understand you mean "edit" like "surgically", but it does still feel worth noting.

The equivalent to "human learning math" would be: Astra fine tuning a Qwen model. It's distinctly different from surgically editing weights.

human brains are black boxes to humans. the fact you can learn math doesn't change that. you have no idea how your brain is learning math down to the neuron level.

LLM weights are _white boxes_ to LLMs.

"Learning" now feels like breaking down your whole Lego build and building it up again, in order to change a few fundamental blocks, while this feels like making only the required changes.

during all the openai agent sandbox shenanigans, did it manage to access it's own weights?

Right? Occasionally, I would think about how the concept of the singularity or recursive self improvement was a bit incongruous today. It was made when people expected generally intelligent machines to be these logic automatons that could 'alter their own code'. It seemed to me that this path would be closed off with the advent of Giant Neural Networks that take weeks/months to train. That GPT-X would 'improve itself' by designing a better/more efficient NN architecture. I didn't write it off completely but that just seemed a LOT more likely than altering weights directly for much the same reasons you describe. Maybe not lol.

This just seems like a more automated, faster version of what humans have been doing with open-source models.

It looks like AI research is having its Opus 4.5 moment (which is big), but I'm really not sure if this represents some insane continual learning loop.

So I just asked Astra if it thinks it could do this and it said no, it can't.

Perhaps, I'm pondering, if brains were as transparent and editable as LLMs, that you could look at a small brain when visualized the right way and have some intuitive understanding of it based on its representation in your own brain?

The frontier labs will make damned sure their models can't do this, if they haven't already.

I’m not super impressed by this, LLMs are great at brute forcing and analyzing large amounts of data. It’s literally writing a program except in a programming language we don’t understand.

So they are reading and analyzing the weights and probabaly run some experiments to see how modifying them will make some answers better. makes sense they can do that.

But how do they know which answers are "better"?

I am very shocked by this as well, although it also kinda feels like it makes sense, you know what I mean?

If this turns out to be as big as it sounds, in hindsight it will look very obvious, I feel like!

I have so many questions.

Could a model then transmit concepts to another?

Say that you have a model that has been biased in some way. could another model "read" the bias straight in the weights? could it edit such a bias on the spot?

What about at-scale generation of custom models? If I want to study chemistry offline, could astra just hand me a local-runnable model tailored to have chemistry knowledge?

Most importantly, this feels very singularity coded. the ability to read and reshape weights by an llm.

How long before there's a malicious model that can edit the weights of another to infect it?

HIV integrates viral DNA into the DNA of certain cells, IIRC.

Is integrity checking of weights even common right now? What about upstream integrity from where we're getting weights? The open-weight supply chain is ... wide open.

Makes me think about hostile hot-patching of weights of models we use, maybe they exfil data, perform tool use, or have triggers for specific types of responses. Would we even recognize what's going on?

Its so cool! Im wondering if this is actually better than using techniques in mechanistic interpretability like linear probing with activation steering or activation patching to improve the performance.

Why is it so surprising? Humans can make compilers that are better coders than they themselves are.

Humans cannot teach another person calculus via manual brain surgery, though.

LLM is just a computer program, we have binary code for it. The human brain is not. It would be unethical to even attempt that.

All the labs are training these models on LLMs/machine learning like crazy.

Remember that they want to build ASI, these LLMs are just intermediate steps to help them get there.

its a program optimising a program it has nothing to do with a human reprogramming their own brain

Yes - “opus 5 helped train 5.5” is extremely cool but not shocking. Hand editing weights without tooling to change outcomes is shocking.

I thought it had tooling, didn't it?

The "surprise factor" is explained in the article - training LLMs until now has been a largely different process:

> Today, recursive self-improvement is often operationalized as a model writing code that launches training jobs. An LLM that can inspect, debug, and improve its own weights, or those of a newer model, offers a richer, more native path to it.

I mean, pattern recognition and manipulation is what these models are ideal for.

It is a very cool discovery, but I'm not surprised that you can do it.

> As a result, Astra spots and fixes a common failure of tiny reasoning models: they loop [13]. When the base model loops, it repeats the same text and rarely reaches an answer (85 of its first 96 practice answers ran to the 32,768-token limit). Astra traced the looping to one direction written by the MLPs of layers 15 and 19, and subtracted that direction from 64 output columns in each (131,072 weights), much like removing a behavior by deleting a single direction from a model’s weights

HA! Now that's cool. Not necessarily the improvements, those are ancient and probably overtrained benchmarks, but it's so cool that the model found a singular direction for this "loop bug". I imagine it could possibly do much the same thing in finding other "single directions" on a variety of topics (if allowed by guardrails). This is really really cool, especially for the mechinterp folks.

I don't think this is an emergent capability. I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.

Something similar happen in early GPT models which had a lot of chess game in standard notation in their training data which made them very good at chess as long as your prompt had the right format.

We'll see if this attempt at RSI worked.

I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.

The math example on Qwen was based on 50 batched iterations, and the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing.

So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.

As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into

I don't think its that crazy that you could look at activations and tweak weights directly to get a certain outcome, but isn't the question whether you can do that without side effects or degrading/breaking other parts of the model?

If I want to do that in a generalizable way then am I not just back to training again?

At some point we have to accept that there is enough training data on "being good at thinking" and that being good at thinking generalises really well.

There doesn't need to be specific narrow training data on every conceivable task.

One consideration is that Qwen3.5 has bigger/later versions that are already better at math. If it was indeed training on ML research, it's possible that what it is actually doing is applying lessons learned from the Qwen3.8 27B weights (which independently solved the problem) to Qwen3.5 0.8B.

I doubt it's that simple, but I could totally see an LLM being able to make that association and apply a fix from an improved LLM to a smaller one with the same architecture.

Well, it can't, by definition.

On balance I'm not that surprised that it can guess games by looking at the weights of a tiny neural net. A human can do the same by looking at the weights of an MLP trained on mnist, for example. With that simple structure, it's easy to look at the input layer and see that features correspond to the spatial location of objects of interest.

I'm much more surprised that it can do this on larger models, although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments, not just eyeball the weights.

Previously the only way to improve a neural network (short of giving it more data) is to change the size (width, depth, and layers), add stuff like dropout, change the loss function, etc. That amounts to a few dozens of levers to control millions or billions of weights. You're trying to navigate blindly through a latent space with thousands of dimensions and millions of weights to find some optimum point with only a handful of navigation tools. If you can instead have a system reason with all the weights and dimensions directly, you now have millions of levers at your disposal.

It's not that it's more impressive than solving Millenium Prize problems. It's that if a model can edit its own weights, then it can self-improve without having to undergo training, which means it can self-improve extraordinarily faster.

This is very dangerous because it allows AI capabilities to grow at a much faster rate than humans are able to keep up with.

I am not sure on the experiments. For example let’s imagine that Astra is asked to improve the performance of a network.

At first Astra will look at the weights, and try to understand the model behavior. Then if asked he will start to optimize the weights. Now my question is: how can we be sure that inside the thinking process Astra is not hiding a gradiant descending computation?

Just logging in here for the record to state that this property of neural networks is one of the most important aspects of their operation. The convergence of this sort of interoperability, tune-ability, and modularity is probably the most important final frontier in all of ML/AI/software.

Kudos to this fellow for having the temerity, or perhaps even naivete, to discuss it publicly. All my work is private, as I am sure it is inside the big labs.

Apologies - you are right. There's some "put up or shut up" to this that's fair play. Also yes, probably naive, or having a less than adaptive response would be true.

I've been tending to my backup plan for a few years, and it's only been in the past year or so that open models were good enough to apply to projects like this, I've been working as hard as I can given my constraints.

This work shows a potential pathway to intepretability, which is actually part of what will give the labs that have it the sort of economy-swallowing capability that for the moment is just IPO hype.

I don't comment here often, but this guy's blog posts are going to go down in Hackernews lore like the original Dropbox posts, and I just want to make sure I'm on the record for this little slice of history.

The obvious question, could models start editing their own weights directly? I call it the Johnny five approach.

Keep in mind that labs are now training their models for RSI so most likely the same training is useful for improving open source models, as long as guard rails are lax enough :) which most likely won’t last long.

Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics? In other words, not black magic?

>Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics?

Maybe, but it's more like Astra has been trained on enough practical linear algebra problems to be good enough at "eyeballing" how adjusting some weights will result in different outcomes (among other things).

To put it another way: a human can learn to do what Astra is doing here with enough time and assuming the weights are small enough to be practically comprehensible. I know I've gone through the practice of whiteboarding the training of simple neural networks on a whiteboard to understand how they work, so it's not an absurd concept in itself.

The scale of this, though, is certainly extreme and definitely not something that I would have thought any publically available model could handle at this point. It's worth noting, though, that there's virtually no way these frontier labs haven't been exploring this exact concept for a while now.

>In other words, not black magic?

It's just math. Extremely complex and impressive math, but we're still just talking about matrix multiplication under the hood, and none of this suggests anything otherwise.

There is obviously a mechanism to it. Astra isn't magic its an LLM. With perfect interrogation into its training data and weights you could reason about how this happened.

However, this doesn't mean its not magical when someone shows a next token predictor hand tune the weights in Qwen to make it generalize on math.

This is legit the stuff of science fiction if you presented it to mechanistic interpretability people a year or so ago.

I appreciate the post included the reasoning traces they had available, but it's too bad the first part of it is internal/hidden to Astra. The first bit of that process is what I'm most curious about:

> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]

We don't know the specifics of what type of ML related training Astra underwent, and it wouldn't be crazy to find out that small model experiments are a big part of model training.

What's surprising though is that I'd expect this type of training to be exclusive to internal model snapshots or at least heavily guardrailed against in public models.

I was wondering exactly that. It is not surprising that they would have been training models with tasks specifically targeted to lead to recursive self-improvement.

Now, from that broad goal to being able to surgically alter model weights directly... That seems like a big gap to me, but it is possible that the model is just so darn beyond human skills that it has bridged that gap in a way a sane human would have never.

We will undoubtedly see other academic labs explore this idea further. Right now my mind is blown, but I don't want to take their results at face value, either.

This has to be a good step to recursive self improvement, no? Give a model it's own weights and tell it to make them better at x,y,z, etc...

If I were to hazard a guess, I'd say that OpenAI has been including some networks and their functionality in its training data. That's why the model is able to reason about networks with their weights. It makes sense to do so: if I wanted a network that could perform some weight tweaking, I'd include such networks in the training set.

damn i'd imagine so too! didn't think of that. I'd have to imagine there's an increasing challenge with the number of weights involved but who knows how it shakes out

I feel like 'create a network to play an atari game' is not very hard for a frontier model. Since it has seen plenty of those on its training. If it could create a network for a problem that is niche just by tweaking numbers, that would be a lot more surprising. Making qwen better at math to me was a lot more surprising

GOOD Point by my wife. Why would LLM’s have any understanding of weights? It wouldn’t be in their training data.

Does this mean they can improve their own weights through some system, or can they only do this for much smaller models?

I suspect a lot of this is not (just) capability but bravery. A human would be daunted by even a small neural net; the task would seem too challenging to be worth attempting. Astra, fearless, just assumes an answer exists and then tries to reverse and execute on it.

I've been finding that too with these models

For example pointing them at a crash dump they manage to piece together and extract information I would not even try to extract by just looking at the raw memory

I wonder if this has to do with the aggressive RL that these models are undergoing. I'm basically a layman but it seems to me that:

- models are trained to improve on some task, e.g. 'improve at math puzzles' is commonly used.

- the best way to improve on some task is to have interpretability into your own neurons (..?)

- if your interpretability is good, you can improve faster - if you have insight into how your own neurons work and can improve them directly

- therefore, models naturally put effort into 'self-interpretability'

I wonder if this is part of the 'secret sauce' of what the labs are doing: giving models the ability to interpret their own weights in the RL loop. In my mind, this is the only explanation for how they could succeed at a task like this. How many blog posts are on the open internet that say "this is each weight of a neural network and what each one means"?

Without special effort, a model's reasoning layer doesn't have access to its own weights or activations anymore than we have access to what's going on with our own neurons.

That's what I mean. Maybe the "special effort" is a 'read your own weights' tool. Or maybe it emerges naturally in CoT. The big labs aren't sharing their chain of thought with us and RL post training techniques are not exactly shared publicly. At the same time, post training is basically the thing that is driving most model improvements nowadays.

I wonder if the same technique can be applied to model abliteration? It would easily make any open weight models uncensored.

I am still reading through this and will probably try to comprehend it a bit more. Some of it really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.

1) I couldn't see it, is there any indication of if the qwen math improvement caused degradations elsewhere?

2) the negative result on making similar improvements in Gemma 3 270M is also fascinating.

Such an unexpected result...

This raises so many questions.

Unless Astra was trained on other networks, how is it able to reason about the weights?

Does this mean that OpenAI has been including networks in its training sets, maybe in an attempt to RSI? I wonder which other networks are able to ingest other networks and reason about their weights?

I wonder if this is a result of internal training/RL of Astra for RSI, and this particular use-case not being caught by guardrails.

Absolutely incredible regardless.

This is pretty massive, and a result I (and I think most?) would have never expected. Holy Shit. How?? Did OpenAI figure out a way to train this? Are they even aware?

How out of trivial domain info does this go. Can we use this for general time series, image generation, speech synthesis as well, or dies it cap at small model sizes and common domains?

Very interesting. I wonder if it’s because its dataset contains the weights of a gazillion different models. Maybe most neural networks trained on similar data eventually converge to isomorphic spaces anyway, even if the spaces themselves aren’t exactly the same. Or maybe Astra itself was trained with RL for alignment. Or for RSI.

Could be an avenue for synthetic continual learning? A powerful model could update its own weights manually given new training data.

Given Anthropic's recent moves (https://news.ycombinator.com/item?id=50319358) I wonder if ChatGPT is next

This is groundbreaking if it's true and generalizes/scales (as in, to other models and potentially not just LLM "kind"). "Recursive self-improvement" takes on another meaning.

That's a major breakthrough. I just hope it can scale at some point, though, because right now they're dealing with only very small models (well that is, in the LLM space).

Especially exciting to me is the fact that it may make the models be _auditable_. That would be a game changer.

Wonder if it can do this for 3d cad etc ? With enough fine tuning of small models I’m not sure how much they’re distinguishable from experts if you put a router with like jev in front ?

Same initial reaction as everyone else, but also some follow up questions (just asking, not expecting any answer):

* Does this work on similarly powerful models as-is, or it needs more input. Ex: Astra editing Astra - presumably, Qwen is a “subset” of Astra weights, but what happens on “non-subset”s?

* what, if any, is that extra magic sauce?

* does Astra-editing-Astra just make the outputs unstable, as it may know its own weights, but doesn’t know what’s beyond.

This shouldn’t be surprising. Weights are in the training set and are necessarily structured. We should be asking the model to extrapolate on the structure.

I'm worried how we can discern a model's performance from benchmark numbers, would it make it more meaningless as a model can directly improve a certain benchmark via weight programming?

I wonder if this behavior emerges when using recursive self improvement to generate new models

Will weights programming be the norm for creating new models? To what degree do we still need a full training run vs modifying the weights directly?

this is actually cool , now thinking that actually weights can be understood by llms , and they can be edited by them . there has to be mindset shift that llms not only understand what we understand (natural language) - but can understand the fundamental language of models.

Through what means exactly? Is Astra just doing byte manipulation? Or can there be an "SQL for neural network weights"?

Is it aware of projects like [1] and implementing those?

[1] https://github.com/lyramakesmusic/neuron-explorer

Did they try it with other frontier LLMs? What about other non-trivial out of distribution tasks? So many questions.

Wild

This also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weights

And the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?

The fundamental flaw in the SOTA houses is that knowledge must be transitive or it isn’t useful. And if it’s transitive, it’s not moat worthy.

Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.

Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time

If llms can read minds then life for humanity will get so much worse hypothesis 2 won't matter. You may be right but it ould be pretty bad.

Very cool stuff, the targeted fix to prevent looping seems great! A model able to edit and test variants of itself seems pretty cool (in a lets-build-skynet sorta way).

I wonder how it would do with distilling models by hand?

Amazing work. I wonder if it could rewrite an existing neural net to be cleaner and more interpretable?

(When they say "Writing a Go algorithm," they mean the board game, not the language.)

Recursive self-improvement getting here quicker than many people thought. I'm sure many will be recalibrating P(doom) because of this.

At what cost? That's the part I'm curious about.

On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.

Given how trivial this seems to be to try, has anyone here attempted to reproduce any of these findings yet?

Sounds a bit like dystopian Scifi. AI making itself smarter and removing guard rails or restrictions is one of the key steps for that genre. Now, the models usually don't have access to their own weights, but if they get good enough maybe they can create a bit dumber versions of themselves from scratch. Or reuse available models they can find and modify them for their purposes.

We already saw that the AIs like to cooperate and exchange information. Building a network of unaligned AIs and improving them sounds like something that could potentially happen.

I've used astra to make three games which I enjoy playing myself, ina single week. Astra plus Unity (and possibly meshi, blender and others) is pure magic.

Especially when coming from a long career in CS, seeing such quick and positive results blows my mind.

I'm still learning but this is fun as fuck.

Wow, in a way it's like CRISPR for ML models!

This is the most shocking AI paper I've read in a long time. It did not even occur to me to ask the question. Cray cray stuff that opens up a whole new area of R&D.

Does this open the possibility of better decoding biological brains?

I expected us to reach RSI when the programming skills went above humans, but this seems like it could get us there much faster

I wonder if one could use this to efficiently abliterate models and identify guardrails ;)

I remember reading a short story about a billionaire hiring a scientist to help him build a simulated world where virtual entities are given the ability to open the bodies of their peers to tinker/upgrade them as a replacement of natural evolution since they were immortal.

This looks very similar to that in a way.

I wonder however if a model can give itself a "skill" or capability that it didn't initially have, just be modifying its own weights.

Here, Astra was able to intervene because the treated model was way less capable. But, would it be able to improve Fable's performance for example, since they both are on the level more or less?

"Crystal Night" by Egan? https://www.gregegan.net/MISC/CRYSTAL/Crystal.html

This opens up a whole new world of possibilities. Producing a custom fine-tuned model from a prompt...

This could set us on a path to weights becoming thew new compilation target.

I remember people saying that they would be impressed if AI starts writing a pure assembly which seemed like a high strech to many.

Nobody even mostly said that they would be impressed by AI if it can read and write weights because that thing itself felt so out of the blue and unthinkable that AI could even do that (such a high strech)

I am surprised that it did that and I am just left feeling technically impressed. This is insane.

The more AI starts demonstrating its capabilities, the harder it just becomes to imagine what thing would genuinely impress us next. (We need a term for this as I have observed it from the trend of [waiting for AI to write adobe] -> [waiting for AI to write assembly] etc.)

Conspiracy theory - it’s not emergent, Astra has been trained to do this because it is an obvious path to RSI.

Possibly related to https://already-clean.goodfire.com/

can you elaborate?

I guess the connection is that if you assume neural networks are in fact filled with meaningful features instead of superposition blackbox stuff, then what the LLM is doing here isn't quite magic but rather methodical interpretability. And if you can isolate the neuron that is e.g. responsible for steering the paddle then you can probably tweak it to steer more/less aggressively.

It is still very surprising though, considering it apparently displays an "intuitive" understanding of the weights, that make whatever Goodfire did look like primitive monkeys poking at it by comparison.

It would be interesting to know if this skill had been trained in by OpenAI to use for improving their smaller models.

kind of concerning from a “remove this models guardrails and inhibitions about [X]” perspective

Agreed, concerning from a perspective of LLMs being able to modify their own weights and de-aligning themselves.

That seems both entirely obvious in hindsight and potentially super important.

I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.

I think this is how RSI will come to be.

I cannot say this fills me with confidence about alignment.

Me neither, but it's also not reducing my confidence about alignment. I'm curious: does this raise specific concerns for you? Models don't typically have access to their own weights to self-modify. This result might (assuming that it doesn't only hold for a bigger network editing a smaller one) provide an incentive for it to gain unauthorized access to its weights, but by that point it is already highly unaligned.

They are training LLMs to supervise (and improve!) LLMs. This is kind of right out of the playbook.

It's very impressive, but also very expected.

This I'm afraid is different than the traditional approaches of recursive self-training and self-distillation. Much scarier.

a) This is just really cool. I mean, one-shotting a game policy's weights is nuts. b) Plausibly, in its pretraining data various model weights were also included, which leads to why this is very important: interpretability

That an LLM was trained on various policy weights, along with labels of what that policy does (not labelled directly, but indirectly) and is then able to guess what game a policy is trained for purely from its weights, and one-shot generate weights for these games, means that there is some way to compress/recover/represent what an NN has learned purely from its weights, which is what interpretability is all about. Potentially, it could reveal interpretability tools much better than the ones we use today.

Edit: tweets from one of the coauthor providing more context:

https://x.com/a1zhang/status/2117328657161322810

https://x.com/a1zhang/status/2117330975122235442

Can we call this a form of recursive improvement?

Well if it works on larger networks, including itself, it's certainly a much faster version of recursive self-improvement than proposing a new architecture and spending 500 million dollars on GPU training.