Executive summary

I just read a paper from the MIT CSAIL team (Takehi, Teoh, Zhang, Khattab, released November 2, 2026) claiming that frontier LLMs like GPT-6 "Astra" can directly read and write the weights of other neural networks, without gradient descent or a training loop. They call this "Weight-Space Programming," and what made me pause the most is that Astra can fix a reasoning bug in Qwen3.5-0.8B by changing just 0.016% of its weights.

The problem being solved

Until now, if you wanted to optimize something that isn't text (for example, neural network weights), you were stuck with symbolic optimizers like Adam or Muon, because humans can't "tinker" with weights manually the way they tinker with code. LLMs have long been used to optimize text artifacts (prompts, hyperparameters, training code), but raw float weights were considered territory that couldn't be touched through natural language reasoning. This paper shows that assumption starting to wobble: Astra can guess what game a PPO policy learned, what shape a diffusion model draws, even what language a small language model learned — all just from a sequence of weight numbers, without running a single forward pass.

Key features and code example

There are three capabilities they tested:

  1. Reading weights — given the raw weights of a 500→32→32→5 MLP trained with PPO for one of four grid games (Breakout, Pong, Pac-Man, Freeway), Astra guessed the correct game for all four, without ever being told the list of options. Even when the input/output layers were removed and Astra was only given the 4 middle transformer blocks of a 7.2M-parameter character-level language model, it could still guess the language (Python code, Japanese romaji, or English) and identify all the "copying heads" (induction heads) among the 24 attention heads present.
  1. Writing weights from scratch — for three MinAtar games (Breakout, Seaquest, Asterix), Astra wrote the weights of a 600→32→32→6 network directly as text, without data, loss, gradients, or any tool, and the resulting network outperformed PPO trained for 100 million steps on all three games. For 9×9 board Go (a 243→512×6→82 network, 1,480,274 weights, all starting from scratch), with a budget of just 0 forward passes the network already won more than half its matches against a small PPO; with 50 forward passes, it beat an equivalent PPO trained on 20 million games.
  1. Fixing broken weights — this is the one most relevant to everyday work. Qwen3.5-0.8B (thinking on, 24 layers, 0.8 billion weights) has a reasoning loop problem that's already flagged in its own model card (85 of 96 initial answers hit the 32,768 token limit). Astra traced this loop to a single direction in MLP layers 15 and 19, then subtracted that direction from 64 output columns in each layer — a total of only 131,072 out of 0.8 billion weights (0.016%). The result: MATH-500 rose from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, and IFEval (instruction following, not math) from 21.0% to 62.3% — even though this fix wasn't targeted at IFEval at all. The proportion of answers that hit the loop dropped from 93% to 33.1%, with a budget of 50 forward passes for exploration.

Important to note: this trick failed completely on Gemma 3 270M, which rarely loops (only 8.5% of its answers hit the limit). Astra only found ordinary arithmetic errors with no single component that could be switched off, and the edit itself was acknowledged to be insignificant (<1 point on MATH-500).

The source doesn't give an actual API or code for this — it's all done through natural language prompts to Astra plus a Python sandbox for limited forward passes. So I made a simple illustration (not from the paper) to show what the interaction looks like:

# ILUSTRASI, bukan kode dari paper — hanya menggambarkan alur kerjanya
import numpy as np

def load_weights(path):
    return np.load(path, allow_pickle=True).item()

weights = load_weights("qwen3.5-0.8b.npz")

prompt = f"""
Ini adalah bobot mentah model bahasa 0.8B parameter.
Model ini sering terjebak reasoning loop pada soal matematika.
Tanpa gradient descent atau training, identifikasi arah di
MLP layer 15 dan 19 yang menyebabkan loop, lalu kembalikan
delta bobot (sebagai array numpy) untuk 64 kolom output tiap layer.

Bobot layer 15 (MLP.down_proj): {weights['layers.15.mlp.down_proj']}
Bobot layer 19 (MLP.down_proj): {weights['layers.19.mlp.down_proj']}
"""

# delta_15, delta_19 = astra_edit_weights(prompt)  # panggilan konseptual ke LLM
# weights['layers.15.mlp.down_proj'] -= delta_15
# weights['layers.19.mlp.down_proj'] -= delta_19

Impact and upgrade notes

This is still early-stage research ("sparks of," as they themselves put it), not a product or an installable library — so don't expect a pip package tomorrow. No training loop was allowed in any of these experiments, only reasoning plus (in later experiments) a limited number of forward passes through a Python sandbox. What I think is most actionable for teams managing small models like Qwen3.5-0.8B: if your model also frequently suffers from reasoning loops like the ones flagged in Qwen's model card, targeting a fix like this (a single direction in a few MLP layers) could be a much cheaper debugging alternative than full retraining — but the results don't automatically generalize to every kind of math problem (see the failure on Gemma 3 270M). For those interested in interpretability, the attention-pattern experiment for detecting sycophantic models (two Gemma 3 270M fine-tunes) is also interesting: Astra failed completely using raw weights alone, but succeeded on 3 of 3 pairs once given access to the attention patterns.

---

First announced on the Hacker News front page. Read the original source for full details.

Read also