Qwen Weight Edit Reproduces Big Gains, but Settings Narrow the Gap

• An unaffiliated maintainer reports reproducing the large gains with Qwen’s thinking mode enabled.
• The edited model still beats the original with thinking disabled, but by 4–11 percentage points across three tests.
• A laptop model identifies three game-playing networks from their weights, a separate task from repairing a language model.

An independent benchmark has reproduced MIT researchers’ large improvements from directly editing a small Qwen model’s weights. But switching off the original model’s extended reasoning mode recovers much of the apparent deficit: on MATH-500, the edited model’s advantage over that stronger baseline is about four percentage points.

The finding supports a useful repair of repetitive output, a limitation the researchers themselves explicitly acknowledge. It also makes the practical question sharper: when is investigating and patching model weights worth more than changing how the model generates answers?

JonathanColetti’s independent benchmark compares four configurations. Thinking-enabled runs allowed 32k tokens; disabled runs allowed 8k and used different sampling settings. Source: Hugging Face.

The baseline changes the size of the gain

Researchers at the Massachusetts Institute of Technology’s Computer Science and Artificial Intelligence Laboratory described “weight-space programming” in a November 2 research post. They used OpenAI’s GPT-6 Astra to inspect neural-network weights—the numerical parameters governing a model’s behavior—and devise changes without gradient descent or a conventional training loop.

Their main language-model target was Qwen3.5-0.8B, which the authors describe as having 0.8 billion weights. It is a small member of Alibaba’s Qwen3.5 model series, introduced in February with a much larger multimodal model. The small checkpoint’s accessible weights make this intervention possible; the independent checkpoint’s model card confirms its Apache 2.0 license. The researchers chose a target whose own model card already warned about repetitive reasoning loops. “Thinking” mode allows extended reasoning before the final answer; here, that process often repeated text until the output budget ran out.

With thinking enabled, the MIT team reported that Astra’s edit raised MATH-500 accuracy from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, and instruction-following performance on IFEval from 21.0% to 62.3%.

A checkpoint maintainer publishing as JonathanColetti said they were unaffiliated with the authors, OpenAI or Qwen, applied the released edit, and verified that the resulting checkpoint matched the authors’ evaluated version hash for hash. The maintainer published an independent benchmark. All three thinking-enabled results reproduced within about two percentage points. The same tests also exposed how much the comparison depends on generation settings.

The maintainer used four samples for each of 500 MATH-500 problems, two for each of 1,319 GSM8K problems, and two for each of 541 IFEval prompts. IFEval used strict prompt-level scoring with official checkers. Thinking-enabled runs allowed 32k tokens; disabled runs allowed 8k and used different sampling parameters. These are comparisons between deployment configurations, rather than a single isolated setting change.

The edited model with thinking enabled still beat the original model’s best tested setting on all three benchmarks. But the reported paired MATH-500 advantage was 4.3 percentage points, with a 95% confidence interval of 1.9 to 6.8 points—far smaller than the 42.8-point gain when both versions used thinking. The corresponding advantages were 10.5 points on GSM8K and 9.2 points on IFEval. With thinking disabled in both, the edit reduced scores across all three tests; the maintainer found that reduction statistically significant only on MATH-500.

A targeted repair, with a failed capability test

The mechanism was already explicit in the authors’ account. Astra traced repetitive output to a direction written by components in layers 15 and 19, then subtracted that direction from 64 output columns in each. The final patch changed 131,072 weights, about 0.016% of the model.

In the authors’ MATH-500 evaluation, the share of answers exhausting the 32,768-token limit fell from 93.0% to 33.1%. The independent benchmark likewise found that the MATH-500 token-limit rate dropped from 93% to 32% with thinking enabled. Its maintainer attributed almost all of the improvement to answers that stopped looping.

The authors also tried editing Gemma 3 270M, an instruction-tuned language model about a third Qwen’s size. Only 8.5% of its answers reached the token limit in their setup. Astra found ordinary arithmetic and algebra errors, but established no improvement: its MATH-500 score was 6.8% before editing and 7.0% after the attempt with a 50-forward-pass budget. That result limits the case for treating loop repair as a general method for creating new reasoning skills.

The broader research goes beyond loop repair

The authors also report constructing small game-playing networks from scratch, using game rules and source code without trial runs or tools. Those networks beat reinforcement-learning baselines trained with proximal policy optimization (PPO) for 100 million steps. Against 500-million-step baselines, Astra won decisively in Breakout but trailed in Seaquest and Asterix. The Breakout result therefore does not describe a uniform advantage across games.

A larger test used a 1.48-million-weight network to play Go on a 9×9 board. Astra wrote weights through Python; with no practice executions, its network won more than half its games against two smaller PPO networks. With a budget of 50 execution requests, it beat an equal-sized PPO network trained on roughly 20 million games. Each request could include up to 128 practice games, so “50” does not mean 50 individual games. These results show strategy construction in bounded environments, beyond merely repairing Qwen.

The boundaries are equally relevant. Astra could not identify which of two language models had been fine-tuned to agree with users’ suggestions from weights alone. It succeeded after receiving attention measurements from model executions. Direct access to parameters did not consistently reveal behavior without observing the network at work.

Takehi and colleagues’ original Breakout identification example. Willison’s laptop experiment tested this weight-reading task rather than the language-model repair. Source: Rikiya Takehi and colleagues.

What the laptop test actually repeats

On November 3, Simon Willison, who develops the LLM command-line tool he used for testing, published a local replication of the game-identification task. He supplied the researchers’ prompts, including raw weights, to Qwen 3.8 27B, a different, larger Qwen model running on his laptop.

This was a deliberately small, structured task: the original four game networks each had 17,253 weights and the same architecture. Prompts described the input channels and available actions, while withholding game names. Weight patterns could reveal where a paddle or maze lay. The test was therefore interpretation with architectural and input context, rather than decoding an unexplained list of numbers.

The model identified Pac-Man, Pong and Breakout, but mistook Freeway for a Pac-Man-like game. Willison used a 17GB model version and enlarged the context window after the default failed. The successful runs took 46 minutes 12 seconds, 1 hour 4 minutes 54 seconds, and 52 minutes 25 seconds, respectively.

This shows that some weight reading works locally. It does not reproduce the Qwen loop repair or establish parity with Astra: the reported task was identifying games, and the local model missed one.

Takehi and colleagues’ reported results by execution budget. Each request could batch many examples; these counts do not measure total cost. Source: Rikiya Takehi and colleagues.

The next comparison is cost and reliability

Finding the small Qwen patch required substantial investigation. The team’s experiment documentation, as described in a report on the work, puts the strongest attempt at 158 conversational turns, 80 Python runs and 20.3 hours, including 4.7 hours waiting through a service outage. Astra could select practice problems from training sets and request batches of model executions; the benchmark test sets were withheld. “No training” did not mean no data or evaluation computation.

The researchers’ smaller budgets of 10 and 25 execution requests produced MATH-500 scores of 12.3% and 14.1%, respectively, against 48.1% with 50 requests. The full gain therefore did not emerge at the lower execution budgets. The Qwen setup separated a Python kernel without a GPU from the tool that executed the target model; the authors identify context length and harness inefficiencies as continuing constraints.

Direct weight editing also has precedents. The report describes ROME’s targeted factual edits, introduced in 2022, and MEMIT’s extension to thousands of associations in 2023. The proposed distinction here is a general-purpose LLM investigating weights and devising changes across tasks. The practical comparison must therefore include specialized editors, conventional fine-tuning and simpler generation-setting changes.

Open weights allow developers to apply and inspect a patch themselves; customers with API-only access depend on their provider to permit or perform it. But access does not settle the economics. Neither the research nor the independent benchmark establishes a cost advantage over those alternatives, and the laptop runs provide elapsed times rather than a comparable total-cost measure.

The team has released experiment logs and says it is building a benchmark and supporting harness. The independent maintainer also disclosed that missing internal evaluation dependencies required a stand-in for loop diagnostics and a separate MATH-500 script matching the authors’ settings. The next useful evidence is a reproducible comparison of repair success, regressions, output-token use and total discovery cost across more models—including ones whose failures are not dominated by loops.