TL;DR
Get everyday essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Four researchers affiliated with MIT CSAIL report that GPT-6 Astra edited 0.016% of Qwen3.5-0.8B’s weights without gradient descent. The model’s MATH-500 score rose from 7.9% to 48.1%, while answers hitting the token limit fell from 93.0% to 33.1%. The result is from an early, non-peer-reviewed blog post, and the loops were reduced rather than eliminated.
Four researchers affiliated with MIT CSAIL report that OpenAI’s GPT-6 Astra edited a small portion of the open model Qwen3.5-0.8B’s weights, reducing repetitive answers on math tasks and raising its MATH-500 score from 7.9% to 48.1%. The November 2 blog post describes an iterative search without gradient descent; it says the edits reduced, but did not eliminate, the model’s tendency to run into its token limit.
The researchers say they traced the looping behavior to a direction associated with the MLPs in layers 15 and 19. GPT-6 Astra subtracted that direction from 64 output columns in each layer, changing 131,072 weights, or 0.016% of the model’s 0.8 billion weights. The edited Hugging Face model and published hashes allow the change to be checked against the original, according to the source account.
On the MATH-500 benchmark, the edited model scored 48.1%, compared with 7.9% before the edit. Answers that reached the 32,768-token limit fell from 93.0% to 33.1%. Those figures indicate a substantial improvement in this test, while also showing that about one-third of answers still hit the cap.
The work was not a one-shot weight edit. Published logs describe 158 turns, 80 Python runs and 50 forward passes, with the process taking 20.3 hours. Each forward pass could evaluate up to 512 problems. The researchers barred gradient descent and kept the final benchmark questions from Astra, while giving it practice problems from GSM8K and MATH. The post says lower budgets of 10 and 25 passes produced MATH-500 scores of 12.3% and 14.1%.
Astra edited model weights to reduce looping
Researchers report that GPT-6 Astra changed a small share of Qwen3.5-0.8B’s weights without gradient descent. The model scored higher on MATH-500, but many answers still hit the token limit.
What changed inside the model
The researchers linked repetitive answers to a direction in two layers, then removed it from selected weights.
Compare looping and non-looping outputs to identify a direction associated with repetition.
Subtract that direction from 64 output columns in layers 15 and 19, changing 131,072 weights.
Use iterative testing without gradient descent; published logs describe 50 forward passes.
Measured results
On MATH-500, fewer answers ran into the token cap, and the benchmark score rose.
- MATH-500 score: 7.9%
- 93.0% of answers hit the token limit
- MATH-500 score: 48.1%
- 33.1% of answers still hit the limit
What the evidence supports
A targeted repair
The reported result suggests a small weight edit can reduce a specific failure pattern in this model and test.
Limited scope
The account is an early, non-peer-reviewed blog post about one model and one headline benchmark.
Effects beyond the test
The report does not establish performance on other tasks or whether the edit changes other behaviors.
A Narrow Repair With Measurable Gains
The result suggests that an AI system can help inspect and alter an open model’s internal weights, potentially offering a way to address specific failure patterns without retraining all of its parameters. For teams running open-weight models, such targeted changes could be useful if they can be reproduced and validated on their own tasks.
The measured change concerns a particular model and benchmark. The researchers’ own account cautions that fixing loops is not the same as making a model better at math. The higher score followed a reduction in a behavior that had prevented many answers from finishing; it does not establish a general increase in mathematical ability or performance across other tasks.
Any patch also creates a modified model whose behavior needs evaluation. The report says the same broad class of direction-based edits has been used to remove refusal behavior in open chat models. That makes the method relevant to model maintenance and control, while also underscoring that small weight changes may have consequences beyond the immediate test.
Popular right now
De'Longhi Magnifica Duo Fully Automatic Espresso Machine with Milk Frother, Silver
Breville Barista Express Espresso Machine, Black Sesame
KitchenAid 5.5-Qt Bowl-Lift Stand Mixer, Porcelain White (KSM55SXXX)
As an affiliate, we earn on qualifying purchases.
What the Experiment Tested
The work appears in a post titled Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights, published November 2 by Rikiya Takehi. The four listed authors—Takehi, Jayden Teoh, Alex Zhang and Omar Khattab—are affiliated with MIT CSAIL; Takehi and Teoh are identified as equal contributors. The post presents eight experiments, including tests of whether Astra could infer a model’s training behavior from weights and repair a failing driving policy. One identification task failed, according to the authors.
The post is not a peer-reviewed paper and was not on arXiv, according to the source account. Its authors characterize the findings as “very early results” and say the capabilities remain limited. OpenAI is not listed as an author, and the account says it found no OpenAI statement about the experiment. The researchers ran Astra through the Codex command-line tool with maximum reasoning effort, using their own CPU-based harness and no internet access.
Qwen3.5-0.8B’s model card already warns that the model is prone to thinking loops, the source account says. In the researchers’ initial practice sample, 85 of 96 answers reached the 32,768-token cap. Their method compared looping and non-looping outputs to identify a direction linked to repetition, then removed it from selected weight matrices. They cite earlier work on removing refusal behavior through a similar kind of weight-space edit.
““Fixing loops, however, is not the same as making a model better at math.””
— The researchers, in their blog post
How Broad the Repair Is
The report does not establish whether the edit improves performance on other math benchmarks, other types of reasoning or real-world use. It also does not show whether the reduced looping persists across different prompts or after further model changes. The published result concerns one small Qwen model and one headline benchmark.
Although the post provides logs and the edited model is available on Hugging Face, the account describes the research as non-peer-reviewed and early-stage. Independent reproduction and broader evaluation would help clarify how reliably the method identifies a behavior-linked direction, and whether the change causes unwanted effects elsewhere. The reported 33.1% token-limit rate also means looping remained a material issue in the tested set.
Reproduction and Wider Testing
The next useful step is independent replication of the weight edit and its MATH-500 results, followed by checks on additional benchmarks and tasks. Evaluations could also test whether the change affects other model behaviors, how it performs under different prompts, and whether similar edits work on models beyond Qwen3.5-0.8B.
The source material does not give a schedule for a peer-reviewed paper or further experiments. For now, readers can inspect the researchers’ published logs and the edited model, while treating the reported gains as a result from an early research post rather than a settled finding.
Key Questions
What did GPT-6 Astra change?
According to the researchers, Astra identified a direction linked to looping in layers 15 and 19 of Qwen3.5-0.8B and subtracted it from selected weight columns, changing 131,072 weights.
Did the model stop looping entirely?
No. The share of MATH-500 answers reaching the 32,768-token limit fell from 93.0% to 33.1%, so a substantial portion still hit the cap.
Was this done without any training or testing?
The researchers say they did not use gradient descent, but the process involved iterative search and test feedback. Their logs describe 158 turns, 80 Python runs and 50 forward passes.
How strong is the evidence?
The reported figures and logs appear in a blog post by researchers affiliated with MIT CSAIL. The post calls the results early-stage; according to the source account, it was not peer-reviewed or posted on arXiv.
Does the result show that Astra made Qwen better at math generally?
It shows a higher score on the reported MATH-500 benchmark after the looping behavior was reduced. The authors caution that fixing loops is not the same as establishing a general improvement in mathematical ability.
Source: Thorsten Meyer AI
