Weight-Space Programming Doesn't Need Transformers

LLM writes plain-text facts→Compiles a sparse table→Model reads a few rows

Transformers were chosen because they train well with gradient descent and run fast on GPUs. In weight-space programming, a large language model (LLM) writes target weights by hand without gradient-based training, so those reasons no longer apply. When left free to design with ten labeled science questions and a sandbox, three frontier LLMs all compile plain-text facts into a sparse table read by a fixed procedure.

Hyunjong Ok1   Jaeho Lee1

1POSTECH

Overview figure contrasting dense weights from gradient descent with sparse table rows from weight-space programming at top, alongside plots of ARC accuracy by model size at bottom left and by GPU speed at bottom right.
The top panel contrasts gradient descent on dense weights with weight-space programming on sparse table rows, and the bottom panels plot ARC accuracy against size on the left and GPU speed on the right, with numbers next to stars giving the generation and the dotted line marking chance. Our associative-memory-based model is about 680 times smaller than the most accurate released architecture that the same LLM wrote and answers up to 17 times faster than a same-sized Transformer.

01 · What the LLMs build

All three LLMs converge on sparse concept tables

Given only ten labeled ARC science questions and an execution sandbox, Claude Opus 5.5, GPT-6.1 Sol, and GPT-6 Astra all build the same kind of model. Each LLM writes plain-text lines of science facts and a build script that groups related words into concepts and compiles each line into a row of a sparse concept table.

At test time, a fixed lookup procedure matches the question and each candidate option against the table, scores each option from its best-matching few rows, and adds hand-written rules. Each LLM refines its model in the sandbox by writing practice questions, inspecting errors, and adding rows or rules, while the three models differ mainly in how a written row is split.

Claude Opus 5.5 wrote these five lines in its first-generation model. The build step files each line into a schematic concept table, and "Which adaptation helps a bird fly?" lights rows sharing its concepts.
Side-by-side architecture diagrams of the first-generation models designed by Claude Opus 5.5, GPT-6.1 Sol, and GPT-6 Astra, each quoting one written fact line and separating colored weight blocks from white fixed operations.
Read each panel from bottom to top to trace the first-generation model designed by Claude Opus 5.5, GPT-6.1 Sol, and GPT-6 Astra, along with one line that each LLM wrote. Colored blocks hold weights written by the LLM, white blocks mark fixed operations, and the three designs differ mainly in how a written row is split.

02 · Why a table

Sparse tables keep answers traceable and edits local

A sparse table lets the LLM state what each weight means, trace a wrong answer to its row, and rewrite that row without changing unrelated answers. In Claude Opus 5.5's models, keeping only the best row per option nearly matches the full model, and removing one written line changes 0.5 to 1.8 answers on average, compared with 4.8 and many more in written Transformers.

Most written two-layer Transformers, including all six from Claude Opus 5.5, score below plain BM25 keyword search over the text that the LLM wrote for them, whereas most table models score above it. A written Transformer stores many memories under dense overlapping keys, and this result is consistent with interference.

On the same question, Claude Opus 5.5's first-generation LLM-designed model maps words to concepts and finds each option's best fact. After the rules, option D (hollow bones) scores 55.9, while others score 20.1 to 26.8.
In our first-generation associative-memory-based model, the same question's four strongest concepts form six pairs. At pair-table row 358,377, (adaptation, fly) recalls hollow, bones, feathers, light, and body; with the concept table, option D scores 52.1.
Step-by-step trace of the ARC test question about which adaptation helps a bird fly through an LLM-designed table model and our associative-memory-based model, highlighting question concepts in blue and option concepts in orange.
The panels follow the unseen ARC test question "Which adaptation helps a bird fly?" through Claude Opus 5.5's LLM-designed model and our associative-memory-based model, marking question concepts in blue and candidate option concepts in orange. Both traces show how the winning score for "hollow bones" comes directly from named table entries that the LLM wrote.

03 · Our associative-memory-based model

Reading computed addresses makes written tables fast on GPUs

LLM-designed models score every stored fact for every question, so work per question grows as the writer adds facts. They also launch from two to nearly three hundred GPU kernels per question across multiple scoring passes. Across 46 LLM-designed variants timed on an RTX 3090, only one runs faster than a same-sized two-layer Transformer, and the median is 9.4 times slower.

Our associative-memory-based model keeps line-by-line writing while fixing the forward pass. A build script folds written lines into a concept table and a pair table read at computed addresses from pairs of question concepts. Each question reads 120 pair-table rows and computes one concept-table product, so GPU time at a fixed concept count does not depend on how many values the pair table holds.

Architecture comparison from left to right of the given two-layer Transformer, an LLM-designed sparse table model, and our associative-memory-based model, with colored blocks for written weights and white blocks for fixed operations.
From left to right, the panels compare the given two-layer Transformer, an LLM-designed model, and our associative-memory-based model. Colored blocks hold weights written by the LLM, and white blocks mark fixed operations.
Two scatter plots of GPU inference time on 3,530 ARC test questions for LLM-designed models and same-sized Transformers against non-zero weight count on the left and GPU kernel launches per question on the right.
Each point times one model on all 3,530 ARC test questions on an RTX 3090, plotting best GPU time against non-zero values on the left and eager GPU time against kernel launches per question on the right. Colored circles show LLM-designed models and gray circles show same-sized two-layer Transformers, which run faster than almost every LLM-designed variant.

04 · Results

Sparse table models score higher across all three LLMs

WriterWritten released architecturesWritten two-layer TransformerLLM-designed (first / last)Ours (first / last)
Claude Opus 5.543.136.150.8 / 54.449.6 / 52.4
GPT-6.1 Sol35.036.344.6 / 47.441.2 / 45.6
GPT-6 Astra34.432.642.7 / 46.242.5 / 47.4

Values report ARC test accuracy (%) on 3,530 questions, where chance accuracy is 25%. The last two columns give the first and last generation of each lineage in row order, with last generations 5, 7, and 5 for LLM-designed models and 5, 5, and 8 for ours. Gradient-descent and fine-tuned baselines are in the paper.

05 · Beyond ARC

Lookup tables extend to text tasks within clear limits

Starting from a released pretrained checkpoint instead of random initialization changes how the LLMs edit weights. In most runs across the three LLMs, the writer changes under 1% of the weights to calibrate answer readout or adjust science-word embeddings, usually keeping the checkpoint's accuracy. Full overwrites with a hand-written model happen mostly on checkpoints near chance and never above 72% accuracy.

Run unchanged on other four-option tests, Claude Opus 5.5's ARC table models reach up to 50.1% accuracy on SciQ science questions while staying near 25% chance on MMLU. When asked to produce text answers on SQuAD 1.1 span extraction and SciQ word-by-word answer generation, the LLM still builds written lookup tables with a fixed reader, scoring above written Transformers on both tasks.

Current weight-space programming has clear limits, and pretrained LLMs remain far more accurate overall. Written models recall only what the LLM states and learn no representations of their own. On CIFAR-10 raw image pixels, written models reach only 21% to 28% accuracy and stay far below gradient descent, because a writer must name what each weight means.

Team

Authors

Hyunjong OkPOSTECH
Jaeho LeePOSTECH

The paper and BibTeX entry will appear here with the arXiv release.