MODEL CARD & BENCHMARK REPORT · NOVEMBER 2026

GRT-30M: Distilled from Qwen 3 in 24 Hours

A 24-hour Qwen 3 knowledge distillation preview of the Geometric Rotation Transformer (GRTForCausalLM, grt30m_kd) with 9.74M non-embedding backbone parameters (29.31M total stored parameters with the 151,669-token Qwen 3 BPE vocabulary), verified token-for-token across PyTorch ROCm, llama.cpp CPU, Vulkan, and HIP.

HUGGING FACE REPO ↗ ARCHITECTURE NOTE →
BACKBONE PARAMS
9.74M
29.31M total stored (55.68M dense-equiv)
24H QWEN 3 DISTILLATION
10.79B
tokens at 123.9k tok/s (eight A100 GPUs)
IN-DOMAIN VAL PPL
88.02
27.2% top-1 accuracy (151k Qwen3 vocab)
STRIX HALO SPEED
5,760
tok/s BF16 HIP prefill · 87.4 tok/s CPU decode

1. Architecture & Parameter Budget

GRT replaces every nn.Linear weight matrix in attention (Q, K, V, O) and feed-forward (expand, contract) with cascades of learned 2×2 Givens rotations interleaved with fixed butterfly permutations and modulated by 4-component Clifford bivector features. Because GRT-30M pairs a compact d_model = 256 backbone with the 151,669-token Qwen3 vocabulary, 66.8% of the stored parameters live in the tied polar embedding table, leaving 9.74M stored parameters across all 12 rotational transformer blocks.

Component Configuration Stored Params Share
Tied Polar Embedding & Cosine Head 151,669 tokens (128 phase angles + 1 log-magnitude per token) + learned output scale 19,565,302 66.76%
12 Rotational Transformer Blocks d_model=256, n_heads=4, d_ff=1024, stages 11 / 1 / 21 (1,044 stages total) 9,740,196 33.24%
Total (GRT-30M) max_position_embeddings = 2048 (55,680,446 dense-equivalent parameters) 29,305,498 100.00%

2. 24-Hour Qwen 3 Distillation Summary

Distilled from Qwen 3 (grt30m_kd, sharing the 151,669-token Qwen 3 BPE vocabulary) for 24.0 wall-clock hours on a 5-corpus educational mix using PairAdam (factored second moments for 2D Clifford coupling matrices plus angular step clamping) and fused CUDA Givens kernels:

Metric Value
Teacher & Student Config Distilled from Qwen 3 into grt30m_kd (151,669-token Qwen 3 BPE vocabulary, tied polar cosine head)
Training Corpus Mix fineweb-edu (45%), cosmopedia-v2 (20%), wikipedia (15%), finemath (12%), stack-edu (8%)
Hardware & Precision Eight NVIDIA A100-SXM4-80GB GPUs (DDP, bfloat16 mixed precision, fused CUDA Givens kernels)
Tokens & Steps Trained 10,789,404,672 tokens (20,580 optimizer steps in 86,969s; exported at checkpoint-20000)
Sustained Throughput 123,900 tokens/sec (~15,490 tokens/sec per GPU, zero NaNs or gradient spikes)
Validation Loss & Perplexity val_loss = 4.4776 · val_ppl = 88.02 · top-1 token accuracy = 27.20% (151,669 vocab)
Held-Out WikiText-2 Test loss = 5.0305 · ppl = 153.01 · top-1 token accuracy = 25.71% (297,984 tokens, T = 1024)

3. 4-Way Exact Greedy Parity

The published model.safetensors (transformers) and grt-30m-f32.gguf (llama.cpp) produce identical token-for-token greedy continuations (temperature = 0, 16 new tokens) across all four execution paths on AMD Strix Halo (gfx1151): PyTorch ROCm (fused HIP kernels), llama.cpp CPU (AVX2), llama.cpp Vulkan (RADV), and llama.cpp ROCm (HIP).

Prompt Greedy Continuation (16 tokens, identical across PyTorch ROCm, CPU, Vulkan, and HIP)
The capital of France is The capital of France is the first to be the first to be the first to be the first to be
Once upon a time Once upon a time, you might be wondering what exactly is the way to do with the right point
In the beginning In the beginning of the world, the world is the most common form of the world. The

4. Zero-Shot Multiple-Choice Benchmarks

Evaluated zero-shot (no task fine-tuning) via teacher-forced length-normalized log-likelihood scoring on checkpoint-20000. With 9.74M non-embedding parameters trained for 10.8B tokens, GRT-30M exceeds random chance on 6 of 7 multiple-choice benchmarks, matching dense models (Pythia-14M / Pythia-31M) of comparable backbone size:

Benchmark Split (n) GRT-30M Random Baseline Delta vs. Random
ARC-Easy (allenai/ai2_arc) test (2,376) 29.50% 25.00% +4.50%
HellaSwag (Rowan/hellaswag) val (1,000) 29.20% 25.00% +4.20%
OpenBookQA (allenai/openbookqa) test (500) 28.40% 25.00% +3.40%
CommonsenseQA (tau/commonsense_qa) val (1,221) 21.70% 20.00% +1.70%
BoolQ (google/boolq) val (3,270) 51.53% 50.00% +1.53%
QNLI (nyu-mll/glue) val (1,000) 51.50% 50.00% +1.50%
ARC-Challenge (allenai/ai2_arc) test (1,172) 21.08% 25.00% -3.92%

5. Edge Inference & Kernel Speed (AMD Strix Halo gfx1151)

Measured on local Strix Halo nodes (AMD Ryzen AI Max+ 395 w/ Radeon 8060S Graphics) across PyTorch ROCm (with all five fused GRT HIP kernels registered: coupled_rotate, plain_rotate, mixer_stage, pearl_norm, radial_gelu) and llama.cpp (grt-30m-f32.gguf):

Runtime / Backend Precision Prefill Throughput Decode / Training Throughput
PyTorch ROCm + 5 Fused GRT HIP Kernels BF16 5,759.6 tok/s (B=4, T=512) 1,156.0 tok/s (Fwd+Bwd)
llama.cpp Vulkan (-ngl 99) FP32 3,790.1 tok/s (pp512) 49.9 tok/s (tg64)
llama.cpp CPU (-t 16, AVX2) FP32 905.4 tok/s (pp512) 87.4 tok/s (tg64)
llama.cpp ROCm (-ngl 99) FP32 195.0 tok/s (pp512) 37.5 tok/s (tg64)

6. Quickstart

import torch
from transformers import AutoTokenizer, GRTForCausalLM

model_id = "MatterWaveAI/grt-30m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = GRTForCausalLM.from_pretrained(model_id, dtype=torch.float32).eval()

ids = tokenizer("The history of the city", return_tensors="pt").input_ids
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))
↑ TOP