GRT-30M-Embedding: Planar-Pair Circular Sentence Encoder

An embedding-only adaptation of GRT-30M (GRTModel + SentenceTransformer) that pools 128 planar (x, y) coordinate pairs by averaging unit phase directions and pair radii separately (pooling_mode="circular"), contrastively fine-tuned on AllNLI and STS-B using fused SYCL kernels on Intel Arc XPU.

1. Why Planar-Pair Circular Pooling?

Standard Cartesian mean pooling averages raw coordinates across tokens, which causes destructive phase cancellation when representations live on 2D rotational subspaces. Because every hidden state in GRT is structured as D / 2 = 128 planar (x, y) coordinate pairs transformed by SO(D) Givens cascades and Cl(2,0) bivector coupling, circular pooling (1_Pooling/config.json: "pooling_mode": "circular") decomposes each token's pair into its unit direction and radius before pooling across non-padded tokens:

# Planar-pair circular pooling (SentenceTransformers Pooling(pooling_mode="circular"))
pairs = token_embeddings.unflatten(-1, (-1, 2))          # (B, T, D/2, 2)
radii = pairs.norm(dim=-1, keepdim=True)                 # (B, T, D/2, 1)
directions = pairs / radii.clamp(min=1e-9)               # unit (cos theta, sin theta)

direction_mean = (directions * mask).sum(dim=1) / count  # circular mean direction
radius_mean    = (radii * mask).sum(dim=1) / count       # mean pair magnitude
sentence_emb   = F.normalize((direction_mean * radius_mean).flatten(1), p=2, dim=-1)

2. Two-Stage Contrastive Fine-Tuning Method (Intel Arc XPU)

Starting from the pretrained MatterWaveAI/grt-30m causal backbone (with the scalar LM head output_scale removed to form a pure GRTModel encoder), we froze the 151,669 lexical phase_angles (19.41M parameters) to preserve the pretrained Qwen 3 polar token geometry and trained only the 9.74M rotational backbone parameters plus 151,669 token log_magnitude weights (9,891,865 trainable parameters) in FP32 on an Intel Arc 140T GPU (xpu:0, 32 GB) using PairAdam and the fused oneAPI DPC++ SYCL Givens and Clifford kernels:

3. Semantic Textual Similarity (STS-B) Benchmarks

Evaluated on the held-out sentence-transformers/stsb validation (n = 1,500) and test (n = 1,379) splits using planar-pair circular pooling and cosine similarity:

¹ Optimizer comparison note: The published checkpoint uses PairAdam (zero weight decay and [-π, π] wrapping on angles, max_angular_step=0.05, and rank-1 factored second moments with coupling_lr_mult=3.0 on Cl(2,0) coupling tensors) for consistency with GRT-30M pretraining. In a head-to-head run on the identical 508-step schedule and seed (42), standard AdamW (coupling_lr_mult=1.0, full diagonal second moments) reaches 0.7289 Val / 0.6317 Test Spearman (+0.0110 / +0.0075), suggesting PairAdam's higher coupling LR multiplier is primarily needed for large-batch pretraining from scratch rather than short, small-batch (B=32) fine-tuning.

4. Multi-Vendor Fused Kernel Ecosystem (kernels-community)

Both MatterWaveAI/grt-30m and MatterWaveAI/grt-30m-embedding are backed by fused forward and backward C++/GPU kernels in huggingface/kernels-community across all three major accelerator stacks:

5. Quickstart (SentenceTransformer & AutoModel)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("MatterWaveAI/grt-30m-embedding")
sentences = [
    "A man is playing a guitar on stage.",
    "A musician performs with an acoustic guitar.",
    "The stock market closed higher today.",
]
embeddings = model.encode(sentences, normalize_embeddings=True)
sim_pos = float(embeddings[0] @ embeddings[1])  # 0.779
sim_neg = float(embeddings[0] @ embeddings[2])  # 0.132
print(f"Positive similarity: {sim_pos:.3f} | Negative similarity: {sim_neg:.3f}")