aecetin / PolyFormer-Tiny

⚡ PolyFormer-Tiny: Orthogonal Chebyshev Polynomial LM (Trained From Scratch)

PolyFormer-Tiny is a native, non-linear neural language model architecture trained completely from scratch without standard multi-layer perceptrons (MLP / SwiGLU).

Instead of conventional matrix projections with static activation functions, PolyFormer replaces all feed-forward network (FFN) blocks with orthogonal Chebyshev polynomial tensor operators and attention heads with idempotent polynomial kernels on an idempotent manifold (Π² = Π).

• Author & Inventor: Dr. A. Emre ÇETİN ( [email protected] )
• Official Patent: Protected under USPTO Application No. 64/149,540 ( Hardware-Accelerated Orthogonal Polynomial Tensor Operators, Zero-Backpropagation Closed-Form Algebraic Solvers, and In-Situ Weight Surgery for Deep Neural Networks and Transformers ).
• Affiliation: Computational Systems and Cognitive Architectures, Izmir, Turkey
• Core Library: idempotent-poly ( pip install idempotent-poly )
• Research Paper: Section IV & V: PolyFormer Architecture & Benchmark - ResearchGate Publication 414060833
• Sister Model (In-Situ Surgery): aecetin/SmolLM2-135M-PolyFFN

---

🔬 Official Verified Benchmarks: Inference & Architecture

Direct side-by-side benchmark evaluated against an architecturally identical Standard Transformer baseline:

---

⚖️ Semantic & Output Alignment (Standard vs PolyFormer)

To rigorously verify that halving parameters does not degrade representational fidelity, both models were evaluated under identical prompt conditions:

1. Vector Alignment (Next-Token Cosine Similarity)

2. Convergence Summary

3. 📈 Compute-Optimal Scaling: Catching Up to Double-Sized Baselines

A central question in compact non-linear design is whether parameter-reduced models hit an insurmountable capacity wall. To empirically verify this, we tracked PolyFormer's learning curve across extended optimization horizons:

Key Theoretical Takeaway: Because orthogonal Chebyshev tensor contractions span a richer non-linear basis than static linear GEMMs, modest compute extension allows a 46.2% smaller model to fully exceed double-sized baseline capacities while permanently locking in 50% inference memory and VRAM savings .

---

📐 Mathematical Formulation

1. Orthogonal Chebyshev Basis Recursion

Input representations are normalized onto the Chebyshev domain $[-1, 1]$:

$\tilde{x}=\tanh \left(x\right)\in [-1,1]\backslash tilde\{x\}\; =\; \backslash tanh(x)\; \backslash in\; [-1,\; 1]$ x ~ = tanh ( x ) ∈ [ − 1 , 1 ]

and projected onto orthogonal Chebyshev polynomials of the first kind:

$T_0\left(x\right)=1,T_1\left(x\right)=x,T_{k+1}\left(x\right)=2x{T}_k\left(x\right)-T_{k-1}\left(x\right)T\_0(x)\; =\; 1,\; \backslash quad\; T\_1(x)\; =\; x,\; \backslash quad\; T\_\{k+1\}(x)\; =\; 2x\; T\_k(x)\; -\; T\_\{k-1\}(x)$ T 0 ​ ( x ) = 1 , T 1 ​ ( x ) = x , T k + 1 ​ ( x ) = 2 x T k ​ ( x ) − T k − 1 ​ ( x )

2. Chebyshev Tensor FFN

Instead of 3 separate high-rank GEMM matrices (Gate, Up, Down projections), the feed-forward mapping is computed directly via 3D tensor contraction:

$\text{PolyFFN}\left(x\right)=\underset{k=0}{\overset{K}{\sum }}{C}_k\cdot T_k\left(\tilde{x}\right)+b\backslash text\{PolyFFN\}(x)\; =\; \backslash sum\_\{k=0\}^K\; C\_k\; \backslash cdot\; T\_k(\backslash tilde\{x\})\; +\; b$ PolyFFN ( x ) = k = 0 ∑ K ​ C k ​ ⋅ T k ​ ( x ~ ) + b

where the orthogonal coefficient tensor is parameterized as:

$C\in R^{(K+1)\times d_{\text{in}}\times d_{\text{out}}}C\; \backslash in\; \backslash mathbb\{R\}^\{(K+1)\; \backslash times\; d\_\{\backslash text\{in\}\}\; \backslash times\; d\_\{\backslash text\{out\}\}\}$ C ∈ R ( K + 1 ) × d in ​ × d out ​

3. Idempotent Chebyshev Attention Kernel

Attention scores pass through an orthogonal polynomial kernel filter:

$S=\frac{Q{K}^T}{\sqrt{d_k}}S\; =\; \backslash frac\{QK^T\}\{\backslash sqrt\{d\_k\}\}$ S = d k ​ ​ Q K T ​

$\text{Attn}(Q,K,V)=\text{softmax}\left(\underset{k=0}{\overset{K}{\sum }}{w}_k{T}_k(\tanh (S\left)\right)\right)V\backslash text\{Attn\}(Q,\; K,\; V)\; =\; \backslash text\{softmax\}\backslash left(\; \backslash sum\_\{k=0\}^K\; w\_k\; T\_k\backslash left(\backslash tanh(S)\backslash right)\; \backslash right)\; V$ Attn ( Q , K , V ) = softmax ( k = 0 ∑ K ​ w k ​ T k ​ ( tanh ( S ) ) ) V

---

💻 Quickstart & Inference

Using Transformers ( AutoModelForCausalLM )

import torch
from transformers import AutoConfig, AutoModelForCausalLM

# Load model directly from Hugging Face Hub
repo_id = "aecetin/PolyFormer-Tiny"
config = AutoConfig.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, config=config, trust_remote_code=True)

# Encode prompt using byte-level tokens
prompt = "Mathematics, artificial intelligence, and geometric deep learning"
input_ids = torch.tensor([list(prompt.encode("utf-8"))], dtype=torch.long)

# Autoregressive generation
with torch.no_grad():
    output_tokens = model.generate_tokens(input_ids, max_new_tokens=48, temperature=0.7)

generated_text = bytes(output_tokens[0].tolist()).decode("utf-8", errors="ignore")
print(generated_text)

Using Core Library ( idempotent-poly )

pip install idempotent-poly

import torch
from idempotent_poly.polyformer import PolyFormerBlock

# Instantiate native PolyFormer block
block = PolyFormerBlock(d_model=128, n_heads=4, poly_degree=3)

# Test adaptive fixed-point early halting
x = torch.randn(1, 16, 128)
output, steps_taken = block.forward_adaptive(x, max_iters=5, tol=0.02)
print(f"Fixed-point reached in {steps_taken} adaptive iteration steps!")

---

📜 Citation & Intellectual Property

This work and its underlying mathematical architectures are protected under United States Patent Law:

@patent{cetin2026orthogonalpoly,
  title={Hardware-Accelerated Orthogonal Polynomial Tensor Operators, Zero-Backpropagation Closed-Form Algebraic Solvers, and In-Situ Weight Surgery for Deep Neural Networks and Transformers},
  author={Dr. Ahmet Emre {\c{C}}etin},
  year={2026},
  month={September},
  note={U.S. Provisional Patent Application No. 64/149,540, Filed at USPTO}
}