Sponsored by YouArt
Toolify Toolify logo
AI Models
ksimplex-llm-prototype

AbstractPhil / ksimplex-llm-prototype

Run on huggingface.co
huggingface.co
Total runs: 5
24-hour runs: 1
7-day runs: 2
30-day runs: 0
Model's Last Updated: February 05 2026

Introduction of ksimplex-llm-prototype

Model Details of ksimplex-llm-prototype

K-Simplex Language Model Prototype

A geometric autoregressive language model using Cayley-Menger validated k-simplex channels. This architecture replaces traditional transformer embeddings with geometrically-constrained structures that maintain mathematical validity throughout training.

Overview

This model explores whether geometric inductive bias can improve language modeling by representing each token position as a hierarchy of k-simplices (edge → triangle → tetrahedron → 5-cell) with learnable deformations validated by the Cayley-Menger determinant.

Key Results:

  • Shakespeare corpus: Val PPL 113.74 at epoch 8
  • 100% geometric validity maintained throughout training
  • Coherent dialogue generation with proper character attribution
  • 54M parameters (due to 50k BPE vocabulary)
Architecture
Conceptual Foundation

Traditional transformers represent tokens as flat vectors. This architecture represents each token as a stack of k-simplex structures where:

K-Level Structure Vertices Distance Pairs Geometric Meaning
k=1 Edge 2 1 1D linear relationship
k=2 Triangle 3 3 2D planar structure
k=3 Tetrahedron 4 6 3D volumetric structure
k=4 5-cell 5 10 4D hypervolume

Each k-level captures progressively higher-dimensional geometric relationships, providing a structured representation space that traditional embeddings lack.

Token Flow 
Token ID
    ↓
Embedding Layer (vocab_size × embed_dim)
    ↓
Positional Encoding
    ↓
┌─────────────────────────────────────────┐
│         TokenToKChannels                │
│  Projects to [B, T, K, feat_dim]        │
│  Each position gets K simplex channels  │
└─────────────────────────────────────────┘
    ↓
┌─────────────────────────────────────────┐
│         GeoBlock × num_blocks           │
│  ┌─────────────────────────────────┐    │
│  │ KChannelCrossAttention          │    │
│  │ K-levels attend to each other   │    │
│  │ (within each token position)    │    │
│  └─────────────────────────────────┘    │
│  ┌─────────────────────────────────┐    │
│  │ CausalSequenceAttention         │    │
│  │ Tokens attend causally          │    │
│  │ (across sequence, masked)       │    │
│  └─────────────────────────────────┘    │
│  ┌─────────────────────────────────┐    │
│  │ MLP                             │    │
│  └─────────────────────────────────┘    │
└─────────────────────────────────────────┘
    ↓
LM Head → Logits [B, T, vocab_size]
Geometric Formulas
Cayley-Menger Determinant

For a k-simplex with vertices $v_0, v_1, \ldots, v_k$, the squared volume is computed via:

Vol 2 = ( − 1 ) k + 1 2 k ( k ! ) 2 det ⁡ ( C M ) \text{Vol}^2 = \frac{(-1)^{k+1}}{2^k (k!)^2} \det(CM)

Where the Cayley-Menger matrix is:

C M = ( 0 1 1 ⋯ 1 1 0 d 01 2 ⋯ d 0 k 2 1 d 01 2 0 ⋯ d 1 k 2 ⋮ ⋮ ⋮ ⋱ ⋮ 1 d 0 k 2 d 1 k 2 ⋯ 0 ) CM = \begin{pmatrix} 0 & 1 & 1 & \cdots & 1 \\ 1 & 0 & d_{01}^2 & \cdots & d_{0k}^2 \\ 1 & d_{01}^2 & 0 & \cdots & d_{1k}^2 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & d_{0k}^2 & d_{1k}^2 & \cdots & 0 \end{pmatrix}

Validity Criterion: $\text{Vol}^2 > 0$ indicates a non-degenerate simplex.

Template Deformation

Each k-simplex starts from a regular (equilateral) template and learns deformations:

v i ( deformed ) = v i ( template ) + α ⋅ Δ v i v_i^{(\text{deformed})} = v_i^{(\text{template})} + \alpha \cdot \Delta v_i

Where:

  • $v_i^{(\text{template})}$ = vertices of regular k-simplex
  • $\alpha$ = deformation scale (BASE_DEFORM = 0.05)
  • $\Delta v_i$ = learned offset from neural network
Geometric Gating

Features are gated by geometric validity:

output = features ⊙ gate ( geo ) ⊙ σ ( Vol 2 ⋅ 10 6 ) \text{output} = \text{features} \odot \text{gate}(\text{geo}) \odot \sigma(\text{Vol}^2 \cdot 10^6)

Where:

  • $\text{gate}(\text{geo}) = \sigma(W \cdot [d^2 | \text{Vol}^2])$
  • The sigmoid on Vol² acts as a soft validity mask
  • Invalid simplices (Vol² < 0) have their features suppressed
Loss Function

L = L C E + λ ⋅ L v a l i d i t y \mathcal{L} = \mathcal{L}_{CE} + \lambda \cdot \mathcal{L}_{validity}

Where:

  • $\mathcal{L}_{CE}$ = Cross-entropy for next-token prediction
  • $\mathcal{L}_{validity} = \text{mean}(\text{ReLU}(-\text{Vol}^2))$ penalizes collapsed simplices
  • $\lambda = 0.1$ (validity weight)
Safe Deformation Analysis

Extensive testing via the K-Simplex Geometric Explorer revealed critical stability zones:

Stability Zones by K-Depth
Configuration Differentiation Zone Collapse Threshold
k=1-4, edim=16 0.15 - 0.35 ~0.50
k=1-4, edim=32 0.15 - 0.50 >2.0
k=1-6, edim=16 0.35 - 0.45 ~0.50
k=1-6, edim=32 0.25 - 0.60 >2.0
Key Findings

  1. Deformation Scale Safety : BASE_DEFORM=0.05 is extremely conservative. The geometry can safely handle 10-40× more deformation.

  2. Embedding Dimension as Stability Buffer : 

    edim / k_max = stability_ratio

ratio ≥ 8×  → Very stable, deform up to 2.0
ratio ≥ 4×  → Comfortable margin
ratio ≥ 2×  → Tight but functional

  3. Vol² Behavior Under Deformation :

    • Low deform (0-0.15): Clear k-level hierarchy, Vol² decreases exponentially with k
    • Medium deform (0.15-0.35): Optimal zone - distinct geometric signatures per k
    • High deform (>0.5): Noise dominates, k-levels converge, geometric meaning lost

  4. Vol² Scaling : 

    k=1: Vol² ~ 1e+0 (edge length squared)
k=2: Vol² ~ 1e-1 (triangle area squared)
k=3: Vol² ~ 1e-2 (tetrahedron volume squared)
k=4: Vol² ~ 1e-3 (5-cell hypervolume squared)

    Exponential decay is expected and healthy.

Recommended Production Settings 
# Conservative (proven)
BASE_DEFORM = 0.05
edim = 16
depth = 4  # k=1,2,3,4

# Aggressive (tested safe)
BASE_DEFORM = 0.15
edim = 32
depth = 4

# Experimental
BASE_DEFORM = learnable_per_k  # Allow network to find optimal
edim = 2 * depth  # Minimum viable
Training Configuration
Model Hyperparameters 
config = {
    "vocab_size": 50257,      # GPT-2 BPE tokenizer
    "max_seq_len": 256,
    "embed_dim": 384,
    "depth": 4,               # k=1,2,3,4
    "edim": 16,               # Vertex coordinate dimension
    "feat_dim": 96,           # Features per vertex
    "hidden": 384,
    "num_heads": 8,
    "num_blocks": 8,
    "dropout": 0.1,
}
Training Hyperparameters 
training = {
    "batch_size": 48,
    "seq_len": 256,
    "lr": 3e-4,
    "weight_decay": 0.1,
    "num_epochs": 50,
    "grad_clip": 1.0,
    "ce_weight": 1.0,
    "validity_weight": 0.1,
    "scheduler": "CosineAnnealingLR",
    "stride": 128,            # Non-overlapping sequences
}
Results
Training Progression
Epoch Train PPL Val PPL Status
1 492 299 Learning
5 77 132 Improving
8 44 114 Best
15 15 145 Overfitting
Geometric Health

Throughout training:

  • Validity : 100% at all k-levels
  • Vol² k=1 : ~0.92 (stable)
  • Vol² k=2 : ~0.16 (stable)
  • Vol² k=3 : ~0.03 (stable)
  • Vol² k=4 : ~0.001 (stable)
Generation Quality

Epoch 1: 

ROMEO: , If, and a head I am IAB, What,

Epoch 15+: 

ROMEO: if thou swear'st the Duke of love of it.
MERCUTIO: Why, is it good.
ROMEO: And for the jest love that.

The model learns:

  • Character names and dialogue structure
  • Turn-taking conventions
  • Shakespearean vocabulary and cadence
  • Coherent multi-turn exchanges
Geometric Dimensions Output

Each k-level contributes to the final representation:

K Geo Dim Components Info Content
1 2 1 d² + 1 vol² Edge metric
2 4 3 d² + 1 vol² Triangle shape
3 7 6 d² + 1 vol² Tetrahedron form
4 11 10 d² + 1 vol² 5-cell structure
Total 24 Pure geometry

With feat_dim=96: Output = 96 + 24 = 120 dims per k-level, ×4 k-levels = 480 total geometric dims per token.

File Structure 
AbstractPhil/ksimplex-llm-prototype/
├── README.md                 # This file
├── trainer.py               # Training script
├── inference.py             # Generation script
├── config.json              # Model configuration
├── checkpoints/
│   ├── checkpoint_epoch_001.pt
│   ├── checkpoint_epoch_008.pt  # Best val PPL
│   └── checkpoint_latest.pt
└── samples/
    └── samples_epoch_*.json  # Generated text samples
Usage
Inference 
from inference import load_model, generate

model, tokenizer = load_model("AbstractPhil/ksimplex-llm-prototype")

text = generate(
    model, 
    tokenizer,
    prompt="ROMEO: ",
    max_tokens=100,
    temperature=0.8,
    top_k=50
)
print(text)
Training 
python trainer.py \
    --data shakespeare.txt \
    --epochs 50 \
    --batch_size 48 \
    --lr 3e-4
Future Directions
Planned Experiments
  1. Learnable Deformation Scale : Per-k learnable α parameter
  2. Volume Consistency Loss : Maintain k-level differentiation 
    coherence_loss = -torch.std(torch.log(vol2_stack + 1e-10))
  3. K-Depth Ablation : Test k=1,2,3 only (remove k=4 noise floor)
  4. Vol² Normalization : Scale by k to equalize magnitudes
  5. Larger Data : WikiText-103, OpenWebText
Theoretical Questions
  • Does the geometric structure provide better length generalization?
  • Can we interpret k-level activations semantically?
  • Does geometric validity correlate with generation quality?
  • Can we prune k-levels without performance loss?
Citation 
@misc{ksimplex-llm-2026,
  author = {AbstractPhil},
  title = {K-Simplex Language Model: Geometric Autoregression with Cayley-Menger Validation},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/AbstractPhil/ksimplex-llm-prototype}
}
License

MIT License - Free to use, modify, and distribute.

Acknowledgments

Built on the foundation of geometric deep learning research exploring k-simplex structures, pentachoron navigation, and Cayley-Menger determinant validation for neural network regularization.

"The geometry is the representation."

Runs of AbstractPhil ksimplex-llm-prototype on huggingface.co

5
Total runs
1
24-hour runs
2
3-day runs
2
7-day runs
0
30-day runs

More Information About ksimplex-llm-prototype huggingface.co Model

ksimplex-llm-prototype huggingface.co

ksimplex-llm-prototype huggingface.co is an AI model on huggingface.co that provides ksimplex-llm-prototype's model effect (), which can be used instantly with this AbstractPhil ksimplex-llm-prototype model. huggingface.co supports a free trial of the ksimplex-llm-prototype model, and also provides paid use of the ksimplex-llm-prototype. Support call ksimplex-llm-prototype model through api, including Node.js, Python, http.

ksimplex-llm-prototype huggingface.co Url

https://huggingface.co/AbstractPhil/ksimplex-llm-prototype

AbstractPhil ksimplex-llm-prototype online free

ksimplex-llm-prototype huggingface.co is an online trial and call api platform, which integrates ksimplex-llm-prototype's modeling effects, including api services, and provides a free online trial of ksimplex-llm-prototype, you can try ksimplex-llm-prototype online for free by clicking the link below.

AbstractPhil ksimplex-llm-prototype online free url in huggingface.co:

https://huggingface.co/AbstractPhil/ksimplex-llm-prototype

ksimplex-llm-prototype install

ksimplex-llm-prototype is an open source model from GitHub that offers a free installation service, and any user can find ksimplex-llm-prototype on GitHub to install. At the same time, huggingface.co provides the effect of ksimplex-llm-prototype install, users can directly use ksimplex-llm-prototype installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

ksimplex-llm-prototype install url in huggingface.co:

https://huggingface.co/AbstractPhil/ksimplex-llm-prototype

Url of ksimplex-llm-prototype

ksimplex-llm-prototype huggingface.co Url

Provider of ksimplex-llm-prototype huggingface.co

AbstractPhil
ORGANIZATIONS
huggingface.co

Other API from AbstractPhil

 huggingface.co
AbstractPhil/clips

Total runs: 7
Run Growth: 2
Growth Rate: 28.57%
Updated:November 25 2025
huggingface.co
AbstractPhil/SIM-V5

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated:March 27 2025