Bochkov / llm-fix-min-baseline-learned-input-table-model-classic

Learned Input Table Model Classic

Research checkpoint for the paper:

Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes

Model variant

This repository contains the learned input table baseline .

The model is a 32-layer decoder-only Transformer with:

• vocabulary size: 65,536
• model width: 1024
• number of layers: 32
• number of attention heads: 32
• context length: 1024
• rotary positional embeddings
• GELU activations
• untied trainable output projection

This baseline uses a standard trainable input embedding table of size:

65,536 x 1024 = 67,108,864 trainable input parameters

Intended use

This checkpoint is provided for reproducibility of the paper's controlled comparison. It is intended for research use only.

Loading example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "Bochkov/llm-fix-min-baseline-learned-input-table-model-classic"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

prompt = "Question: What is the capital of United Kingdom?\nAnswer:"
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)

with torch.no_grad():
    output_ids = model.generate(input_ids, max_new_tokens=3, do_sample=False)

print(tokenizer.decode(output_ids[0].tolist()))

Limitations

This is a small research language model trained for architectural comparison. It is not instruction-tuned for safe deployment and should not be used as a production system.

Training data

The model was trained on the same FineWeb-Edu + Cosmopedia mixture used for the matched comparisons in the paper. Dataset terms and licenses are those of the original datasets.

🧑‍🔬 Citation & Concept

If you use this model or the underlying concepts in your research, please cite our work:

@misc{bochkov2026languagemodelstrainableinput,
      title={Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes}, 
      author={A. Bochkov},
      year={2026},
      eprint={2605.09751},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.09751}, 
}