Bochkov / growing-transformers-model-unfrozen-baseline-monolyth-247m

Growing Transformers — Unfrozen Baseline (Monolithic, 247M)

This repository contains growing-transformers-model-unfrozen-baseline-monolyth-247m , a classic monolithic baseline model from the paper:

📚 Paper (Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate) -

📚 Paper (Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations) -

It is part of the comparative-study collection:
https://huggingface.co/collections/Bochkov/growing-transformers-layer-wise-expansion-comparative-study

Code:
https://github.com/AVBochkov/PGT

What this model is (in one paragraph)

This is a 9-layer decoder-only Transformer trained in the fully classical way : monolithic end-to-end training from scratch , with no constructive / layer-wise growth and no frozen embeddings . The token embedding matrix is randomly initialized and fully trainable , so semantic structure can be learned directly in the input embeddings (as in standard GPT-like training).

This repo exists as a clean baseline for controlled comparisons against the constructive-growth models and against models with frozen embedding substrates.

Primary comparison (why this repo exists)

This model is intended to be compared to:

• Bochkov/growing-transformers-model-16-bit-1-9-181m
  (constructive, layer-wise growth; frozen 16-bit embeddings)

What is identical

• Same controlled-study Transformer stack architecture ( 9 layers , d_model=1024 , n_head=32 )
• Same tokenizer family / vocabulary size ( 65,536 )
• Same context length used in training ( 1024 )

What differs

• Training procedure
  • This repo: monolithic end-to-end training (all layers trained together from scratch)
  • 16-bit constructive repo: trained in stages (1–3, then 4–6, then 7–9), freezing previously trained layers
• Embedding layer
  • This repo: standard trainable embedding matrix ( vocab_size × d_model )
  • 16-bit repo: extremely small frozen embedding substrate (16-dim binary signal expanded to d_model )

Important note on parameter count (why this model is larger)

This model has more parameters than the 16-bit models because it includes a full trainable embedding matrix:

• Trainable embedding size here: 65,536 × 1,024 ≈ 67.1M parameters
• In the 16-bit setup, the embedding-related parameters are ~ 1.0M (as reported in the paper)

So even with the same Transformer block stack , the total parameter count differs primarily due to the embedding matrix.

Model architecture (controlled study)

• Type: decoder-only Transformer (GPT-like)
• Layers: 9
• Hidden size: d_model = 1024
• Heads: n_head = 32
• Vocabulary size: 65,536
• Context length used in training: 1024
• Embedding: standard trainable token embedding matrix ( vocab_size × d_model )

Parameter count

• Total: ≈247.6M
• Trainable: ≈247.6M
• Frozen: 0.0M

(Counts follow the paper’s controlled-study table.)

Tokenizer

Canonical tokenizer repository:

• https://huggingface.co/Bochkov/bvv241-2-3
  (collection: https://huggingface.co/collections/Bochkov/tokenizers )

Intended use

Research / analysis of:

• monolithic end-to-end training vs constructive (layer-wise) growth
• the role of embedding learning vs frozen embedding substrates
• controlled comparisons across embedding types (trainable vs UNICODE frozen vs 16-bit frozen)

Not intended as a general-purpose assistant model. Outputs may be unreliable and the model may reflect biases present in the training data.

How to use (Transformers)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Bochkov/growing-transformers-model-unfrozen-baseline-monolyth-247m")
model = AutoModelForCausalLM.from_pretrained("Bochkov/growing-transformers-model-unfrozen-baseline-monolyth-247m", trust_remote_code=True).to('cuda')

inputs = torch.tensor([tokenizer.encode("Write a short poem about the ocean. ")], dtype=torch.long, device='cuda')

outputs = model.generate(
    inputs, 
    max_new_tokens=50,
    do_sample=False
)
print(tokenizer.decode(outputs[0].tolist()))
#Write a short poem about the ocean. The poem is a poem about the sea and the sea and the sea and the sea. The poem is ab

inputs = torch.tensor([tokenizer.encode("Question: What is the capital of India?\nAnswer:")], dtype=torch.long, device='cuda')

outputs = model.generate(
    inputs, 
    max_new_tokens=10,
    do_sample=False
)
print(tokenizer.decode(outputs[0].tolist()))
#Question: What is the capital of India?
#Answer:Chennai
#    </s><

🧑‍🔬 Citation & Concept

If you use this model or the underlying concepts in your research, please cite our work:

@article{
      bochkov2025emergent,
      title={Emergent Semantics Beyond Token Embeddings: Transformer {LM}s with Frozen Visual Unicode Representations},
      author={Andrey Bochkov},
      journal={Transactions on Machine Learning Research},
      issn={2835-8856},
      year={2025},
      url={https://openreview.net/forum?id=Odh8IynO1o},
      note={}
}

@misc{bochkov2025growingtransformersmodularcomposition,
      title={Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate}, 
      author={A. Bochkov},
      year={2025},
      eprint={2507.07129},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2507.07129}, 
}