CMSManhattan / JiRack_GPT3_empty

Model Architecture Overview

Architectures Included

I have added my empty models based on the following architectures:

• GPT-3 Standard
• Llama 3
• Mistral

For smaller models modeled after GPT-2 , I utilize LayerNorm and FFN layers. For larger models, these layers are replaced with RMSNorm and SwiGLU , enabling a smoother transition to architectures with larger parameter sizes (8B, 33B, 70B, and 120B).

Tokenizer Choices

• For English models: GPT-2 Hugging Face tokenizer
• For multilingual models: BERT tokenizer from the Hugging Face library

Training and Tuning

The Transformer block is not frozen , providing greater flexibility and power when tuning models from scratch.

Model Architecture Details

GPT-2 Architecture (Classic, Transformer-like)

CustomEmbedding
FrozenSignatureLayer
LearnedPositionalEmbedding
[TransformerBlock]
    ├── MultiHeadAttention
    ├── LayerNorm
    ├── LayerNorm
    ├── FFN
          ├── Linear
          ├── Activation: GELU
          └── Linear
LayerNorm
Linear

GPT-3 Architecture (Similar to Llama 3 & Mistral)

CustomEmbedding
# Positional Embedding removed, RoPE integrated in Attention
[TransformerBlock]
    ├── MultiHeadAttention
    ├── SwiGLUFeedForward
          ├── Linear (Gate Layer)
          ├── Linear (Up Layer)
          └── Linear (Projection/Down Layer)
    └── RMSNorm
RMSNorm
Linear
FrozenSignatureLayer

Note: The large model architectures replace specific layers:

• LayerNorm → RMSNorm
• FFN → SwiGLU

Welcome to ask to design your corp model over 33B or 70B or more parameters

CMS Manhattan
Copyright © 2002–2026