borisbat / dasllama-stories

dasLLAMA story models

The two story models behind dasllama.io's browser examples, as GGUF. dasLLAMA is the daslang inference engine ( https://github.com/GaijinEntertainment/daScript , modules/dasLLAMA ); the examples are examples/dasLLAMA/storyteller (stories15M writes, KittenTTS reads) and examples/dasLLAMA/storywish (you type the words, tinystories-instruct-27M writes a story that uses them, KittenTTS reads it). Both are plain llama architecture, so llama.cpp and anything GGUF-based runs them too.

Files

Both GGUFs carry llama.context_length as the models were trained: 256 for stories15M (the llama2c converter writes 128; the storyteller pins 256 itself) and 512 for tinystories-instruct-27M.

tinystories-instruct-27M

A 27M-parameter llama (dim 512, 8 layers, 8 heads, SwiGLU, RoPE, tied embeddings, 4096-token vocabulary) trained from scratch on the TinyStoriesInstruct corpus, so that a request written the corpus's way steers the story. The point of it: the published TinyStories-Instruct models are GPT-Neo, which GGUF engines do not run; this one is llama architecture with the same behaviour.

Prompt

The corpus's own layout - any subset of the fields, in any order, Story: last, then a blank line. Sample at temperature 0.8, top-k 40; stop at EOS. The corpus orders its fields at random, so the model sometimes appends a Summary: line after the story: cut at the first line that opens with a field name.

Features: Dialogue
Words: dragon, cake, moon
Story: 

Fields the corpus knows: Words: (a comma list), Summary: (a sentence), Features: (any of Dialogue, BadEnding, MoralValue, Twist, Foreshadowing, Conflict), Random sentence: (a sentence the story must contain).

dasllama-server --model tinystories-instruct-27M-Q8_0.gguf
llama-cli -m tinystories-instruct-27M-Q8_0.gguf -p $'Words: dragon, cake, moon\nStory: \n\n' --temp 0.8 --top-k 40

How well it follows

24 sampled stories per word triple (temperature 0.8, top-p 0.9), counting the stories that contain all three words; the official GPT-Neo TinyStories-Instruct-33M under the same test:

Without a Words: line the same words appear in 0 to 1 of 24 stories. Common nouns land; rarer ones (robot, pizza) land one at a time. Validation loss 1.144 nats per token on the corpus's validation split under this vocabulary.

Words: dragon, cake, moon

Once upon a time there was a dragon who enjoyed cake very much. Every day, he would go flying around the world with a big cake in his cave. He would look up at the moon and remember how the moon made him feel. One day, the dragon was walking...

Recipe

• Data : roneneldan/TinyStoriesInstruct , the train file as published (2,476,533 examples, <|endoftext|> -separated), tokenized as-is with BOS before and EOS after each example: 683,848,778 tokens, 276 per example. The validation file (25,027 examples) is the held-out split.
• Tokenizer : SentencePiece BPE, 4096 pieces, trained on the first 150,000 examples with llama2.c's settings (byte fallback, identity normalization, digits split) - tok4096.model .
• Model : llama2.c's model.py , dim 512, 8 layers, 8 heads, 8 KV heads, hidden 1376 (multiple of 32), max sequence 512, no dropout, tied embeddings. 27.4M parameters.
• Training : llama2.c's train.py on one H100 SXM, bf16 autocast, torch.compile ; AdamW (betas 0.9/0.95, weight decay 0.1), learning rate 5e-4 with 500 warmup steps and cosine decay to 0, gradient clip 1.0, 256 sequences of 512 tokens per step (131,072 tokens), 9500 steps = 1.25B tokens = 1.8 epochs, 15.5 minutes. Train loss 1.163, validation loss 1.144 at the end, still falling.
• Export : export.py --version 0 to the llama2.c checkpoint, llama.cpp's llama-convert-llama2c-to-ggml --copy-vocab-from-model tok4096.bin (98c4764b6) to f32 GGUF, llama-quantize Q8_0 , llama.context_length set to 512.

The trainer, the data preparation and the hit-rate test are llama2.c's scripts with a data module for this corpus.

stories15M

Andrej Karpathy's llama2.c stories15M (dim 288, 6 layers, 6 heads, the llama-2 32K vocabulary), trained on TinyStories. Converted from karpathy/tinyllamas ( stories15M.bin + tokenizer.bin ) with llama.cpp's llama-convert-llama2c-to-ggml and quantized to Q8_0. The storyteller prompts it with Once upon a time and stops at BOS, the corpus's story separator. No instruction following: it writes what it wants.