aimeri / SpoomplesMaxx-CPT-2-Base

SpoomplesMaxx Base Continued Pre-Training - Checkpoint 2/3

A continued pre-training (CPT) checkpoint of Qwen/Qwen3-14B-Base fine-tuned on creative writing and roleplay data. This is checkpoint 2 of 3 (2000/3000 steps).

Model Description

This model is part of the SpoomplesMaxx training pipeline: CPT → SFT → DPO

The CPT stage teaches the model:

• Character understanding and portrayal
• Creative fiction writing patterns
• Fandom/wiki-style lore knowledge
• Dialogue patterns for roleplay

Training Data

Phase 1: Core Knowledge

This checkpoint was trained on data focused on character knowledge and lore:

Total training samples: ~25,000 (after priority repeats and sampling)

Note on private data: The character cards and AO3 datasets are private. The character cards contain ~200 curated SillyTavern character definitions with detailed personalities, scenarios, and example dialogues. The AO3 dataset contains ~20k creative fiction entries from Archive of Our Own.

Training Configuration

Hardware

• GPU: 1× NVIDIA A800
• Training Time: ~12 hours for 2000 steps

Intended Use

This is an intermediate checkpoint intended for:

• Monitoring training progress
• Evaluating knowledge acquisition during CPT
• Research into continued pre-training dynamics

Not Recommended For:

• Production deployment (use final model after full CPT → SFT → DPO pipeline)
• Direct chat/instruction following (this is a base model continuation, not instruction-tuned)

Limitations

• Incomplete training: This is checkpoint 2/3 - the model has only seen 66% of planned CPT training.
• No instruction tuning: This model continues raw text, not chat/instructions
• Private data bias: Heavy weighting toward private character cards may introduce specific character patterns
• NSFW content: Training data includes creative fiction that may contain mature themes. No safety filtering was applied at this stage.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "aimeri/SpoomplesMaxx-CPT-2-Base",
    dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("aimeri/SpoomplesMaxx-CPT-2-Base")

# CPT models continue text, not chat
prompt = "The castle stood silent against the darkening sky, its towers reaching toward clouds that promised rain. Inside,"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.8)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

If you use this model, please cite the base model and datasets:

@misc{qwen3-14b-base,
  title={Qwen3-14B-Base},
  author={Qwen Team},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/Qwen/Qwen3-14B-Base}
}

Acknowledgments

• Qwen Team for the excellent base model
• nyuuzyou for the Fandom wiki dataset
• Archive of Our Own for creative fiction