Abiray / Sutra-Instruct-350M-GGUF

Sutra-Instruct-350M-GGUF

This repository contains the official GGUF quantizations of Sutra-Instruct-350M , a 350-million parameter model trained entirely from scratch by Abhiray.

The Sutra (सूत्र) series is designed to be a high-speed, rule-based instruction model. These GGUF versions are optimized for ultra-low latency inference on consumer-grade hardware, including CPUs and mobile devices.

📁 Available Quantizations

🚀 Quick Start (Inference)

1. Using llama.cpp

Run the model directly from your terminal using llama-cli :

./llama-cli -m sutra-Q8_0.gguf -p "Instruction: What is the law of gravity?\n\nResponse:" -n 400 --temp 0.55 --repeat-penalty 1.2

2. Using Ollama (Plug-and-Play)

To use this with Ollama, create a file named Modelfile with the following content:

FROM ./sutra-Q8_0.gguf TEMPLATE """Instruction: {{ .Prompt }}

Response: """ PARAMETER stop "Instruction:" PARAMETER stop "\nInstruction:" PARAMETER temperature 0.55 PARAMETER repeat_penalty 1.2 PARAMETER top_k 50 PARAMETER num_predict 512

Then, initialize and run the model:

ollama create sutra -f Modelfile ollama run sutra

⚙️ Optimized Settings To prevent the model from looping or hallucinating, we strongly recommend these inference parameters:

Temperature: 0.55

Repeat Penalty: 1.2

Top-K: 50

Max Tokens: 512