lightonai / colgrep-agent-2B-GGUF

ColGrep Agent 2B

The model behind colgrep --agent .

ColGrep Agent is a small code-localization agent. Give it a question about a repository ("where are retries configured?", "why does the export crash on empty files?") and it searches the code with colgrep (semantic + regex search), reads the candidate files through a read-only terminal, and returns the files and line ranges that answer the question. It runs locally, on the GPU when there is one, and works as a search sub-agent for a larger coding agent.

Quick start

1. Install colgrep and the model

Homebrew (macOS / Linux):

brew install lightonai/tap/colgrep && colgrep --install-agent

Shell installer (macOS / Linux):

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/lightonai/next-plaid/releases/latest/download/colgrep-installer.sh | sh && "${CARGO_HOME:-$HOME/.cargo}/bin/colgrep" --install-agent

Windows (PowerShell):

irm https://github.com/lightonai/next-plaid/releases/latest/download/colgrep-installer.ps1 | iex
colgrep --install-agent

Cargo (from source):

cargo install colgrep && colgrep --install-agent

colgrep --install-agent downloads colgrep-agent-2B-Q8_0.gguf from this repository (one time, 2.5 GB), checks that it loads, and prepares the prompt cache so the first question is as fast as the next ones. Nothing else is needed: no Python, no server, no API key.

2. Index a repository

cd /path/to/project
colgrep init

After the first init , every search notices changed files and updates the index before answering.

3. Ask a question

colgrep --agent "where does the agent decide which GPU backend llama.cpp uses"

The tool calls and answer of a real session on colgrep's own source code (M3 Pro, Metal; each call's result line is left out):

$ colgrep --agent "where does the agent decide which GPU backend llama.cpp uses"
◆ colgrep agent  · colgrep-agent-2B-Q8_0.gguf (GPU)
❯ where does the agent decide which GPU backend llama.cpp uses

  ● colgrep   llama.cpp GPU backend
  ● terminal  cat -n agent/src/llm/llama.rs
  ● terminal  cat -n agent/src/llm/server.rs
  ● terminal  cat -n agent/src/engine.rs
  ● terminal  cat -n agent/src/llm/llama.rs | sed -n '80,150p'
  ● terminal  cat -n agent/src/llm/server.rs | sed -n '65,130p'
  ● colgrep   find_gpu
  ● terminal  cat -n agent/src/llm/server.rs | sed -n '1,100p'
  ● finish    agent/src/llm/server.rs:103-105, agent/src/llm/server.rs:65-69, …

✓ 5 locations · 9 turns · 2 searches · 6 terminal commands · 21.7s

agent/src/llm/server.rs:103-105
agent/src/llm/server.rs:65-69
agent/src/llm/llama.rs:101-101
agent/src/llm/llama.rs:115-116
agent/src/engine.rs:57-67

Progress goes to stderr, and only on a terminal; stdout carries just the file:start-end locations, so the output pipes cleanly into other tools.

More ways to ask:

colgrep --agent "crash when the config file is empty" ./backend -c   # scope to a folder, print the lines
colgrep --agent --json "where are retries configured"              # JSON, for scripts and agents

Use it from a coding agent

colgrep installs hooks and skills for coding agents:

colgrep --install-claude-code   # Claude Code
colgrep --install-codex         # Codex
colgrep --install-opencode      # OpenCode

Once the model is installed, the Claude Code hooks tell the coding agent about colgrep --agent , so it can hand open-ended "where is X handled?" questions to this model and get back exact locations, instead of spending its own context on exploratory searches and file reads.

What happens during a question

• Read-only. The model's terminal is interpreted inside colgrep: no process is spawned and no file is opened for writing. cat , head , tail , sed -n , grep , find , wc , pipes and globs work; redirections, sed -i , rm , interpreters and git are refused. Paths are confined to the repository.
• Reproducible. The sampling seed is fixed: the same question on the same repository state gives the same answer, run after run ( colgrep settings --agent-seed N draws other samples).
• Local GPU inference. On Apple Silicon, llama.cpp is built into colgrep and runs on Metal. On Linux, Windows and Intel Macs, colgrep downloads ggml-org's official llama.cpp build (pinned, SHA-256 verified) and uses the GPU through Vulkan (NVIDIA, AMD, Intel) when a driver is installed, or the CPU otherwise.

Measured on an M3 Pro with the Q8_0 file, for typical 6–10 turn sessions: about 20 s per question on Metal and 75–90 s on CPU only, with 3–4 GB of peak memory.

Files

colgrep downloads Q8_0 by default. To use another quantization, for a smaller download or faster CPU-only inference, select it with colgrep settings --agent-model-file colgrep-agent-2B-Q4_K_M.gguf . Only Q8_0 has been evaluated through colgrep; the smaller quantizations may localize code less accurately.

The model is a 2B-parameter, 42-layer Llama-family model with a 131,072-token context window. The tokenizer and chat template are embedded in every file.

Serving the model on a GPU server

colgrep can send the agent's prompts to any OpenAI-compatible server through /v1/completions , so no server-side tool parser is needed:

# llama.cpp: --special keeps the tool-call tokens in the output
llama-server -m colgrep-agent-2B-Q8_0.gguf --port 8000 --special
colgrep settings --agent-endpoint http://localhost:8000/v1

colgrep settings --agent-endpoint default switches back to the local model.

Evaluation

F1 values are mean ± 95% confidence interval across five evaluation seeds (300 tasks per benchmark), using strict predictions without rescue.