A high-performance inference router that orchestrates local LLM requests across multiple machines using an intelligent 7-signal scoring engine.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install local-llm-router
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install local-llm-router using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Local LLM Router (Ollama Herd)?
The local LLM router is a sophisticated orchestration layer designed for developers and homelab enthusiasts who run AI models across multiple devices. Instead of manually switching endpoints or dealing with VRAM limitations on a single machine, this Openclaw Skills integration provides a unified OpenAI-compatible gateway. It intelligently monitors the state of every node in your fleet—including macOS, Linux, and Windows machines—to ensure your inference requests are handled by the most capable device at any given moment.
By sitting in front of your Ollama instances, the local LLM router eliminates the friction of manual configuration. It handles the heavy lifting of load balancing, ensuring that large models like Llama 3.3 or DeepSeek are routed to high-VRAM nodes while smaller tasks are distributed to available secondary machines. This setup maximizes hardware utilization and significantly reduces latency by keeping models 'hot' in memory whenever possible.
Local LLM Router (Ollama Herd) Use Cases
- Multi-GPU Orchestration: Distributing inference loads across a cluster of Mac Studios, Linux rigs, and Windows PCs.
- High-Availability AI: Implementing automatic failover so that if one local LLM node goes offline, requests are instantly re-routed.
- Latency Optimization: Using historical p75 data to pick the fastest available node for specific model requests.
- VRAM Management: Automatically routing requests based on available memory headroom and model size requirements.
- Development Workflows: Providing a single, stable API endpoint for local LLM development that doesn't change when backend hardware is upgraded or moved.
How Local LLM Router (Ollama Herd) Works
- Node Discovery: The local LLM router identifies all active nodes running the herd-node agent across the network.
- 7-Signal Scoring: Every incoming request is processed by a scoring engine that evaluates thermal state, memory headroom, queue depth, latency history, role affinity, availability trends, and context fit.
- Intelligent Routing: The request is dispatched to the node with the highest score, prioritizing instances where the model is already loaded in VRAM.
- Context Protection: The router inspects the request's context size requirements and may upgrade the model or strip unnecessary parameters to prevent performance degradation.
- Telemetry & Feedback: Post-inference data (latency, tokens, success/failure) is stored in a local SQLite database to refine future scoring decisions.
Local LLM Router (Ollama Herd) Setup
Install the local LLM router and its node agents using the following commands:
# Install the core package
pip install ollama-herd
# Start the central router and scoring engine
herd
# On every other machine in your fleet, start the node agent
herd-node
Once running, point your OpenAI-compatible SDKs to http://localhost:11435/v1.
Local LLM Router (Ollama Herd) Data Schema & Taxonomy
The local LLM router maintains a structured persistent layer within the ~/.fleet-manager/ directory to manage performance and logs:
| Component | File/Path | Description |
|---|---|---|
| Latency DB | latency.db |
SQLite database storing p75 latency, TTFT, and request traces. |
| Logs | logs/herd.jsonl |
JSON-formatted logs for system events and routing errors. |
| Traces | /dashboard/api/traces |
Real-time API access to recent routing decisions and score breakdowns. |
| Metadata | api/tags |
Aggregated list of all models available across the entire fleet. |
Local LLM Router (Ollama Herd) Advanced Features
- VRAM-Aware Fallback: Automatically shifts requests to similar models if the primary selection is at capacity.
- Context-Size Protection: Intercepts and adjusts
num_ctxvalues to prevent unnecessary model reloading in Ollama. - Zombie Reaper: Automatically identifies and cleans up stalled or orphaned inference requests to free up GPU resources.
- Auto-Pull Logic: Can be configured to automatically download missing models to the most suitable node based on fleet inventory.
- Interactive Dashboard: A full Web UI providing real-time insights into fleet health, model recommendations, and hardware benchmarks.
SKILL.md
Loading
Related Openclaw Skills
Peekdata
A powerful data analytics integration that automates visualization, reporting, and dataset management via the Membrane CLI.
Search This Image
An advanced reverse image search skill for finding image origins, similar visuals, and object identification via URL or file upload.
qBittorrent Web API Skill
An automation-focused skill for controlling qBittorrent instances via its Web API to manage torrent lifecycles, speed limits, and RSS feeds.
Stock Portfolio Management
A comprehensive stock portfolio management and price alert skill supporting A-shares, Hong Kong, and US markets with local data storage.
CSP Reference Tool
A specialized reference tool for managing, debugging, and implementing Content Security Policy (CSP) within developer tool contexts.
Job Tracker
A local SQLite-powered utility for AI agents to manage job applications, recruiter contacts, and interview schedules.
Featured*
