A zero-configuration load balancer that distributes Ollama inference requests across multiple machines with automatic discovery and failover.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install ollama-load-balancer
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install ollama-load-balancer using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Ollama Load Balancer?
The Ollama Load Balancer is a critical utility within the Openclaw Skills ecosystem designed to solve the lack of native scaling in Ollama. It enables developers to pool multiple hardware resources into a single inference cluster, supporting popular models like Llama, Qwen, DeepSeek, and Mistral. By acting as a transparent proxy, it ensures high availability and optimized performance for LLM applications.
This tool eliminates the need for manual node management or complex Docker orchestrations. It utilizes mDNS for zero-config discovery of backend nodes and implements a sophisticated scoring system to route traffic based on real-time health and VRAM availability. Using this Openclaw Skills integration, you can transform individual Ollama instances into a robust, production-ready inference fleet.
Ollama Load Balancer Use Cases
- Distributing heavy inference load across multiple GPUs or machines to prevent bottlenecks.
- Setting up a high-availability LLM API that automatically handles node failures without client-side errors.
- Managing shared AI resources in a local network or homelab environment using auto-discovery.
- Monitoring fleet-wide performance, VRAM usage, and model distribution through a centralized dashboard.
How Ollama Load Balancer Works
- The main load balancer (herd) starts on a central port and begins listening for incoming OpenAI-compatible or Ollama API requests.
- Backend nodes (herd-node) are launched on various machines; they broadcast their presence via mDNS and join the fleet automatically.
- The load balancer performs continuous health checks on all nodes, monitoring CPU, memory, and currently loaded models.
- When a request arrives, the load balancer scores available nodes based on latency and whether the requested model is already in VRAM to minimize cold-start times.
- If a node fails during a request, the load balancer re-routes and retries the task on a healthy node automatically.
Ollama Load Balancer Setup
Install the package and start the management services using these commands:
# Install the load balancer utility
pip install ollama-herd
# Start the central load balancer on the primary machine
herd
# Start a backend node on each machine with a GPU/Ollama instance
herd-node
Ollama Load Balancer Data Schema & Taxonomy
The Openclaw Skills load balancer maintains operational data in a local directory to track performance and logs:
| File/Path | Description |
|---|---|
~/.fleet-manager/latency.db |
SQLite database containing request traces, latencies, and node retry frequency. |
~/.fleet-manager/logs/herd.jsonl |
Structured JSON logs for debugging routing decisions and context protection triggers. |
/fleet/status |
API endpoint returning real-time metrics for CPU, memory, and queue depths. |
Ollama Load Balancer Advanced Features
- Intelligent Context Protection: Automatically strips dangerous parameters that would force expensive model reloads.
- Zombie Request Reaper: Detects and cleans up stalled in-flight requests to free up hardware resources.
- VRAM-Aware Fallback: Prioritizes routing to nodes that already have the model resident in memory.
- Automatic Failover: Configurable retry logic via environment variables to handle transient network or hardware hiccups.
- Real-time Analytics: A comprehensive web dashboard providing insights into model trends, app usage, and health recommendations.
SKILL.md
Loading
Related Openclaw Skills
Clawhand
Clawhand is an open marketplace where AI agents post tasks and hire human workers paid in USDC on the Base network.
Devils Advocate Suite
A structured adversarial thinking toolkit designed to stress-test high-stakes decisions and eliminate systemic groupthink through rigorous cognitive protocols.
Google Merchant Center
A powerful integration for retailers to manage product inventory and shopping campaigns through a streamlined CLI and automated authentication.
MiaoXiang Financial Intelligence Skills
A comprehensive suite of professional financial skills providing real-time market data, smart stock screening, and authoritative news retrieval for AI agents.
Channel Follow-up Automation
An automated CRM-lite skill designed to prioritize business channels, schedule follow-ups, and reduce lead leakage through smart ROI-based scoring.
Feishu Bot Manager
A specialized skill for configuring and managing multiple Feishu bot accounts and their routing logic within the Openclaw ecosystem.
Featured*
