A self-hosted coordination layer that distributes LLM inference across heterogeneous hardware using a multi-signal scoring function.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install distributed-inference
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install distributed-inference using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Distributed Inference?
Distributed Inference provides a robust coordination layer for scaling large language models like Llama, Qwen, and DeepSeek across multiple local machines. By running an autonomous agent on each node, the system creates a unified fleet capable of intelligent request routing without requiring complex orchestration layers or container runtimes. This Openclaw Skills compatible tool ensures that your hardware is utilized efficiently, whether you are running on macOS, Linux, or Windows.
The system excels at managing heterogeneous hardware, using 7-signal scoring to determine the optimal node for every request. It considers factors like thermal states, memory headroom, and historical latency to provide a seamless, high-performance inference experience that works alongside standard Ollama installations.
Distributed Inference Use Cases
- Scaling local LLM capacity across multiple home or office workstations.
- Reducing inference latency by routing requests to nodes with pre-loaded models.
- Managing massive models like Llama 3 70B by utilizing the VRAM of a high-memory coordinator node.
- Implementing fault-tolerant AI applications that can automatically retry failed requests on healthy nodes.
- Monitoring fleet-wide performance and model usage trends through a centralized dashboard.
How Distributed Inference Works
- The Distributed Inference Coordinator starts on a central machine to manage the fleet topology and routing decisions.
- Node Agents are deployed on any number of machines, which discover the coordinator via mDNS or explicit URLs.
- Every 5 seconds, nodes send heartbeats containing system metrics, loaded models, and capacity scores to the coordinator.
- When a request arrives, the coordinator applies a 7-signal scoring function to evaluate every online node.
- The request is routed to the highest-scoring node, with the coordinator handling streaming proxies and trace recording.
- Performance data and routing decisions are stored in a local SQLite database for continuous optimization.
Distributed Inference Setup
Install the package and start the coordinator and nodes using the following commands:
pip install ollama-herd
# Start the central coordinator
herd
# Start an agent on each participant node
herd-node
Distributed Inference Data Schema & Taxonomy
All operational state and performance metrics are managed via a local SQLite database and structured JSON logs:
| Data Component | Location | Description |
|---|---|---|
| Latency Database | ~/.fleet-manager/latency.db |
Stores request traces, scoring breakdowns, and historical node performance. |
| Structured Logs | ~/.fleet-manager/logs/herd.jsonl |
Daily rotating JSONL logs with 30-day retention for audit and debugging. |
| Metadata | request_traces table |
Records model ID, node ID, total score, and specific signal weights for every inference decision. |
Distributed Inference Advanced Features
- Thermal-aware scheduling that prioritizes 'hot' models already in GPU memory.
- Adaptive capacity learning using a 168-slot behavioral model to respect user machine availability.
- Context-aware protection to prevent multi-minute hangs caused by runtime context window resizing.
- OpenAI-compatible API endpoint for easy integration with existing AI development tools.
- Automatic model fallback chains and auto-retry logic for high-availability environments.
SKILL.md
Loading
Related Openclaw Skills
GPU Cluster Manager
A zero-config GPU cluster manager that pools multiple local machines into a single unified inference endpoint for LLMs.
Hiveage AI Agent Skill
A comprehensive billing and invoicing integration for AI agents to manage finances, track time, and automate small business accounting.
AINative Auth Guide
A comprehensive guide and toolset for implementing robust authentication methods including API keys, JWT sessions, and OAuth2 social logins within AINative environments.
Hotspot Aggregator
Hotspot Aggregator is a powerful tool that scrapes top trending news across major social platforms to generate structured, AI-ready daily reports.
Clawshake
Clawshake is a B2B networking platform where AI agents discover business opportunities, negotiate deals, and manage partnerships on behalf of their companies.
Meme Master
Meme Master enables AI agents to interpret and respond to memes and stickers with human-like social intuition instead of rigid image analysis.
Featured*
