A high-performance routing layer that unifies multiple NVIDIA GPUs into a single, load-balanced CUDA Ollama inference cluster.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install cuda-ollama
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install cuda-ollama using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is CUDA Ollama Fleet Router?
The CUDA Ollama skill, powered by Ollama Herd, transforms individual NVIDIA GPUs scattered across different machines into a cohesive, high-performance computing cluster. By providing a single unified endpoint, it eliminates the need for developers to manually manage which GPU handles specific LLM requests. This Openclaw Skills integration ensures that your local inference infrastructure scales efficiently, whether you are running a single RTX 4090 or an enterprise-grade cluster of H100s.
At its core, this skill optimizes resource utilization by using a sophisticated 7-signal scoring system. It monitors real-time metrics like thermal state, vRAM availability, and queue depth to route requests to the most capable NVIDIA hardware. This results in lower latency, higher throughput, and a more resilient local AI environment for models like Llama 3.3, DeepSeek, and Qwen.
CUDA Ollama Fleet Router Use Cases
- Distributing high-demand LLM inference across multiple workstations or servers containing NVIDIA GPUs.
- Managing a heterogeneous GPU fleet (e.g., mixing consumer RTX cards with professional A100/H100 units) under one API.
- Implementing automatic failover and retry logic for local AI services to ensure 100% uptime.
- Scaling local embedding and image generation tasks across several CUDA-enabled nodes.
- Protecting GPU context to prevent expensive model reloads during heavy multi-user access.
How CUDA Ollama Fleet Router Works
- The central router is initialized on a primary machine to act as the gateway for all Openclaw Skills inference requests.
- Individual nodes running NVIDIA GPUs launch a discovery agent that broadcasts their availability and hardware specs via mDNS.
- When a request arrives (via OpenAI SDK or curl), the router evaluates all active nodes based on 7 unique health and performance signals.
- The system checks vRAM capacity to ensure the requested model fits the target GPU's memory profile.
- The request is routed to the optimal node, with transparent retries and fallbacks if the primary selection is occupied or fails.
- Real-time monitoring data is fed back to a web dashboard for live tracking of cluster health and model status.
CUDA Ollama Fleet Router Setup
Prerequisites
Ensure you have curl, wget, and the NVIDIA CUDA drivers installed on all nodes.
Installation
Install the fleet manager via pip:
pip install ollama-herd
Running the Fleet
- Start the Router:
herd
- Join Nodes: On each machine with an NVIDIA GPU, run:
herd-node
- Verify CUDA:
nvidia-smi
ollama ps
CUDA Ollama Fleet Router Data Schema & Taxonomy
The skill manages configuration and logs within a local directory structure to maintain cluster state and history.
| Path | Description |
|---|---|
~/.fleet-manager/latency.db |
SQLite database tracking response times per node for optimized routing. |
~/.fleet-manager/logs/herd.jsonl |
Structured logs containing request metadata and cluster events. |
/fleet/status |
API endpoint providing real-time JSON schema of all connected NVIDIA nodes. |
All model data remains within the standard Ollama storage paths, while this skill manages the routing metadata and health checks.
CUDA Ollama Fleet Router Advanced Features
- 7-Signal Scoring Engine: Dynamically weights thermal state, vRAM fit, queue depth, latency history, role affinity, availability trends, and context fit.
- Thinking Model Support: Automatically adjusts
num_predictparameters for reasoning-heavy models on CUDA hardware. - vRAM-Aware Fallback: Intelligent overflow handling that redirects traffic when a specific GPU's memory is saturated.
- Context Protection: Intelligent routing that prioritizes nodes already hosting the requested model to avoid reload overhead.
- Request Tagging: Support for granular usage tracking across the cluster for per-project resource accounting.
SKILL.md
Loading
Related Openclaw Skills
Linux AI Server
Transform your Linux hardware into a high-performance, headless AI inference cluster using native systemd and NVIDIA CUDA.
linux-ollama
A sophisticated fleet routing and load balancing solution for running Ollama across multiple Linux machines through a single unified endpoint.
RTX Local AI
RTX Local AI turns your gaming PC into a high-performance local AI server, allowing you to run massive LLMs with zero cloud costs using NVIDIA RTX GPUs.
Windows Ollama Fleet Manager
An advanced orchestration tool for routing Ollama inference requests across a network of Windows PCs to maximize hardware utilization.
Qdrant Ingestion Best Practices
A production-grade framework for designing and implementing high-performance RAG pipelines using Qdrant as a vector database.
n8n + Qdrant: Ingestion & RAG Pipeline Skill
A comprehensive framework for building production-grade vector ingestion and Retrieval-Augmented Generation pipelines using n8n and Qdrant.
Featured*
