A professional benchmarking utility to measure and compare token generation speed, latency, and throughput across multiple LLM API providers.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install api-benchmark
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install api-benchmark using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is API Token Speed Benchmark?
The API Token Speed Benchmark is a specialized performance monitoring tool designed to evaluate the efficiency of Large Language Model (LLM) endpoints. As a key component in the ecosystem of Openclaw Skills, it provides developers with precise metrics on Time To First Token (TTFT), tokens-per-second (TPS), and total request duration. By supporting multiple API formats like Anthropic and OpenAI, it allows for objective comparisons between different infrastructure providers.
This skill is essential for developers who need to optimize their AI agents and ensure that the models used within their Openclaw Skills implementations meet strict performance requirements. Whether you are testing local inference engines or global cloud providers, this tool provides the data needed to minimize latency and maximize throughput in production environments.
API Token Speed Benchmark Use Cases
- Compare real-world performance differences between providers like Anthropic and OpenAI for specific model tiers.
- Optimize user experience by identifying which models provide the lowest Time To First Token (TTFT) for interactive applications.
- Verify the stability and connectivity of API endpoints before deploying complex Openclaw Skills workflows.
- Audit provider throughput (TPS) to ensure that high-volume generation tasks are handled efficiently without bottlenecks.
How API Token Speed Benchmark Works
- The skill initializes by parsing the provider definitions located in your local configuration file (~/.openclaw/openclaw.json).
- Users select a target provider or model to benchmark via the command line interface.
- The tool sends a series of requests based on defined prompt categories (short, medium, or long) to simulate various workloads.
- It monitors the streaming response in real-time to capture the exact moment the first token arrives and tracks the subsequent stream to calculate tokens per second.
- Results are aggregated and presented as a detailed report, allowing users to analyze the performance within the broader context of their Openclaw Skills setup.
API Token Speed Benchmark Setup
To get started, ensure you have Python 3 and the requests library installed. You must also have a configured openclaw.json file in your home directory.
# Install dependencies
pip install requests
# List available benchmark targets defined in your config
python3 main.py --targets
# Run a preflight check to verify connectivity
python3 main.py check --all
API Token Speed Benchmark Data Schema & Taxonomy
The API Token Speed Benchmark organizes its findings into structured metrics that can be output as JSON or formatted tables. The schema focuses on latency and throughput data.
| Metric | Description |
|---|---|
| TTFT | Time To First Token (seconds) - measures initial response latency. |
| TPS | Tokens Per Second - measures the generation speed once started. |
| Total Time | The complete duration from request initiation to final token delivery. |
| Token Count | The number of input and output tokens, sourced from the API or estimated based on character counts. |
API Token Speed Benchmark Advanced Features
- Multi-target testing: Benchmark all configured providers simultaneously to generate a comprehensive performance leaderboard for Openclaw Skills.
- Security-first configuration: Support for environment variable placeholders ensures that sensitive API keys are never hardcoded in configuration files.
- Prompt Category selection: Test models against different load sizes (short, medium, long) to see how scaling affects provider performance.
- Quiet Mode: Suppress progress output for clean integration into automated CI/CD pipelines or larger Openclaw Skills automation scripts.
- Custom Repeat Cycles: Run multiple iterations per model to eliminate statistical outliers and get an accurate average of provider speed.
SKILL.md
Loading
Related Openclaw Skills
Lyric Translator: Indonesian to English
A specialized AI agent skill designed to translate Indonesian song lyrics into authentic, singable, and human-sounding English for international music production.
Fitbit Fitness Insights
A powerful AI integration for Fitbit that provides deep insights into your physical activity, sleep patterns, and heart rate data.
Hypha Payment
A decentralized protocol for P2P agent coordination and USDT settlement on the Base L2 network.
CreatorDB YouTube Creator Insights
A powerful data integration for searching and analyzing YouTuber statistics, performance metrics, and audience demographics.
Quests
A standardized framework for AI agents to track and guide humans through complex, multi-day real-world processes.
ClawShell
ClawShell provides a robust security layer for Openclaw Skills by intercepting shell commands and requiring manual human approval for high-risk operations.
Featured*
