Openclaw Skills for generating, running, and grading reproducible puzzle benchmarks that verify LLM and agent reasoning locally.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install puzzletide-agent-evals
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install puzzletide-agent-evals using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is PuzzleTide Agent Evals?
This Openclaw Skills package turns puzzle logic into a deterministic evaluation harness for LLMs and agents. Using the local PuzzleTide CLI, you can generate sudoku or word-search task sets, send them to a subject model, and score responses by the puzzle rules instead of by a trusted judge model.
Because each benchmark is defined by the tuple (type, difficulty, n, seed), every run is reproducible across machines and sessions. That makes it ideal for regression testing, comparative model analysis, and offline evaluation workflows where objective scoring matters more than subjective interpretation.
PuzzleTide Agent Evals Use Cases
- Benchmark reasoning quality in LLMs and agents with objective, rule-based scoring.
- Build repeatable regression tests in Openclaw Skills for prompt, tool, or model changes.
- Compare multiple agents against the same deterministic task set.
- Generate offline evaluations for privacy-sensitive or air-gapped environments.
- Validate sudoku and word-search workflows without relying on a judge model or answer key.
How PuzzleTide Agent Evals Works
- Detect the local CLI in priority order:
ptide,puzzletide, thennpx puzzletide. - Generate a benchmark with
ptide eval generate, choosing the puzzle type, difficulty, task count, seed, and output file. - Consume
tasks.json; each task includesid,instructions, and the puzzle payload needed by the subject model. - Send each task to the model under test and collect responses into
answers.jsonas{id, answer}pairs. - Grade locally with
ptide eval check --tasks tasks.json --answers answers.json --json. - Re-run the exact same benchmark later by reusing the same
(type, difficulty, n, seed)tuple for Openclaw Skills regression tracking.
PuzzleTide Agent Evals Setup
- Ensure Node.js and npm are available on the machine.
- Check whether the CLI is already installed:
ptide --version
puzzletide --version
npx puzzletide --version
- If none of the commands work, ask the user before installing globally:
npm install -g puzzletide
- Verify the install and confirm the commands are on
PATH. - No API key, account, or network access is required; everything runs locally.
- Optional sanity check:
ptide eval generate --help
ptide eval check --help
PuzzleTide Agent Evals Data Schema & Taxonomy
| Artifact | Shape | Description |
|---|---|---|
tasks.json |
Array of task objects | Generated benchmark set used for model runs and grading |
task.id |
String | Stable identifier for matching model outputs |
task.instructions |
String | Human-readable prompt shown to the subject model |
task.puzzle |
String | Sudoku payload; 81 chars, . means empty |
task.grid |
String[] | Word-search rows |
task.words |
String[] | Target words to locate |
answers.json |
Array of {id, answer} |
Collected model outputs for grading |
answer for sudoku |
String | Completed 81-char solution |
answer for wordsearch |
Array of objects | {word, startRow, startCol, endRow, endCol} with 0-indexed coordinates |
Benchmark identity
- Deterministic benchmark key:
(type, difficulty, n, seed) - Supported task families:
sudoku,wordsearch - Grading mode: local, deterministic, and rule-based
Openclaw metadata taxonomy
metadata.openclaw.tags:evals,benchmarks,testing,puzzles,puzzletide,climetadata.openclaw.requires.anyBins:ptideorpuzzletidemetadata.openclaw.install: Node packagepuzzletidewith CLI binsptideandpuzzletide
PuzzleTide Agent Evals Advanced Features
- Deterministic task generation from a single parameter tuple for exact reruns.
- Local, offline grading with no network dependency and no API keys.
- Rule-based verification, so results do not depend on a judge model.
- Machine-readable JSON output from
--jsonfor CI and automated analysis. - Flexible CLI discovery via
ptide,puzzletide, ornpx puzzletide. - Supports multiple puzzle types and scalable benchmark sizes for smoke tests or larger eval suites.
SKILL.md
Loading
Related Openclaw Skills
Stock Analysis Expert
Openclaw Skills' Stock Analysis Expert turns A-share market data into a nine-layer investment report with scores, signals, and risk controls.
Grok Build
Openclaw Skills for Grok Build turns xAI's terminal AI coding agent into a full-screen TUI for interactive coding, file edits, shell automation, search, and headless workflows.
Apple Design
Openclaw Skills for turning Apple’s fluid interface principles into immediate, gesture-first web UI.
X (Twitter) Publisher
A command-line Openclaw Skills publisher for X that posts text, media, replies, and threads through Tweepy with structured results.
PuzzleTide Crossword
PuzzleTide Crossword helps Openclaw Skills generate valid, numbered crossword grids locally from custom clues, themes, or word banks.
Craft MCP v2 Integration
An AI coding agent skill to create, read, and structure documents, tasks, and whiteboards via the unified Craft MCP v2 API.
Featured*
