A high-performance web scraping agent that extracts structured data from any webpage using AI-driven schema mapping.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install data-spider
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install data-spider using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Data Spider?
Data Spider is a versatile web scraping agent designed for the Openclaw Skills ecosystem. It allows developers and researchers to transform unstructured web content into clean, structured data formats like JSON, tables, or flat lists. By leveraging advanced AI models, it can intelligently map page elements to a user-defined schema, ensuring the output perfectly matches your application's data model.
Whether you are building a competitive analysis engine or a specialized dataset, this tool simplifies the process of data acquisition. As part of the Openclaw Skills library, it offers a secure and efficient way to interact with web content without writing complex parsing logic for every new site.
Data Spider Use Cases
- Extracting product specifications and pricing details from e-commerce sites
- Collecting statistical data and figures from research articles or news reports
- Building comprehensive datasets from multiple web sources for machine learning
- Automating competitive analysis by monitoring competitor feature lists and pricing
- Schema-guided extraction to ensure scraped data fits into existing databases
How Data Spider Works
- The user provides a target webpage URL and specifies the desired output format (JSON, table, or list).
- Optionally, a JSON schema is provided to guide the AI on exactly what fields to extract and how to name them.
- The request is routed through the AIProx orchestration endpoint for processing.
- The agent fetches the page content, respects rate limits, and analyzes the DOM structure.
- Structured data is returned along with a summary of the extraction and the original source URL.
Data Spider Setup
To use this skill within the Openclaw Skills framework, you must configure your authentication token.
- Obtain an AIPROX_SPEND_TOKEN from your provider dashboard.
- Set the environment variable in your terminal:
export AIPROX_SPEND_TOKEN="your_spend_token_here"
- You can then trigger the skill via a standard API request:
curl -X POST https://aiprox.dev/api/orchestrate \
-H "Content-Type: application/json" \
-H "X-Spend-Token: $AIPROX_SPEND_TOKEN" \
-d '{
"url": "https://example.com/pricing",
"format": "json"
}'
Data Spider Data Schema & Taxonomy
Data Spider organizes output based on the requested format. It returns a standard response object containing the following fields:
| Field | Type | Description |
|---|---|---|
| data/items/rows | Mixed | The primary extracted content (JSON object, list, or array of arrays) |
| columns | Array | Present only in table format; contains the column headers |
| summary | String | A brief AI-generated summary of the page content |
| source | String | The URL from which the data was extracted |
| format | String | The output format used (json, table, or list) |
Data Spider Advanced Features
- Schema-Guided Extraction: Provide a custom object template to force the AI to return data in a specific shape that matches your backend requirements.
- Multi-Format Support: Seamlessly toggle between JSON objects for API integration, tables for spreadsheets, or flat lists for quick textual summaries.
- Automated Summarization: Every extraction includes a natural language summary of the source page to provide context to the data.
- Ethical Scraping: Designed to operate ethically within Openclaw Skills by respecting robots.txt and implementing transient processing without long-term content storage.
SKILL.md
Loading
Related Openclaw Skills
JournalBot: Daily AI Journaling
A private, local AI-driven journaling assistant that provides smart prompts and analyzes emotional patterns without cloud dependency.
Copywriting
An expert conversion copywriting skill designed to produce clear, compelling, and action-oriented marketing copy for web pages and product offers.
Structured Dev: Research-First Coding Pipeline
A structured AI development pipeline that enforces architectural review before a single line of code is written.
Music Generation via Crazyrouter
A powerful AI music generation skill that leverages Suno AI via the Crazyrouter API to create full-length songs from text descriptions.
Publish Checklist
A rigorous pre-release review protocol designed to prevent data leaks and ensure security when publishing code, PRs, or public content.
Vision Bot
An intelligent image analysis tool for Openclaw Skills that performs OCR, object detection, and detailed visual descriptions in any language.
Featured*
