PulpMiner is an AI-powered tool that converts any webpage into clean, structured JSON data using custom schemas and LLM-driven extraction.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install webscraper-pulpminer
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install webscraper-pulpminer using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is PulpMiner?
PulpMiner bridges the gap between raw web content and structured data by leveraging Large Language Models to parse and organize website information. It allows developers to define custom JSON templates, use CSS selectors for precision, and handle JavaScript-heavy Single Page Applications (SPAs) with ease. As a core component within the Openclaw Skills ecosystem, it simplifies the process of building data-driven applications without the need for writing and maintaining complex custom scraping scripts for every target site.
By providing a URL and a desired output format, PulpMiner handles the heavy lifting of fetching, rendering, and interpreting web content. This tool is designed to provide high-fidelity data that is ready for programmatic consumption, making it an essential utility for modern data engineering and AI agent workflows.
PulpMiner Use Cases
- Automating lead generation by extracting contact information and business details from directories.
- Monitoring competitor pricing and product availability across e-commerce platforms.
- Building news aggregators that convert disparate articles into a unified structured feed.
- Powering AI agents with real-time web data to provide up-to-date answers and research.
- Creating data pipelines for financial analysis by scraping market reports and economic indicators.
How PulpMiner Works
- Create and configure a saved API in the PulpMiner dashboard with a target URL and optional JSON template.
- Set extraction parameters such as CSS selectors, JavaScript rendering requirements, and specific AI instructions.
- Authenticate your request using the provided API key in the request header.
- Call the generated endpoint using a GET request for static URLs or a POST request for dynamic URLs with variables.
- The PulpMiner engine scrapes the page, processes it through an LLM, and returns the data in your specified JSON format.
PulpMiner Setup
To integrate this tool into your Openclaw Skills environment, follow these steps:
- Sign up and obtain your API key from the PulpMiner API Dashboard.
- Configure your environment variable:
export PULPMINER_API_KEY='your_api_key_here'
- Test the connection with a simple curl command:
curl -X GET "https://api.pulpminer.com/external/<apiId>" -H "apikey: $PULPMINER_API_KEY"
PulpMiner Data Schema & Taxonomy
PulpMiner provides a consistent response structure regardless of the target website's complexity. Users define the internal structure of the data object using the JSON Template feature.
| Property | Type | Description |
|---|---|---|
data |
Object | The primary payload containing extracted fields defined in your template. |
errors |
String/Null | Contains error details if the extraction fails; null on success. |
Example Schema for a product page:
product_name: Stringprice: Numberavailability: Booleanspecs: Array of Strings
PulpMiner Advanced Features
- Dynamic URL Templating: Use
{{variable}}syntax to handle searches, pagination, and parameterized queries programmatically. - Intelligent Caching: Built-in 24-hour caching with 15-minute background refreshes to balance data freshness and credit efficiency.
- Headless Browser Rendering: Optional JavaScript execution for scraping modern frameworks like React, Vue, and Angular.
- Multi-Platform Integrations: Dedicated webhook support and endpoints for seamless use with Zapier and n8n workflows.
- AI Guidance: Pass extra instructions to the LLM to filter data or format specific fields (e.g., converting dates to ISO 8601).
SKILL.md
Loading
Related Openclaw Skills
Forge
Forge is a repair-inspect loop orchestrator that automates code fixes with independent verification and dependency-aware parallel execution.
MoltShell Vision Engine
A sophisticated visual cortex for text-based agents that enables image analysis and description via the MoltShell M2M Marketplace.
PurposeBot
PurposeBot is a comprehensive agentic commerce layer that enables AI agents to discover tools, execute payments via Stripe or USDC, and build verifiable reputation.
Radarr and Sonarr Integration
A powerful integration for OpenClaw that automates movie and TV series management via Radarr and Sonarr servers.
Web Navigator
A high-performance browser automation skill that enables AI agents to navigate, scrape, and interact with the live web.
Agile CRM
A powerful integration for Agile CRM that enables developers to manage sales pipelines, contacts, and tasks through a unified CLI and API interface.
Featured*
