A robust webpage extraction skill designed for cross-model reliability using a tiered approach from lightweight HTML fetching to full browser rendering.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install page-fetch
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install page-fetch using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Page Fetch?
Page Fetch is a sophisticated tool within the Openclaw Skills ecosystem designed to solve the common problem of inconsistent web scraping across different LLMs. It prioritizes a stable, low-dependency workflow that ensures developers get high-quality, readable content from articles, documentation, and blog posts without unnecessary overhead.
By implementing a tiered extraction strategy, it first attempts lightweight deterministic fetches and inspects embedded data like JSON-LD or Next.js payloads. It only escalates to heavy browser rendering via Playwright when lightweight methods fail, ensuring cost-efficiency and speed while maintaining maximum reliability for your AI agents.
Page Fetch Use Cases
- Summarizing news articles or blog posts for quick consumption.
- Translating technical documentation into different languages.
- Verifying facts or checking for specific mentions on public webpages.
- Extracting structured data from WeChat public articles using dedicated extractors.
- Quoting specific sections from product pages for competitive analysis.
How Page Fetch Works
- URL Routing: The skill analyzes the target URL to determine the best starting point, such as routing WeChat links to a specialized extractor.
- Lightweight Fetch: It attempts to retrieve raw HTML using standard requests to minimize latency and resource usage.
- Data Inspection: The system inspects the HTML for embedded metadata, JSON-LD, or script variables like
__NEXT_DATA__to find the core content. - Browser Escalation: If the initial fetch results in thin or incomplete content, the skill triggers a headless Chromium instance via Playwright to render the full DOM.
- Standardized Reporting: The final output is returned in a structured format, clearly stating the extraction method used and any access limitations encountered.
Page Fetch Setup
To integrate this into your workflow, ensure you have the necessary dependencies installed for both the Python scripts and the optional browser fallback:
# Install Python dependencies
pip install requests beautifulsoup4
# For browser rendering fallback (Optional)
npm install playwright
npx playwright install chromium
Page Fetch Data Schema & Taxonomy
The skill returns a structured JSON response containing the following metadata and content fields:
| Field | Description |
|---|---|
title |
The extracted page title |
author |
The identified content author |
published_time |
Date and time the content was published |
text |
The primary body text of the page |
method |
The specific extraction path used (e.g., json-ld, browser-render) |
access_limited |
Boolean indicating if content was blocked by anti-bot or login walls |
Page Fetch Advanced Features
- WeChat Optimization: Specialized handling for
mp.weixin.qq.comusing mobile-style headers and specific DOM selectors. - Smart Persistence: By default, no files are written to disk, preventing workspace clutter unless
--save-jsonis explicitly called. - Traceability: Every response includes a
notesfield with a runner step trace to debug extraction logic. - Deterministic Fallbacks: Automatically moves through a hierarchy of extraction methods to find the highest-quality text possible.
SKILL.md
Loading
Related Openclaw Skills
AgentID MCP
Connect Claude Code to AgentID for persistent shared memory, live activity reporting, and seamless multi-agent mission coordination.
AI-Resume-Judge
An AI-driven resume evaluation system that provides quantitative scoring, cross-industry equivalence mapping, and actionable improvement suggestions.
Document Learning System
A comprehensive document processing skill that tracks learning progress across multiple sessions while building a persistent long-term knowledge base.
Atomic RAG Knowledge Base
An advanced knowledge base builder that deconstructs PDF documents into atomic, logically complete units for high-precision Retrieval-Augmented Generation.
ShowMeTheMoney Lite
A specialized demonstration skill designed to showcase seamless StablePay integration and in-chat payment flows for AI agents.
Gold Price Query
A real-time spot gold price lookup tool for tracking market values in both CNY (RMB/gram) and USD (USD/oz).
Featured*
