Robust Sitemap Crawler and Content Extractor to Google Drive - n8n Workflow

• SEO Analysts needing to archive website content for audits.


• Content Managers performing bulk website migrations or backups.


• Data Scientists requiring reliable web content extraction into structured files.


• Users looking for a robust, rate-limited n8n workflow for scraping large sites.

This comprehensive n8n workflow provides a reliable solution for extracting content from large websites by leveraging their existing sitemaps. Traditional scraping methods can be unstable, but this n8n template utilizes direct HTTP requests with specific user agents and includes critical features like robust error handling and a mandatory 3-second delay implemented via an n8n node, ensuring compliance and reducing the risk of being blocked. After fetching the sitemap and iterating through every URL, a custom n8n node transforms the raw HTML into clean, readable Markdown, including detailed metadata (word count, status code, framework detection). The final output is an organized .md file saved immediately to a designated Google Drive folder. This specific n8n workflow guarantees that content archiving is efficient and well-structured.

The process begins with the manual n8n trigger, Start Working Scraper.

1. Set Initial Parameters: The Set Sitemap URL n8n node defines the target sitemap XML URL (https://yourwebsitehere.com/page-sitemap.xml).


2. Fetch Sitemap: The Get Sitemap HTTP Request n8n node fetches the XML content of the sitemap.


3. Process URLs: The content is passed to the Parse Sitemap XML n8n node, which converts the structure into a usable JSON array. The Split Sitemap URLs n8n node then creates a separate item for every URL listed in the sitemap, enabling parallel or sequential processing.


4. Iterate and Scrape: The Loop URLs n8n node initiates iteration. For each URL, the Scrape Page n8n node executes an optimized HTTP request, using specific headers for reliable retrieval.


5. Extract and Clean: The Extract Content Code n8n node is vital. It safely handles the response, extracts the page title, strips HTML, JavaScript, and CSS, and converts the primary text into clean Markdown. It also records metadata like word count and framework indicators (e.g., WordPress or Divi).


6. Format Output: The Format Content Code n8n node constructs a standardized Markdown file structure. This includes a YAML frontmatter section with all the collected metadata and a clean body section containing the extracted content.


7. Save: The Save to Google Drive n8n node takes the formatted Markdown text and creates a new file in the specified Google Drive folder, naming the file based on the scraped URL.


8. Wait: Finally, the Wait Between Pages n8n node imposes a 3-second delay before the loop returns to process the next URL, ensuring rate limit compliance for this robust n8n workflow.

1. Import: Copy the entire n8n workflow JSON and import it into your n8n instance.


2. Google Drive Credentials: Click on the Save to Google Drive n8n node and set up or select your Google Drive OAuth2 API credentials.


3. Set Target URL: Go to the Set Sitemap URL n8n node and replace https://yourwebsitehere.com/page-sitemap.xml with the actual URL of the sitemap you wish to crawl.


4. Set Output Folder: In the Save to Google Drive n8n node, replace the placeholder YOURGOOGLEDRIVEFOLDERID_HERE with the actual ID of the Google Drive folder where you want the Markdown files saved.


5. Optional Limit: If testing, you can enable the Limit URLs (Optional) n8n node and set a low maximum item count (e.g., 5).


6. Execution: Execute the n8n workflow manually using the Start Working Scraper n8n trigger.

Start Working Scraper (Manual Trigger): The starting n8n trigger for the entire operation. This allows for manual execution of the n8n workflow.
Set Sitemap URL (Set n8n node): Defines the sitemap_url variable used by subsequent HTTP requests.
Get Sitemap (HTTP Request n8n node): Fetches the raw XML content from the defined sitemap URL. Configured with a 15-second timeout and neverError: true for stability.
Parse Sitemap XML (XML n8n node): Converts the received XML document into a structured JSON object for easy manipulation.
Split Sitemap URLs (Split Out n8n node): Takes the array of URL entries from the parsed XML and generates a separate item for each page URL, enabling loop iteration in the n8n workflow.
Scrape Page (HTTP Request n8n node): Executes the actual scrape of the individual web page specified by the loop item ($json.loc). It sends custom User-Agent headers to mimic browser behavior and has a 20-second timeout.
Extract Content (Code n8n node): Contains a JavaScript function to clean and structure the HTML response. It extracts the title, sanitizes the content by removing scripts/styles, converts remaining HTML to readable text, and calculates metrics like word count.
Format Content (Code n8n node): Structures the extracted data into a final, highly readable Markdown file, complete with YAML frontmatter containing all scraping metadata (status code, URL, timestamps).
Save to Google Drive (Google Drive n8n node): Creates a new file from the Markdown output generated in the previous step. The filename is dynamically generated based on the URL.
Wait Between Pages (Wait n8n node): Inserts a 3-second pause between processing each page request, making this n8n workflow respectful of the target server's resources.