Sitemap URL and Link Extractor for Google Sheets - n8n Workflow

• SEO Specialists needing to audit and index links from large, complex website structures.


• Web Developers or Technical Content Writers looking for robust web scraping n8n templates.


• Users requiring an automated, recursive n8n solution for decompressing and parsing sitemap index files.


• Anyone who needs a comprehensive list of all URLs indexed by a domain, delivered directly by an n8n node automation.

This sophisticated n8n workflow provides a reliable solution for extracting every unique URL contained within a potentially complex, multi-layered sitemap index. Initiated by an external n8n trigger (the Webhook), the automation requires only a domain name to begin its work. It intelligently starts by checking the robots.txt file for explicit sitemap declarations, ensuring maximum link discovery. The n8n workflow is engineered to handle common real-world challenges, such as dealing with binary file responses, decompressing .xml.gz files, and recursively following nested sitemap links using the ScrapingBee n8n node. Custom Code n8n nodes are used to filter and separate the discovered links: sitemap links are fed back into the scraper, while final webpage links are outputted to the target Google Sheet. This makes the n8n workflow an essential tool for comprehensive technical SEO audits.

This recursive n8n workflow template is executed through the following logical sequence:

1. Initiation (n8n Trigger): The workflow begins when the Webhook n8n trigger receives a request containing the target domain (e.g., ?domain=example.com).


2. Robots.txt Check: The first ScrapingBee n8n node scrapes the domain's /robots.txt file to find any declared sitemap links.


3. Sitemap Logic: An If n8n node checks the content of robots.txt. If it contains 'Sitemap:', it proceeds to extract those XML links. If not, the n8n workflow falls back and scrapes the standard /sitemap.xml file.


4. Binary and Compression Handling: If the resulting XML file is returned as a binary (which often happens with large files or compressed files), another If n8n node checks if the file extension is .gz. If it is a .gz file, the Compression n8n node decompresses it, and a subsequent Code n8n node prepares the data for parsing.


5. Parsing XML: The Extract From File n8n node converts the raw XML content into a standard JSON structure, making the URL data accessible.


6. Dual Extraction: The flow splits into two custom Code n8n nodes:

1. Final Output: The Google Sheets n8n node securely appends all unique, non-XML links gathered by the entire n8n workflow to the specified spreadsheet.

To deploy this powerful n8n workflow template, follow these steps:

1. Import: Copy the provided JSON code and import it directly into your n8n instance.


2. ScrapingBee Credentials: You must provide credentials for the ScrapingBee n8n node. This API key is required to reliably fetch and process the robots.txt and sitemap files without being blocked.


3. Google Sheets Credentials: Configure the Google Sheets n8n node ('Append links to sheet') with your OAuth2 credentials. Ensure the account has write access to the target spreadsheet.


4. Configure Sheet: Specify the exact Google Sheet Document ID and Sheet Name where you wish to append the links. Crucially, the target sheet must contain a column named links as this is the field the n8n workflow uses for mapping.


5. Activate and Test: Activate the n8n workflow. The Webhook n8n trigger is now ready to receive inputs. Test the setup by sending a request like https://n8nurl>/webhook/1da30868-fbca-4e8e-8580-485afb3fd956?domain=n8n.io to start the link extraction process.

Webhook (Domain to scrape): This is the primary n8n trigger. It listens for HTTP requests and extracts the target domain from the query parameters to initiate the recursive scraping n8n workflow.
ScrapingBee (Scrape robots.txt file / Scrape sitemap.xml file / Scrape xml file): A crucial n8n node used throughout the workflow to reliably fetch the contents of robots.txt and various sitemap XML files. It handles the requests needed for this complex n8n automation.
If n8n node (If it's a binary file / If it's a .gz file / If sitemap links are found): These flow control n8n nodes manage the execution path based on data structure (text vs. binary) and file type (.gz vs. .xml), ensuring the correct processing steps are applied.
Compression (Decompress .gz file): This n8n node is essential for handling sitemaps that have been gzipped (.gz extension), decompressing them so the content can be read by the parser.
Extract From File (Load the xml file as JSON): Converts the raw XML data obtained from the ScrapingBee n8n node into a standard, accessible JSON format for easier manipulation in subsequent steps of the n8n workflow.
Code n8n node (Extract non-xml links / Extract xml links): These powerful custom n8n nodes use JavaScript logic to precisely filter the extracted URLs. One path pulls out the final webpage links, while the other pulls out nested sitemap links for recursive iteration.

• Google Sheets (Append links to sheet): The final destination n8n node. It takes the list of non-XML links and appends them to the configured spreadsheet, completing the n8n workflow's goal.