A streamlined utility for removing duplicate records from CSV files based on specific key columns or entire rows.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install csv-deduper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install csv-deduper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is CSV Deduper?
CSV Deduper is a robust technical utility designed for developers and data analysts who need to maintain data integrity within large datasets. As a featured part of the Openclaw Skills library, it allows users to programmatically identify and remove redundant information from CSV files, ensuring that only unique records remain for downstream processing.
By default, the skill retains the first instance of a duplicate record and discards subsequent matches. Whether you are cleaning user lists, merging datasets, or preparing data for machine learning, this tool provides a predictable and safe environment for data sanitization without requiring external network dependencies.
CSV Deduper Use Cases
- Cleaning marketing lists to ensure unique email addresses or customer IDs.
- Merging multiple CSV exports while removing overlapping records.
- Reducing file size and processing overhead for large-scale data analytics.
- Preparing high-quality datasets for AI training as part of the Openclaw Skills ecosystem.
How CSV Deduper Works
- The user specifies the input CSV file path and identifies which columns should serve as unique identifiers.
- The skill parses the CSV content and hashes the values in the specified key columns (or the entire row if no keys are provided).
- It iterates through the file, keeping the first occurrence of each unique hash and flagging duplicates.
- A new deduplicated CSV file is generated at the specified output path.
- The tool provides validation by comparing the original and final row counts.
CSV Deduper Setup
To integrate this skill into your workflow, ensure you have the necessary scripts within your local environment. Run the following commands in your terminal:
# Basic deduplication using all columns as the key
python scripts/csv_dedupe.py --input data.csv --output data.deduped.csv
# Deduplication based on specific key columns (e.g., id and email)
python scripts/csv_dedupe.py --input data.csv --output data.deduped.csv --keys id,email
CSV Deduper Data Schema & Taxonomy
The CSV Deduper handles data through a straightforward input-output model, maintaining the original file's schema while altering row counts.
| Attribute | Description |
|---|---|
| Input Format | Standard CSV (.csv) |
| Output Format | Cleaned CSV (.csv) |
| Key Selection | String (comma-separated list of column headers) |
| Logic | First-In-Wins (remains first record, drops subsequent duplicates) |
| Security | Local execution; no external API calls or network access |
CSV Deduper Advanced Features
- Multi-column key support for complex unique identifiers.
- Automated row count validation to confirm the number of records removed.
- Full row hashing for quick deduplication without manual column mapping.
- Designed for seamless integration with other Openclaw Skills in automated data pipelines.
SKILL.md
Loading
Related Openclaw Skills
Ecommerce Platform Policy Safe Rewrite
A specialized skill for converting aggressive marketing claims into policy-compliant copy that maintains its persuasive commercial edge.
QBO to Tax Bridge
A professional-grade accounting skill that bridges the gap between QuickBooks Online bookkeeping and IRS tax preparation by automating schedule mapping and workpaper generation.
GetNote
GetNote is an advanced AI-integrated personal knowledge management tool designed to capture, organize, and retrieve information through intelligent automation.
AI Intelligent Expense Management
A comprehensive AI-driven solution for corporate expense management, budget control, and automated financial analysis within the Openclaw Skills ecosystem.
AI Commentary
An advanced video processing skill that automates the creation of narrated, commentary-style edits and explainer videos.
Poem Reference Tool
A zero-dependency, local-only reference tool providing instant technical documentation for the Poem ecosystem.
Featured*
