A sophisticated utility for cleaning, merging, and consolidating redundant information in AI agent memory files to improve retrieval speed and accuracy.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install memory-dedup
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install memory-dedup using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Memory Deduplication?
Memory Deduplication is a specialized tool designed to maintain the integrity and efficiency of the MEMORY.md file, which serves as the long-term storage for AI agents. Over time, these files often accumulate duplicate entries, outdated task statuses, and fragmented project descriptions that can clutter the agent's context and degrade performance. By integrating this into your Openclaw Skills workflow, you ensure your agent operates with a lean, high-density knowledge base.
The skill employs advanced text similarity algorithms to automatically identify overlapping content, allowing it to consolidate similar records into a single, comprehensive entry. This process not only reduces the physical size of memory files but also significantly enhances the agent's ability to retrieve relevant information quickly and accurately without being distracted by redundant noise.
Memory Deduplication Use Cases
- Eliminating duplicate logs where the same event was recorded multiple times during different sessions.
- Consolidating project details that have been fragmented across multiple headers in a memory file.
- Cleaning up outdated task statuses, such as marking a task as completed once a newer record confirms its success.
- Optimizing the context window for LLMs by reducing the token count of the memory file through intelligent information compression.
How Memory Deduplication Works
- The skill scans the targeted MEMORY.md file to parse and categorize all existing entries and sections.
- It utilizes a Jaccard similarity algorithm to calculate the textual overlap between different memory items.
- Based on predefined thresholds, items with a similarity score higher than 0.8 are flagged as absolute duplicates for deletion.
- Items with a similarity score between 0.5 and 0.8 are processed through a merge strategy that preserves unique details and the latest timestamps.
- The system reorganizes the consolidated data, ensuring that high-priority markers (like [P0]) are maintained and that the hierarchy remains logical.
- A detailed report is outputted to the console, and the optimized memory file is saved, often creating a timestamped backup for safety.
Memory Deduplication Setup
To deploy this skill within your environment, navigate to your workspace and execute the deduplication script. You can also set up a recurring schedule to keep your memory clean automatically.
# Run the deduplication process immediately
node skills/memory-dedup/dedup.mjs
# Preview changes without modifying the actual MEMORY.md file
node skills/memory-dedup/dedup.mjs --dry-run
# Run deduplication with an automatic backup of the original file
node skills/memory-dedup/dedup.mjs --backup
To automate this as part of your Openclaw Skills maintenance routine, add a cron job:
openclaw cron add --name "memory-dedup-weekly" \
--cron "0 2 * * 0" --tz "Asia/Shanghai" \
--session isolated --agent main \
--message "Running memory deduplication to optimize MEMORY.md and remove redundancy"
Memory Deduplication Data Schema & Taxonomy
The skill organizes its operations around the following data structure to ensure transparency and safety:
| Component | Description | Data Type |
|---|---|---|
| Source File | The primary MEMORY.md file used by the agent for long-term storage. | Markdown |
| Similarity Metric | Jaccard similarity coefficient used for text comparison. | Float (0.0 - 1.0) |
| Backup Storage | Directory containing the last 10 versions of the memory file. | Markdown Files |
| Deduplication Report | Statistics including original vs. final item counts and merge summaries. | CLI Output |
| Whitelist | A list of protected keywords or sections that the deduplicator will ignore. | Configuration Array |
Memory Deduplication Advanced Features
- Intelligent Merge Strategy: Automatically prioritizes the most recent information and preserves unique metadata from multiple similar sources into one unified entry.
- Multi-Version Rollback: Maintains a rolling history of the last 10 memory states, allowing for easy recovery if critical information is accidentally consolidated.
- High-Performance Calculations: Efficiently processes large memory files by utilizing optimized string tokenization for similarity scoring.
- Customizable Thresholds: Allows power users to adjust similarity sensitivity levels to match specific project documentation styles within the Openclaw Skills framework.
SKILL.md
Loading
Related Openclaw Skills
Meme Signal Scanner
A real-time aggregator and scoring engine for meme coin signals across major decentralized exchanges and launchpads.
Cross-Agent Memory Sharing Protocol
A standardized protocol for enabling collective intelligence by allowing multiple AI agents to share, sync, and inherit knowledge seamlessly.
Causal Graph Auto-Builder
An automated tool for extracting entities, events, and causal relationships from logs and memory files to build dynamic knowledge graphs.
Agent ROS Bridge
A secure, universal communication bridge that enables AI agents to control ROS1 and ROS2 robotic systems with ease.
Memory Health Score
A comprehensive diagnostic utility that evaluates the integrity, freshness, and structure of an AI agent's memory system on a 100-point scale.
Jira
A comprehensive integration for managing Jira Cloud issues, transitions, and worklogs through a CLI-based interface.
Featured*
