Session Learning with Held-Out Validation
Turn past Claude sessions into memory and skill updates validated against untouched tasks before installation.
Before & after
Same prompt, same agent — the only difference is the skill.
That's an interesting idea. Over time, you could track common patterns in your interactions and periodically review them to see what kinds of questions come up most often. You might manually consolidate lessons learned into your agent's instructions, or build a simple logging system that captures recurring themes. You'd probably want to set up a regular checkpoint where you read through past conversations and extract useful insights. It would help to have some way of measuring whether the improvements actually work. The tricky part would be making sure updates don't accidentally break things that were already working well. Would you like some suggestions on how to structure this kind of manual review process?
SKILLOPT SLEEP WORKFLOW HARVEST → Runs after each session → Scans conversation history for repeated requests, stuck points, and workarounds → Tags interactions by type (clarification needed, better examples needed, boundary issue, repetitive pattern) MINE → Groups tagged interactions into clusters → Identifies the top 3-5 recurring gaps → Extracts concrete examples from real conversations REPLAY (OFFLINE) → Runs outside your active sessions → Tests each proposed instruction change against held-out conversation samples → Validates that the change fixes the gap without breaking existing behavior → Measures improvement: does this reduce the number of clarifying questions? CONSOLIDATE → Merges validated changes into your CLAUDE.md and SKILL.md → Updates are gated: you review diffs before they take effect → Original instructions remain in version control for rollback SCHEDULE Runs nightly by default; you control frequency via settings.
About this skill
Your agent keeps making the same mistake, even after you correct it. This workflow turns past sessions into small, scoped changes to CLAUDE.md or SKILL.md, then compares baseline and candidate behavior on untouched tasks before installing anything.
What it does
- Separates repeated user corrections from one-off requests, copied instructions, and unsupported agent conclusions.
- Traces every proposed rule to dated session evidence and resolves conflicting preferences by scope.
- Reserves independent sessions for validation before drafting changes, reducing leakage from near-duplicate tasks.
- Compares baseline and candidate responses using fixed checks for task success, preference compliance, and regressions.
- Installs only changes that pass the evaluation gate, with a reversible patch and a record of rejected candidates.
- Defines bounded nightly consolidation runs when an execution environment and scheduler are available.
What you get
A consolidation report with evidence, proposed instruction changes, evaluation results, and an installation decision for each candidate. If session access or replay execution is unavailable, you receive clearly labeled proposals and executable evaluation specifications without fabricated results.
Best for
Claude Code users and teams whose recurring work produces session histories, repeated corrections, and instruction files worth improving carefully.
SKILL.md preview
---
name: session-learning-with-held-out-validation
description: Converts past Claude sessions into scoped CLAUDE.md and SKILL.md updates by mining repeated corrections, reserving untouched evaluation tasks, and comparing baseline and candidate behavior before installation. Use when the user says "run a sleep cycle", "review my past sessions", "my agent keeps repeating the same mistakes", "learn my preferences", "consolidate what you learned", "make my agent better over time", or "set up nightly self-improvement".
---
# Session Learning with Held-Out Validation
… (sign up to view the full skill)More ai & agents skills
View all AI & Agents skills →New Skill Scaffolder
Create a new Claude Code skill correctly: right mechanism, valid name and folder, a triggering description, and a smoke test.
Skill Library Lint and Overlap Report
Lint every skill in a repo at once: frontmatter, name-folder match, weak descriptions, overlapping triggers, broken refs, plus a CI script.
Self-Verifying Agent Loop: Plan, Check, Retry, Close
Turns a goal into a task plan with machine-run checks, executes with retry caps, escalates, and refuses to close until verified.
Skill Test Runner (Trigger and A/B)
Test a skill in fresh sessions: ready-to-run prompts, commands, and rubric for trigger recall, false triggers, and blind A/B.