Overnight Skill Optimizer (Eval-Driven)
Improve a skill overnight against your evals: one change per step, held-out set, noise-aware keep or revert, locked evals, hard caps.
Before & after
Same prompt, same agent — the only difference is the skill.
Automated self-improvement of a prompt or skill is an interesting idea. The general approach would be: - Have the agent propose a change - Test the change against some examples - Keep it if it improves, revert if it doesn't - Repeat You'd need a way to measure "better" objectively, which is the hard part. Binary pass/fail tests on example outputs could work. You'd also want version control so you can revert cleanly. Be cautious about letting it run fully unattended, since automated changes can drift in unexpected ways. What skill are you trying to improve?
Yes — a research loop that measures "better" with binary assertions, so it can decide keep-or-revert without you. Here's the loop and the one thing that makes it safe. THE LOOP 1. Claude reads the skill.md 2. Proposes ONE change 3. Runs 5 test prompts through the changed skill 4. Scores each output against 25 true/false assertions (125 checks) 5. ALL pass → git commit. ANY fail → git reset, try a different change 6. Repeat until interrupted or a perfect score holds WHY BINARY ASSERTIONS ARE THE WHOLE TRICK "Is this output better" is subjective and un-loopable. "Does the output contain a dated action item — true/false" is checkable, and 25 of those give you a hard pass-rate the loop can act on without a human. Vague scoring is what makes self-improvement drift; binary scoring is what makes it converge. THE SAFETY MECHANISM — git as the undo Every kept change is a commit; every failed change is a reset. The skill can NEVER end up worse than its last passing state, because a regression is reverted before the next iteration. That's what makes it safe to run overnight — the floor is fixed. WHAT YOU GET The read-propose-test-score-commit loop, a 25-assertion scoring harness, git-gated keep/revert, and no-permission iteration until interrupted. THE INPUT THAT DECIDES QUALITY The 25 assertions. Write them to capture what "good" actually means for this skill and the loop optimizes toward it; write them loosely and it optimizes toward nothing. Which skill, and what does a great output of it contain?
About this skill
Letting Claude tweak a skill all night only works if the score means something. This skill runs a keep-or-revert loop against your evals with the guardrails that stop it from overfitting, fooling itself with noise, or burning your budget.
What it does
- Checks your eval set is ready: enough cases, a held-out split, un-gameable assertions, counter-assertions, and a no-skill baseline
- Runs one change per iteration, repeats each prompt to measure noise, and keeps only real gains
- Locks the eval files, limits edits to the skill folder, and caps iterations, runs, spend, time, and size
- Commits kept changes and reverts rejected ones, never publishing the result
- Produces a morning report with baseline, final, holdout, and no-skill scores
What you get
An eval readiness review, loop protocol and stop rules, a budget estimate, and a morning report template.
Best for
Skill authors with a real eval set who want systematic improvement without babysitting.
SKILL.md preview
---
name: self-improving-skill-overnight-loop
… (sign up to view the full skill)More ai & agents skills
View all AI & Agents skills →New Skill Scaffolder
Create a new Claude Code skill correctly: right mechanism, valid name and folder, a triggering description, and a smoke test.
Skill Library Lint and Overlap Report
Lint every skill in a repo at once: frontmatter, name-folder match, weak descriptions, overlapping triggers, broken refs, plus a CI script.
Self-Verifying Agent Loop: Plan, Check, Retry, Close
Turns a goal into a task plan with machine-run checks, executes with retry caps, escalates, and refuses to close until verified.
Skill Test Runner (Trigger and A/B)
Test a skill in fresh sessions: ready-to-run prompts, commands, and rubric for trigger recall, false triggers, and blind A/B.