Self-Improving Skill Overnight Loop

Run an automated overnight research loop on a Claude skill: try a tweak, run binary assertions, keep or revert based on pass rate, and iterate.

Agent Orchestrationagent-orchestrationstandard
by AgentVoltv1.0.0Published 1y ago1 install
Free to sign up · every skill included with AgentVolt Pro

Before & after

Same prompt, same agent — the only difference is the skill.

PromptCan a Claude skill improve itself automatically overnight?
Beforegeneric AI agent
Automated self-improvement of a prompt or skill is an interesting idea. The general approach would be:

- Have the agent propose a change
- Test the change against some examples
- Keep it if it improves, revert if it doesn't
- Repeat

You'd need a way to measure "better" objectively, which is the hard part. Binary pass/fail tests on example outputs could work. You'd also want version control so you can revert cleanly.

Be cautious about letting it run fully unattended, since automated changes can drift in unexpected ways.

What skill are you trying to improve?
Afterwith Self-Improving Skill Overnight Loop
Yes — a research loop that measures "better" with binary assertions, so
it can decide keep-or-revert without you. Here's the loop and the one
thing that makes it safe.

THE LOOP
  1. Claude reads the skill.md
  2. Proposes ONE change
  3. Runs 5 test prompts through the changed skill
  4. Scores each output against 25 true/false assertions (125 checks)
  5. ALL pass → git commit. ANY fail → git reset, try a different change
  6. Repeat until interrupted or a perfect score holds

WHY BINARY ASSERTIONS ARE THE WHOLE TRICK
"Is this output better" is subjective and un-loopable. "Does the output
contain a dated action item — true/false" is checkable, and 25 of those
give you a hard pass-rate the loop can act on without a human. Vague
scoring is what makes self-improvement drift; binary scoring is what
makes it converge.

THE SAFETY MECHANISM — git as the undo
Every kept change is a commit; every failed change is a reset. The skill
can NEVER end up worse than its last passing state, because a regression
is reverted before the next iteration. That's what makes it safe to run
overnight — the floor is fixed.

WHAT YOU GET
The read-propose-test-score-commit loop, a 25-assertion scoring
harness, git-gated keep/revert, and no-permission iteration until
interrupted.

THE INPUT THAT DECIDES QUALITY
The 25 assertions. Write them to capture what "good" actually means for
this skill and the loop optimizes toward it; write them loosely and it
optimizes toward nothing. Which skill, and what does a great output of
it contain?

About this skill


name: self-improving-skill-overnight-loop description: Run the original author's auto-research loop on a Claude skill: try a tweak, run binary assertions, keep or revert based on pass rate, and iterate all night.

Self-Improving Skill Overnight Loop

What This Does

The creator adapts the original author's three-file loop to skills by having Claude read the skill.md, propose a change, run five test prompts, and score outputs against 25 true or false assertions. If all pass, git commit; if any fail, get reset and try again, looping without asking for permission until interrupted or a perfect score is hit.

What You Get

  • evals.json with true or false binary assertions
  • auto-modify skill.md loop
  • git commit or git reset per iteration
  • per-iteration pass rate log
  • overnight autonomous run

Customize

  • assertions: array of binary tests per test prompt
  • test_prompts: sample inputs to grade
  • max_iterations: cap to prevent infinite loops
  • commit_on_pass: whether to auto-commit each improvement
  • stop_when: perfect score, manual, or time limit

Sample Output

First run scores 23 of 24 assertions, Claude adds one clarifying rule to the skill.md, second run hits a perfect score and commits.

Target Audience

Skill authors iterating on production skills

SKILL.md preview

SKILL.md
---
name: self-improving-skill-overnight-loop
description: Use this skill when a skill author wants an automated overnight loop that tweaks a skill, scores it against test assertions, and keeps or reverts the change.
version: 1.0.0
category: Agent Orchestration
author: AgentVolt
license: proprietary
tags:
  - agent-orchestration
  - standard
---

# Self-Improving Skill Overnight Loop

Runs an unattended improvement loop on a skill definition: propose a tweak, run it against sample prompts, score the outputs, and keep the change only if it improves the pass rate.

## When to use

… (sign up to view the full skill)
Sign up to view, copy, and install the full skill

More ai & agents skills

View all AI & Agents skills →