Autoresearch Experiment Setup and Validation
Configure a file optimization experiment with an explicit objective, reproducible evaluator, parsed baseline, and runner handoff.
Before & after
Same prompt, same agent — the only difference is the skill.
Great! Autoresearch is a powerful workflow for optimizing code. To get started, I'll need to understand a few things about what you're trying to achieve. You'll want to think about your target domain, which file you're optimizing, what success looks like, and how to measure improvements. Once we understand your goals, we can set up the experiment loop and start iterating. What file are you working with, and what's your main optimization goal?
AUTORESEARCH SETUP QUESTION 1: DOMAIN What problem domain are you targeting? Example: API response optimization, LLM prompt tuning, search ranking improvement. Your answer: _______________________ QUESTION 2: TARGET FILE Which file or function should autoresearch modify? Include full path. Your answer: _______________________ QUESTION 3: EVAL COMMAND How do we measure success? Provide command that runs your test/benchmark. Example: npm run benchmark -- --file src/search.ts Your answer: _______________________ QUESTION 4: METRIC What single number are we optimizing for? (latency, accuracy, throughput) Your answer: _______________________ QUESTION 5: DIRECTION Are we maximizing or minimizing that metric? Your answer: minimize / maximize QUESTION 6: EVALUATOR Optional — Claude evaluation criteria beyond the metric. Example: "must not break existing test suite" Your answer: _______________________ Once submitted, autoresearch will begin iterating: modify file → run eval → update metric → next iteration.
About this skill
Your optimization loop keeps failing because the evaluation command breaks, the score parser grabs the wrong number, or nobody specified whether a higher score is better. This skill turns a target file and an evaluation idea into a checked experiment configuration before the loop starts.
What it does
- Collects the domain, target file, evaluation command, metric, optimization direction, and evaluator without asking for information already supplied.
- Resolves paths and the evaluator's working directory so execution does not depend on where the loop starts.
- Defines exactly how a score is extracted, including missing values, failed commands, and ambiguous output.
- Captures an LLM judge's complete rubric and scoring rules when that is the evaluator.
- Runs one baseline evaluation when execution is available and authorized, then records the result and validation status.
- Saves configuration in the existing runner's supported format, or provides a clearly labeled standalone draft when that format is unknown.
What you get
An experiment configuration, a baseline validation receipt, and a concise handoff identifying what is ready and what still needs confirmation. Setup stops before candidate edits or optimization iterations.
Best for
Developers configuring autoresearch loops for prompts, code modules, or configuration files, especially when evaluation output is messy or the loop has never completed a trustworthy baseline.
SKILL.md preview
---
name: autoresearch-experiment-setup-and-validation
description: >-
… (sign up to view the full skill)More development skills
View all Development skills →Performance Bottleneck Profiling: Node, Python, Go
Classifies a slowdown, gives exact profiler commands, reads the output, and proves the fix with same-conditions p95/p99 numbers.
Feature Scaffolding with Working Tests
Turn a short feature spec into code, runnable tests, and integration wiring that match your existing project.
Feature Flag Rollouts and Kill Switches
Design staged flag rollouts, verify rollback paths, and audit flag debt with explicit defaults, gates, and retirement plans.
Microcontroller Firmware and RTOS Engineering
Build and debug MCU firmware with bounded ISR/RTOS timing, safe DMA ownership, and measured power budgets.