Prompt Optimization with Task Evals
Get revised prompts, representative eval cases, and measurable checks for RAG grounding, tool selection, output schemas, and cost.
Before & after
Same prompt, same agent — the only difference is the skill.
JSON output can be tricky with language models. You'll want to be very explicit in your instructions about the exact structure you want. Maybe use examples of what good output looks like. You could try putting the JSON schema in backticks or code blocks. Be specific about field names, data types, and any required fields. You might also want to validate the output after you get it to catch any formatting errors. Do you have examples of what you're getting vs. what you want? Are there particular fields that are inconsistent? What model are you using? If you can show me some examples, I can help you refine the prompt to get more consistent results.
PROMPT OPTIMIZATION WORKFLOW STEP 1: DEFINE STRUCTURED-OUTPUT CONTRACT Required fields with types: name (string, required) value (number, required) category (enum: 'A', 'B', 'C', required) metadata (object, optional) STEP 2: BUILD EVALUATION SET Created 15 test cases covering: • Edge cases (null values, special characters in strings) • All enum variants (A, B, C category) • Large numbers and decimal precision • Empty objects and optional field omission Step 3: TEST & MEASURE Ran 5 prompt variants against eval set: Variant 1 (current): 73% correct JSON, 8% type errors, 19% missing fields Variant 2 (explicit schema): 91% correct JSON, 2% type errors, 7% missing fields Variant 3 (schema + example): 96% correct JSON, 0% type errors, 4% missing fields STEP 4: APPLY OPTIMIZED PROMPT Updated instructions: • Embed JSON schema as object type definition (not prose) • Add 2 worked examples with exact output format • Add validation instruction: 'Must be valid JSON parseable by JSON.parse()' • Specify: 'Omit optional fields if no value' STEP 5: TOKEN BUDGET ANALYSIS Original prompt: 420 tokens Optimized prompt: 580 tokens (+38%) Output token reduction: 12% (tighter formatting) Net cost change: +18% per request (acceptable) RECOMMENDATION: Deploy Variant 3 with schema + examples. Retest monthly as model updates ship.
About this skill
Your prompt works on the example you wrote, then breaks on real inputs. This skill turns those failures into test cases, revises the prompt around explicit success criteria, and separates wording problems from retrieval, tool, schema, or model limitations.
What it does
- Builds representative eval cases with expected behavior, failure labels, and hard pass conditions.
- Revises prompts through isolated experiments that expose improvements and regressions.
- Separates RAG retrieval relevance, evidence coverage, and answer faithfulness.
- Checks agent tool selection, arguments, clarification behavior, retries, and stopping rules.
- Defines structured-output contracts and distinguishes valid syntax from correct answers.
- Breaks token usage and cost down by component, including retries and cost per successful task.
What you get
A paste-ready prompt or contract, a compact eval set, and a concrete comparison plan. When outputs or traces are available, you also get an evidence-based diagnosis with measured results clearly separated from estimates and untested recommendations.
Best for
Developers and AI product teams improving an existing LLM feature, debugging inconsistent responses, or validating a prompt before investing in a larger evaluation run.
SKILL.md preview
---
name: prompt-optimization-with-task-evals
description: >-
Improves LLM prompts through failure diagnosis, representative task evals, isolated revisions, and explicit scoring of grounding, tool behavior, schema adherence, and cost. Use when the user says "optimize this prompt", "my prompt keeps giving inconsistent answers", "build an eval set", "my RAG answers make things up", "the agent keeps choosing the wrong tool", "make this return valid JSON", or "why is this prompt so expensive".
---
# Prompt Optimization with Task Evals
… (sign up to view the full skill)More ai & agents skills
View all AI & Agents skills →Autonomous /goal Prompt Builder
Turn rough ideas into one paste-ready /goal prompt with clear scope, constraints, acceptance checks, and autonomy rules.
Autonomous Creative Build Goal Prompts
Turn a rough game, 3D site, or video idea into one build prompt with scoped autonomy, acceptance tests, and delivery criteria.
Claude Power-User Coaching: One Tip Per Turn
Spots the Claude features and prompt habits you're missing in your actual work and coaches one concrete, copyable fix at a time.
Claude Skill Build and Launch
Turn an idea into a complete Claude skill with scoped triggers, acceptance tests, installation steps, and an agent launch plan.