Chief AI Officer Plan Review
Evidence-first review of one AI plan: eval gaps, cost per successful task, model and vendor risk, prompt injection, and EU AI Act screen.
Before & after
Same prompt, same agent — the only difference is the skill.
That's an important decision. Using an AI API for moderation involves thinking about accuracy, latency, cost, and regulatory compliance. You'll want to consider whether a fine-tuned model might be better than API calls, what your tolerance is for false positives versus false negatives, and how that affects user experience. Budget-wise, API costs can add up quickly at scale. I'd also recommend thinking about GDPR, data retention, and whether you're meeting any industry-specific compliance requirements. Before making a final choice, you should probably evaluate how different approaches perform on representative content. What volume of content are you moderating per day?
CHIEF AI OFFICER EVAL — CONTENT MODERATION RISK CLASSIFICATION: Medium-to-High (user-generated content, regulatory touch) COST ANALYSIS: Benchmark: 1M content moderation decisions/month • Claude API: 0.3¢ per decision = $3,000/month + latency overhead • Fine-tuned GPT: $2,500 setup + $500/month = faster inference • Human baseline: 5 moderators at $40k/year = $200k/year EVAL SET REQUIRED: Minimum 200 test samples: 50 hate speech, 50 spam, 50 on-brand, 50 edge cases. Test API once, fine-tune on 20 samples, re-test. Accuracy gap acceptance: 2-4% vs human baseline is reasonable; over 6% triggers escalation protocol. COMPLIANCE CHECKPOINTS: • GDPR: API logs retention policy (claude.ai logs moderated content 30 days) • Appeal mechanism: flagged content reviewable by human within 24hr • Transparency: user notices when content removed via AI; link to policy RECOMMENDATION: API + human escalation queue. Start with 5% human review; scale to 10% if false positive rate exceeds 8%. This is illustrative; not legal or compliance advice.
About this skill
Before an AI feature ships, answer the question a Chief AI Officer asks first: how do you know it works? This review checks the evidence, the real cost, and the risks a demo hides.
What it does
- Checks eval sets, labels, baselines, and severity weighting
- Computes cost per successful task at production volume, including agent loops
- Flags model deprecation, data-use terms, and single-vendor risk
- Checks prompt injection exposure and tool permissions
- Screens EU AI Act and sector risk, always marked for verification
What you get
A ship / ship with conditions / not yet / stop verdict, an evidence gap table, cost math, ranked risks, a regulatory screen for counsel, and the artifacts needed for yes.
Best for
Product, engineering, and AI leads shipping LLM features or agents, and executives approving AI spend.
SKILL.md preview
---
name: caio-review
description: Critique one AI plan - an AI feature launch, model or vendor choice, agent rollout, fine-tuning project, or AI hire - through a Chief AI Officer's lens and return a verdict, the eval evidence that is missing, cost per successful task at production volume, model and vendor risks, security gaps like prompt injection, and a regulatory risk screen marked for verification. Use when the user says "review our AI launch plan", "is this AI feature ready to ship?", "we have no evals yet, is that a problem?", "check our LLM cost estimate", "should we fine-tune or use RAG, here's the plan", or "would a Chief AI Officer approve this?".
---
… (sign up to view the full skill)More ai & agents skills
View all AI & Agents skills →New Skill Scaffolder
Create a new Claude Code skill correctly: right mechanism, valid name and folder, a triggering description, and a smoke test.
Skill Library Lint and Overlap Report
Lint every skill in a repo at once: frontmatter, name-folder match, weak descriptions, overlapping triggers, broken refs, plus a CI script.
Self-Verifying Agent Loop: Plan, Check, Retry, Close
Turns a goal into a task plan with machine-run checks, executes with retry caps, escalates, and refuses to close until verified.
Skill Test Runner (Trigger and A/B)
Test a skill in fresh sessions: ready-to-run prompts, commands, and rubric for trigger recall, false triggers, and blind A/B.