Optimizing skill descriptions

​

How skill triggering works

​

Writing effective descriptions

• Use imperative phrasing. Frame the description as an instruction to the agent: “Use this skill when…” rather than “This skill does…” The agent is deciding whether to act, so tell it when to act.
• Focus on user intent, not implementation. Describe what the user is trying to achieve, not the skill’s internal mechanics. The agent matches against what the user asked for.
• Err on the side of being pushy. Explicitly list contexts where the skill applies, including cases where the user doesn’t name the domain directly: “even if they don’t explicitly mention ‘CSV’ or ‘analysis.’”
• Keep it concise. A few sentences to a short paragraph is usually right — long enough to cover the skill’s scope, short enough that it doesn’t bloat the agent’s context across many skills. The specification enforces a hard limit of 1024 characters.

​

Designing trigger eval queries

[
  { "query": "I've got a spreadsheet in ~/data/q4_results.xlsx with revenue in col C and expenses in col D — can you add a profit margin column and highlight anything under 10%?", "should_trigger": true },
  { "query": "whats the quickest way to convert this json file to yaml", "should_trigger": false }
]

​

Should-trigger queries

• Phrasing: some formal, some casual, some with typos or abbreviations.
• Explicitness: some name the skill’s domain directly (“analyze this CSV”), others describe the need without naming it (“my boss wants a chart from this data file”).
• Detail: mix terse prompts with context-heavy ones — a short “analyze my sales CSV and make a chart” alongside a longer message with file paths, column names, and backstory.
• Complexity: vary the number of steps and decision points. Include single-step tasks alongside multi-step workflows to test whether the agent can discern the skill is relevant when the task it addresses is buried in a larger chain.

​

Should-not-trigger queries

• "Write a fibonacci function" — obviously irrelevant, tests nothing.
• "What's the weather today?" — no keyword overlap, too easy.

• "I need to update the formulas in my Excel budget spreadsheet" — shares “spreadsheet” and “data” concepts, but needs Excel editing, not CSV analysis.
• "can you write a python script that reads a csv and uploads each row to our postgres database" — involves CSV, but the task is database ETL, not analysis.

​

Tips for realism

• File paths (~/Downloads/report_final_v2.xlsx)
• Personal context ("my manager asked me to...")
• Specific details (column names, company names, data values)
• Casual language, abbreviations, and occasional typos

​

Testing whether a description triggers

• should_trigger is true and the skill was invoked, or
• should_trigger is false and the skill was not invoked.

​

Running multiple times

#!/bin/bash
QUERIES_FILE="${1:?Usage: $0 <queries.json>}"
SKILL_NAME="my-skill"
RUNS=3

# This example uses Claude Code's JSON output to check for Skill tool calls.
# Replace this function with detection logic for your agent client.
# Should return 0 (success) if the skill was invoked, 1 otherwise.
check_triggered() {
  local query="$1"
  claude -p "$query" --output-format json 2>/dev/null \
    | jq -e --arg skill "$SKILL_NAME" \
      'any(.messages[].content[]; .type == "tool_use" and .name == "Skill" and .input.skill == $skill)' \
      > /dev/null 2>&1
}

count=$(jq length "$QUERIES_FILE")
for i in $(seq 0 $((count - 1))); do
  query=$(jq -r ".[$i].query" "$QUERIES_FILE")
  should_trigger=$(jq -r ".[$i].should_trigger" "$QUERIES_FILE")
  triggers=0

  for run in $(seq 1 $RUNS); do
    check_triggered "$query" && triggers=$((triggers + 1))
  done

  jq -n \
    --arg query "$query" \
    --argjson should_trigger "$should_trigger" \
    --argjson triggers "$triggers" \
    --argjson runs "$RUNS" \
    '{query: $query, should_trigger: $should_trigger, triggers: $triggers, runs: $runs, trigger_rate: ($triggers / $runs)}'
done | jq -s '.'

​

Avoiding overfitting with train/validation splits

• Train set (~60%): the queries you use to identify failures and guide improvements.
• Validation set (~40%): queries you set aside and only use to check whether improvements generalize.

​

The optimization loop

1. Evaluate the current description on both train and validation sets. The train results guide your changes; the validation results tell you whether those changes are generalizing.
2. Identify failures in the train set: which should-trigger queries didn’t trigger? Which should-not-trigger queries did?
   • Only use train set failures to guide your changes — whether you’re revising the description yourself or prompting an LLM, keep validation set results out of the process.
3. Revise the description. Focus on generalizing:
   • If should-trigger queries are failing, the description may be too narrow. Broaden the scope or add context about when the skill is useful.
   • If should-not-trigger queries are false-triggering, the description may be too broad. Add specificity about what the skill does not do, or clarify the boundary between this skill and adjacent capabilities.
   • Avoid adding specific keywords from failed queries — that’s overfitting. Instead, find the general category or concept those queries represent and address that.
   • If you’re stuck after several iterations, try a structurally different approach to the description rather than incremental tweaks. A different framing or sentence structure may break through where refinement can’t.
   • Check that the description stays under the 1024-character limit — descriptions tend to grow during optimization.
4. Repeat steps 1-3 until all train set queries pass or you stop seeing meaningful improvement.
5. Select the best iteration by its validation pass rate — the fraction of queries in the validation set that passed. Note that the best description may not be the last one you produced; an earlier iteration might have a higher validation pass rate than later ones that overfit to the train set.

​

Applying the result

1. Update the description field in your SKILL.md frontmatter.
2. Verify the description is under the 1024-character limit.
3. Verify the description triggers as expected. Try a few prompts manually as a quick sanity check. For a more rigorous test, write 5-10 fresh queries (a mix of should-trigger and should-not-trigger) and run them through the eval script — since these queries were never part of the optimization process, they give you an honest check on whether the description generalizes.

# Before
description: Process CSV files.

# After
description: >
  Analyze CSV and tabular data files — compute summary statistics,
  add derived columns, generate charts, and clean messy data. Use this
  skill when the user has a CSV, TSV, or Excel file and wants to
  explore, transform, or visualize the data, even if they don't
  explicitly mention "CSV" or "analysis."

​

Next steps