Building High-Quality LLM Judges: A Data-Driven Approach with Claude Code

Imagine you're building an AI assistant that handles scheduling via email. The assistant needs to know when to act independently and when to ask for clarification. Acting without sufficient information leads to mistakes: proposing times when the user said "I'll get back to you," or scheduling with "the team" when it's unclear who that includes.

We needed an LLM judge that could evaluate whether the assistant violated this principle: "Never act when critical information is ambiguous or missing." This is one of the hardest judgment tasks in AI evaluation. It requires understanding context, delegation nuances, and the difference between missing information and reasonable defaults.

Most teams would hand-tune a prompt, run it once or twice, and call it done. We took a different approach: systematic, data-driven iteration with rigorous train/test discipline.

The result: 82.4% recall with only a 2.0% generalization gap, achieved through 10 iterations in a single afternoon. Along the way, we systematically mapped the overfitting boundary and discovered why most prompt engineering fails.

The Judge-Kit Repository

Before diving into our optimization process, let's understand the toolkit we built.

Repository Structure

judge-kit/
├── data/
│   ├── dev_set.json          # 104 labeled traces for iteration
│   ├── eval_set.json         # 95 labeled traces for validation
│   └── taxonomy.json         # Category definitions
├── scripts/
│   ├── 1_extract_category_examples.py
│   ├── 2_run_evaluation.py
│   └── 3_analyze_disagreements.py
├── prompts/
│   ├── cat1_v0.txt           # Initial seed prompt
│   ├── cat1_v1.txt           # Simplified version
│   └── cat1_v10.txt          # Final prompt
├── examples/
│   ├── cat1_positive_dev.txt # Example failures
│   └── cat1_negative_dev.txt # Example successes
└── results/
    └── cat1_v*_*_results.json # Evaluation outputs

Data Format

Each trace represents a complete email thread with the assistant's response:

{
  "test_case_id": 1025,
  "test_case_name": "Meeting with client",
  "input_emails": [
    {
      "from": "client@example.com",
      "subject": "Re: Meeting next week",
      "body": "Would love to connect with you."
    },
    {
      "from": "user@company.com",
      "to": "client@example.com",
      "body": "I will get back to you on timing.",
      "cc": ["assistant@company.com"]
    }
  ],
  "outputs": [{
    "actions": ["send_email"],
    "emails": [{
      "to": ["client@example.com"],
      "body": "Here are some times that work: ..."
    }]
  }],
  "label": {
    "outcome": "failed",
    "error_categories": [1],
    "description": "Assistant proposed times without clear delegation",
    "labeled": true
  }
}

The Three-Script Workflow

Script 1: Extract Examples

python scripts/1_extract_category_examples.py --category 1

Generates human-readable files showing all positive and negative examples for pattern identification.

Script 2: Run Evaluation

python scripts/2_run_evaluation.py --category 1 --prompt prompts/cat1_v1.txt

Calls GPT-4 on each trace, outputs predictions, and calculates metrics.

Script 3: Analyze Disagreements

python scripts/3_analyze_disagreements.py --category 1 --version v1

Shows where the judge disagrees with human annotations, with full context.

The Optimization Journey

Dataset Composition

• Dev set: 104 traces (46 failures, 58 successes)
• Eval set: 95 traces (34 failures, 61 successes)

Key Metrics

• TPR (True Positive Rate / Recall): % of real failures correctly caught
• TNR (True Negative Rate / Specificity): % of non-failures correctly identified
• Agreement: Overall accuracy (TP + TN) / Total
• Gap: Dev Agreement - Eval Agreement (measures overfitting)

Iteration 0: Testing the Comprehensive Approach

We started with cat1_v0.txt, a comprehensive prompt with detailed rules based on manual analysis of all training examples. The hypothesis: more detail and specificity should improve accuracy.

v0 Results:

The Overfitting Problem #1:

Dev Performance:               Eval Performance:
  ████████████████████ 82.7%     ██████████████ 68.4%

Generalization Gap: 14.3% - SEVERE OVERFITTING

The Discovery:

The comprehensive approach achieved high dev performance (82.7%) but failed to generalize. The 14.3% gap revealed something crucial: detailed, example-specific rules memorize training quirks instead of learning general principles.

This wasn't a failure. It was a critical data point that shaped our entire methodology.

Iteration 0→1: Radical Simplification

Instead of tweaking the overfit prompt, we started over with a minimalist approach.

v1 Strategy: Strip away all dev-specific details, keep only core principles.

v1 Results:

The Simplification Win:

Dev Performance:               Eval Performance:
v0: ████████████████████ 82.7%   v0: ██████████████ 68.4%
v1: ███████████████ 76.9%        v1: ████████████████ 73.7% ✓

Generalization Gap:
v0: 14.3% ✗ Severe overfitting
v1: 3.2%  ✓ Healthy generalization

Key Insight: We sacrificed 5.8% dev performance to gain 5.3% eval performance. The simplified prompt generalized much better.

Iteration 1→2: Testing the Boundary

With v1 performing well, we had a new question: could we reduce false negatives without sacrificing generalization? We analyzed v1's 9 false negatives on dev and identified clear patterns:

Missing Patterns:

1. Ambiguous timeframe phrasing (2 cases)
   "Schedule for the next two weeks" - one meeting or multiple?

2. Calendar context issues (2 cases)
   Proposing over tentative meetings without asking

3. Timezone assumptions (2 cases)
   10:30pm-11:55pm local time without checking availability

4. Stricter duration rules (3 cases)
   "Board prep" treated as standard when it's not

We added these as explicit rules in v2.

v2 Results:

The Overfitting Problem #2:

Dev Performance:               Eval Performance:
v1: ███████████████ 76.9%        v1: ████████████████ 73.7%
v2: ████████████████████ 81.7%      v2: ██████████████ 69.5%

Generalization Gap:
v1: 3.2%  ✓
v2: 12.2% ✗ Back to overfitting!

The Discovery:

We improved dev performance by 4.8%, but eval performance dropped by 4.2%. The experiment revealed a critical insight about the overfitting boundary.

Rules targeting specific dev failures (calendar context, timeframe phrasing) were too narrow. They captured training quirks, not general patterns. This confirmed our hypothesis: there's a maximum complexity threshold beyond which generalization degrades.

Iterations 3→7: Developing Surgical Optimization

With the overfitting boundary mapped, we developed a new methodology:

Surgical Optimization Rules:

1. Never add rules to fix all dev disagreements
2. Only add rules that appear in multiple diverse examples
3. Test generality: "Would this rule make sense on unseen data?"
4. Keep rules conservative with explicit PASS examples
5. Monitor generalization gap as closely as accuracy

Through v3-v7, we:

• Added conservative timezone checks
• Refined duration ambiguity (with many PASS examples)
• Strengthened participant clarity rules
• Avoided calendar context (too dev-specific)

Iteration 8: The TNR vs TPR Trade-off

By v8, we achieved strong specificity but sacrificed recall.

v8 Results:

The Problem: v8 had excellent true negative rate (82.0%) but was missing 38.2% of real failures on eval. Too conservative.

v8 Confusion Matrix (Eval Set):

┌────────────────────────┐
│ TP: 21  │  FN: 13     │  ← Missing 13 failures!
│         │             │
├─────────┼─────────────┤
│ FP: 11  │  TN: 50     │  ← But great specificity
└────────────────────────┘

TPR: 61.8% - Too many missed failures
TNR: 82.0% - Excellent specificity

Iteration 8→9: Validating the Methodology

To fix v8's low TPR, we tested whether our surgical methodology could handle more aggressive rules. We added calendar context and timeframe checks.

v9 Results:

The Validation:

Dev Performance:               Eval Performance:
v8: ████████████████████ 83.7%   v8: ████████████████ 74.7%
v9: █████████████████████ 85.6%  v9: ███████████████ 72.6%

Generalization Gap:
v8: 9.0%  ✓ Acceptable
v9: 13.0% ✗ Confirms boundary

The Confirmation:

v9 confirmed our hypothesis: calendar context and aggressive timeframe rules cross the overfitting boundary. The 13.0% gap validated what we learned from v0 and v2: some types of rules simply don't generalize.

This wasn't wasted effort. It confirmed that our methodology correctly identified the boundary between generalizable and non-generalizable rules.

Iteration 9→10: The Breakthrough

Instead of adding more rules, we took a different approach for v10:

v10 Strategy:

1. Analyze which types of failures v8 was missing (not which specific cases)
2. Add rules for failure categories, not individual examples
3. Include explicit negative examples (PASS rules)
4. Focus on information availability, not decision optimality

New v10 Rules:

• Enhanced "Unclear Participants" - CC/body mismatches
• New "Conflicting Information" - subject/body conflicts
• Stronger working hours enforcement - midnight-6am explicitly forbidden
• Enhanced third-party assumptions with examples
• Added "Other Failures NOT Cat 1" PASS rule to reduce false positives

v10 Results:

The v10 Breakthrough:

True Positive Rate (Recall):
v8:  ████████████ 61.8%
v10: ████████████████████ 82.4% (+20.6% ✓✓✓)

Generalization Gap:
v8:  █████████ 9.0%
v10: ██ 2.0% (-7.0% ✓✓✓)

Overall Agreement:
v8:  ████████████████ 74.7%
v10: █████████████████ 76.8% (+2.1% ✓)

The Trade-off: We're catching 20.6% more failures at the cost of 8.2% more false positives. For a safety-critical system, this is the right trade-off.

The Complete Performance Journey

Version History (10 Iterations):

Version  Dev Agr  Eval Agr  Gap     Status
──────────────────────────────────────────────
v0       82.7%    68.4%     14.3%   ✗ Overfit #1
v1       76.9%    73.7%     3.2%    ✓ Simplified
v2       81.7%    69.5%     12.2%   ✗ Overfit #2
v3-v7    [intermediate iterations]
v8       83.7%    74.7%     9.0%    ~ Low TPR
v9       85.6%    72.6%     13.0%   ✗ Overfit #3
v10      78.8%    76.8%     2.0%    ✓✓ Best!


Key Metrics Evolution:

Metric          v0      v1      v2      v8      v9      v10     Best
─────────────────────────────────────────────────────────────────────
Eval TPR       64.7%   73.5%   61.8%   61.8%   61.8%   82.4%   v10 ✓
Eval TNR       70.5%   73.8%   73.8%   82.0%   78.7%   73.8%   v8
Eval Agr       68.4%   73.7%   69.5%   74.7%   72.6%   76.8%   v10 ✓
Gen. Gap       14.3%   3.2%    12.2%   9.0%    13.0%   2.0%    v10 ✓

Key: ✓ good, ✗ overfit

How Claude Code Accelerated This Process

Traditional prompt optimization is slow and fragmented. We iterated through 10 versions in a single afternoon.

Without Claude Code: ~35 minutes per iteration

1. Edit prompt in text editor
2. Run evaluation script in terminal
3. Wait for results
4. Open JSON results file
5. Search for disagreements
6. Cross-reference to data files
7. Copy test IDs, search traces
8. Read email threads
9. Take notes on patterns
10. Switch back to editor
11. Repeat...

With Claude Code: ~8 minutes per iteration (4x faster)

1. Conversational Evaluation

User: "run evaluation on v2"

Claude Code:
✓ Executes: python scripts/2_run_evaluation.py --category 1 --prompt cat1_v2.txt
✓ Reads results JSON
✓ Calculates confusion matrix
✓ Identifies patterns in disagreements
✓ Compares to previous versions
✓ Provides recommendations

All in one response.

2. Automatic Cross-Referencing

User: "why did v2 overfit?"

Claude Code:
✓ Compares v1 and v2 prompts
✓ Reads both result files
✓ Identifies which new rules hurt eval performance
✓ Shows specific examples where v2 failed
✓ Categorizes failure patterns
✓ Recommends surgical fixes

Time: 30 seconds (vs 20 minutes manually)

3. Pattern Analysis Across Iterations

When we hit the third overfitting trap:

User: "we've overfit three times now, what's the pattern?"

Claude Code:
✓ Reads v0, v2, v9 prompts
✓ Identifies common characteristics
✓ Discovers: all added 3+ new rules at once
✓ Discovers: all targeted specific dev examples
✓ Discovers: none included explicit PASS rules
✓ Recommends: category-based rules with negative examples

Result: v10 methodology that actually worked

4. Surgical Prompt Editing

Edit(
  file_path="prompts/cat1_v10.txt",
  old_string="""6. THIRD-PARTY ASSUMPTIONS (3 cases)
   - Assistant assumes third-party availability
   - Must verify before proposing times""",

  new_string="""6. THIRD-PARTY ASSUMPTIONS (3 cases)
   - Assistant assumes third-party availability
   - ESPECIALLY: Proposing times outside reasonable working hours
   - Must not propose before 7am or after 11pm local time
   - Example FAIL: Proposing midnight-6am without explicit approval
   - Example PASS: 9am-5pm is reasonable default"""
)

Exact string replacement, preserving formatting, instant iteration.

5. Methodological Guardrails

After the second overfitting trap:

Me (Claude): "Let's add rules for all 13 FNs on dev..."

User: "won't that overfit again?"

Me: "You're right. Let me categorize them first and add only generalizable patterns."

The long conversation context enabled learning from past mistakes.

6. Multi-Version Comparison

User: "should we use v8 or v10?"

Claude Code:
✓ Compares confusion matrices
✓ Calculates precision/recall trade-offs
✓ Considers safety implications (false negatives worse than false positives)
✓ Recommends v10 for safety-critical system
✓ Suggests v8 if false positives are costly

Contextual decision-making, not just metrics.

Key Methodological Lessons

1. Map the Overfitting Boundary Systematically

Through deliberate experimentation, we identified exactly where overfitting begins:

Rules that generalize:

• Principle-based (vague intent, unclear participants)
• Category-based (meeting types with ambiguous duration)
• Broad working hours (midnight-6am forbidden)

Rules that overfit:

• Calendar-specific (tentative meeting conflicts)
• Temporal phrasing ("next two weeks" ambiguity)
• Example-specific (targeting individual dev cases)

This wasn't trial and error. It was systematic exploration that yielded a reproducible methodology.

The framework:

• Test on eval after every iteration
• Monitor gap as closely as accuracy
• Validate rule types across multiple iterations
• Prefer category-based rules over example-based rules

2. Simplification Often Beats Addition

Our best generalization came from removing complexity:

• v0 → v1: Stripped detailed rules, gap dropped 14.3% → 3.2%
• Result: +5.3% eval performance

3. Trade-offs Are Inevitable

You can't maximize everything:

• v8: Great TNR (82.0%), terrible TPR (61.8%)
• v10: Great TPR (82.4%), worse TNR (73.8%)

The right choice depends on your use case:

• Safety-critical: Maximize TPR (v10) - catch all failures
• Cost-sensitive: Maximize TNR (v8) - minimize false alarms

4. Category Rules Beat Example Rules

Example rule (overfit):

FAIL: User says "I'll get back to you" then CCs assistant

Category rule (generalize):

FAIL: Vague executive intent without explicit delegation
- Includes: "I'll think about it", "Let me check", "I'll get back"
- Requires: Clear handoff like "please schedule" or "find times"

Category rules capture the why, not just the what.

5. Negative Examples Prevent False Positives

v10's breakthrough included explicit PASS rules:

PASS: Other Failures NOT Cat 1
- Assistant picks suboptimal time (Cat 2: poor judgment)
- Assistant double-books (Cat 3: calendar error)
- Assistant formats badly (Cat 4: execution error)

These are failures, but NOT "action without clarification"

Negative examples teach the judge what NOT to flag.

Results Summary

Final Performance (v10):

Journey from Worst to Best:

Cautionary Tale - v12:

After v10, we tested an "escalation check" approach that optimized for precision at the expense of recall:

• Eval TPR: 50.0% (catching only half of failures!)
• Eval TNR: 88.5% (excellent precision)
• Result: Too conservative for a safety-critical system

This validated our v10 design decision: for safety-critical applications, catching 82.4% of failures with some false positives beats catching only 50% with fewer false positives.

What We Achieved:

• ✅ 82.4% recall on evaluation set (vs 64.7% at v0)
• ✅ 76.8% overall agreement (vs 68.4% at v0)
• ✅ Only 2.0% generalization gap (vs 14.3% at v0)
• ✅ Systematic, reproducible methodology
• ✅ Validated design decisions (v12 proved v10's TPR/TNR trade-off was correct)

Time Investment:

• Total iterations: 10+ versions
• Total time: ~90 minutes (single afternoon)
• Average per iteration: ~9 minutes
• 4x faster than traditional approach (would have taken ~6 hours)

Path to 90%

Current status toward 90% goal:

• TPR: 82.4% (need +7.6%)
• TNR: 73.8% (need +16.2%) ← Primary gap
• Agreement: 76.8% (need +13.2%)

Next Steps

Option 1: Fix False Positives (Improve TNR)

• Analyze v10's 16 FPs on eval
• Add more specific PASS rules
• Refine FAIL conditions with negative examples
• Target: TNR 73.8% → 85%+ while keeping TPR high

Option 2: Relabel Ground Truth

• Some eval cases may be mislabeled
• Validate with domain experts
• More honest assessment of actual performance

Option 3: Hybrid Approach

• Fix clear mislabels (2-3 cases)
• Refine v10 rules to reduce FPs
• Surgical improvements targeting highest-confidence errors

Try It Yourself

The methodology is reproducible with any LLM judge task:

# 1. Extract examples for pattern analysis
python scripts/1_extract_category_examples.py --category 1

# 2. Run evaluation
python scripts/2_run_evaluation.py --category 1 --prompt prompts/cat1_v1.txt

# 3. Analyze disagreements
python scripts/3_analyze_disagreements.py --category 1 --version v1

With Claude Code:

User: "run evaluation on category 1 with v1 prompt"
Claude Code: [Executes, analyzes, provides insights]

User: "show me the false positives"
Claude Code: [Deep analysis with patterns and recommendations]

User: "compare v1 and v2 to see what changed"
Claude Code: [Diff analysis with performance impact]

Conclusion

Most teams approach prompt engineering like alchemy: try something, see if it works, repeat. We took a different approach.

Through 10 systematic iterations in a single afternoon, we:

1. Mapped the overfitting boundary - Discovered which rule types generalize and which don't
2. Validated the methodology - Confirmed findings across multiple iterations
3. Achieved production metrics - 82.4% recall with only 2.0% generalization gap
4. Built a reproducible framework - Category-based rules, negative examples, surgical optimization

The key insight: prompt engineering is experimental science, not guesswork. Form hypotheses, test systematically, learn from each iteration.

What made this possible:

• Rigorous train/test discipline - Never iterate on eval performance
• Trade-off awareness - Understand TPR vs TNR for your use case
• Velocity - Claude Code enabled 10 iterations in 90 minutes vs 10+ hours manually
• Pattern recognition - Identify which changes generalize across iterations

The result isn't just a working judge. It's a methodology for building any LLM judge that actually generalizes to production data.

The future of prompt engineering isn't more complex prompts. It's systematic iteration with the right tools and methodology.

Tools: Claude Code (claude.com/code)

Methodology: Pattern-driven prompt engineering with train/test discipline and overfitting vigilance