How to Run LLM Evals in CI/CD

What belongs in CI

CI should catch obvious regressions quickly. It does not need to prove your entire AI system is perfect. The right pull request suite is small, representative, and strict enough to block dangerous changes.

• 20 to 50 examples for fast pull request feedback.
• Stable examples that cover high-value workflows.
• Per-metric thresholds, not only average scores.
• Segment-level checks for critical query categories.
• A larger scheduled suite for deeper coverage.

Recommended workflow

1. Generate application responses for your eval dataset.
2. Send user input, response, retrieved context, and reference answers to eval.ninja.
3. Store the scored result as a CI artifact.
4. Fail the build if required metrics fall below thresholds.
5. Compare against the main-branch baseline to catch regressions.

name: RAG Evaluation
on:
  pull_request:
    paths:
      - "src/rag/**"
      - "prompts/**"
      - "evals/**"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Generate eval responses
        run: npm run eval:generate -- --dataset evals/smoke.json
      - name: Score with eval.ninja
        run: |
          curl -X POST https://api.eval.ninja/v1/evaluate \
            -H "Authorization: Bearer ${{ secrets.EVAL_NINJA_KEY }}" \
            -H "Content-Type: application/json" \
            -d @evals/generated-responses.json \
            -o eval-results.json
      - name: Check thresholds
        run: |
          node scripts/check-eval-thresholds.js \
            --results eval-results.json \
            --min-faithfulness 0.85 \
            --min-answer-relevancy 0.80 \
            --max-regression 0.05

Set thresholds by metric

Use metric-specific thresholds because each score means something different. Faithfulness protects against unsupported claims. Context recall protects against missing retrieval evidence. Answer relevancy protects against evasive answers.

{
  "minimums": {
    "faithfulness": 0.85,
    "answer_relevancy": 0.80,
    "context_precision": 0.70,
    "context_recall": 0.80
  },
  "max_regression_from_main": 0.05,
  "required_segments": ["billing", "security", "setup"]
}

Cloud and self-hosted paths

For most teams, the cloud API is the fastest way to add CI gates. For regulated environments, point the same request shape at a self-hosted eval.ninja deployment inside your network. The CI pattern does not change; only the base URL and credentials do.

The CLI is coming soon. Today, the reliable integration path is the REST API.

Common mistakes

Running the full benchmark on every small change

Large suites are useful, but they slow feedback. Use a smoke suite for pull requests and the full dataset for scheduled runs or merges to main.

Only checking the average score

A high average can hide a severe faithfulness regression. Gate on individual metrics and required segments.

Letting nondeterminism fail builds randomly

Pin models where possible, use stable judge configuration, and compare against tolerances instead of expecting exact score equality.

What to read next

• Golden datasets for LLM evals
• RAG evaluation metrics
• eval.ninja API docs
• Cloud pricing