Relevance judge

The relevance judge classifies candidate products against a shopper query on the 4-class ESCI rubric — Exact / Substitute / Complement / Irrelevant, mapped to gains 3 / 2 / 1 / 0 (Substitute is a soft positive). The same judge implementation powers both the eval loop below and llmRerank — one judge, two consumers.

Note

makeLlmJudge, llmRerank, calibrateJudge, and the metric functions (ndcgAtK, hitAtK, mrr, …) are standalone exports from @samesake/server and compose with the canonical search/samesake primitives. The all-in-one matcher.runEval() orchestration convenience is still Matcher-instance-only and has not been ported to @samesake/query/@samesake/postgres — see Eval gate for how to compose the loop yourself against app.search.

Two honesty rules are enforced by the framework:

1. Family separation — an eval judge must come from a different model family than the collection’s enrich pipeline (a Gemini judge grading Gemini-written search_documents flatters itself). runEval and evaluateSearch throw on a same-family (or undeclared) judge when the collection is enriched.
2. Prompt-pinned version — the judge version embeds a content hash of the rubric (esci-v1@<hash>), so any prompt edit automatically invalidates cached grades.

API

import { makeLlmJudge } from "@samesake/server";

const judge = makeLlmJudge(generate, {
  model?: string;
  version?: string;
  batchSize?: number;
  onError?: (msg: string) => void;
});

// RelevanceJudge
interface RelevanceJudge {
  version: string;      // "<tag>@<sha256(rubric)[:8]>"
  model?: string;       // used for enrich/judge family separation
  grade(query: string, candidates: JudgeCandidate[]): Promise<JudgedHit[]>;
}

interface JudgedHit {
  id: string;
  grade: 0 | 1 | 2 | 3; // ESCI gain: E=3, S=2, C=1, I=0
  esci: "E" | "S" | "C" | "I";
  reason: string;
}

Model is BYO / opaque pass-through

The judge calls your generate with model: opts.model. Default undefined — your generate picks. No hardcoded model in the judge layer.

To pin a model (declare it — runEval needs it for family separation):

const judge = makeLlmJudge(openaiGenerate, {
  model: "gpt-4.1-mini", // cross-family vs a Gemini enrich pipeline
});

System prompt

The judge uses ESCI_JUDGE_SYSTEM verbatim:

You are a strict multilingual e-commerce search relevance judge. Classify each candidate against the shopper’s query as exactly one of: E (Exact) — satisfies every explicit constraint in the query; S (Substitute) — not exact but a reasonable alternative for the same need; C (Complement) — typically bought or worn together with what was asked for; I (Irrelevant) — fails the intent or conflicts with an explicit attribute. A candidate with a conflicting required attribute (wrong base color, wrong gender, over an explicit price bound) is I, not S.

User prompt shape

renderCandidates builds the user message:

Shopper query: {query}
Candidate products:
1. {candidate.text}
2. {candidate.text}
...
Return one ESCI class (E|S|C|I) per candidate with a short reason. Keep the original candidate order.

Candidate text comes from candidateSummary when built from product data (title, brand, price, category, colors, occasions, styles, material, pattern, fit, description) or from the text field passed in (rerank path uses rerankCandidateText). Price is included so the judge can verify numeric constraints like “under 5000” — without it, numeric queries get under-graded (a judge that can’t see the price can’t confirm the bound).

Structured output schema

The judge passes judgeSchema as generate’s schema (provider JSON / responseSchema mode):

{
  "type": "object",
  "properties": {
    "grades": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": { "type": "string" },
          "esci": { "type": "string", "enum": ["E", "S", "C", "I"] },
          "reason": { "type": "string" }
        },
        "required": ["id", "esci", "reason"],
        "additionalProperties": false
      }
    }
  },
  "required": ["grades"],
  "additionalProperties": false
}

parseJudgeOutput maps the response back to JudgedHit[], preserving candidate order. Missing or malformed rows get grade: 0, esci: "I", reason: "judge-error".

Mechanics

Batching: batchSize defaults to 10 — one generate call per batch.

Cache (eval path): runEval caches judge calls under evals/.cache/ with key sha1(judgeVersion|query|candidate.text) (judgeCacheKey in core/eval/cache.ts). Re-runs over an unchanged golden set issue zero new judge calls.

Never throws (grading): generate errors, malformed output, or omitted candidates → grade: 0, esci: "I", reason: "judge-error". The search and eval paths continue. (Family separation does throw — an eval that would lie should not run.)

Versioning and calibration

version resolves to <tag>@<sha256(ESCI_JUDGE_SYSTEM)[:8]> (default tag esci-v1), so the rubric content is pinned into the cache key and the calibration unit — editing the prompt invalidates caches automatically. Changing the model still warrants re-running calibrateJudge before trusting the judge to gate.

import { calibrateJudge, isJudgeTrusted } from "@samesake/server";

const result = await calibrateJudge(judge, humanLabels, { minLabels: 5 });
const trusted = isJudgeTrusted(result); // default bar: F1 ≥ 0.80

Calibration reports precision, recall, F1, and Cohen’s κ on the 0–3 gains (κ-primary for ordinal agreement); the relevance floor defaults to 2 — Substitute or better counts as relevant. See Eval gate — tune floor and exponents.

Caution

Do not change ranking or gate thresholds based on an uncalibrated judge. The rubric hash handles prompt drift; pass a new version tag for model or methodology changes.

Two consumers

Consumer	Score mapping	Use
Rerank (llmRerank)	grade / 3 → [0, 1]	Second-stage blend with RRF
Eval (runEval)	raw grade (0–3)	nDCG@K; grade ≥ floor (default 2 = Substitute) for Hit@K/MRR

Related

• Reranking — how judge grades become rerank scores
• Providers — wire generate for the judge
• Eval from search snapshots — build a corpus before going live
• Eval gate — CI gate on the golden set