Measure enrichment accuracy

Search relevance is only as good as the attributes the enrich pipeline extracts: a mis-labeled color or a missed neckline silently corrupts ranking. Search-relevance eval measures the downstream symptom (see Eval gate); the Enricher’s evaluate(gold) measures the root cause — did classify + extract pull the right structured attributes?

Tip

This is the enrichment twin of the search eval. Run both: enrichment accuracy first (is the data right?), then search relevance (is the ranking right?).

How it scores

Every attribute value is treated as a set token. For each product × attribute:

• TP = values in both gold and prediction, FP = predicted but not in gold (hallucination), FN = in gold but not predicted (a miss — “NULL is worse than wrong”).
• Aggregated per attribute → precision / recall / F1, plus micro (pooled) and macro (mean per attribute).

A gold label that is absent means “unlabeled” (skipped). A label of [] or "unknown" means “explicitly no value” and is scored — so a hallucinated value counts against you.

Run it

1. Label a gold set — real products, attributes labeled independently of the pipeline. The fashion example ships one: evals/golden-enrichment-fashion-lk.json (50 LK products, labeled from titles). Bootstrap a blank template for a new corpus:

   cd examples/fashion-search
   bun eval-enrichment.ts --bootstrap        # → evals/golden-enrichment.template.json

2. Score offline (no DB, no LLM — CI-safe) against captured pipeline output:

   bun eval-enrichment.ts --fixture

3. Score live against your seeded corpus:

   bun --env-file=../../.env eval-enrichment.ts

4. Testing an enrich-prompt / taxonomy change? The seeded corpus is baked, so re-enrich the gold products live through the current pipeline, then score — this is what actually exercises a prompt change. Run once before and once after your change to get a clean pre/post:

   bun --env-file=../../.env eval-enrichment.ts --reenrich --tag=pre    # before the change
   # …edit the enrich prompt/schema, rebuild…
   bun --env-file=../../.env eval-enrichment.ts --reenrich --tag=post   # after

Caution

Global enrich-prompt edits are risky — a change that fixes one product often breaks others (e.g. tightening the non-apparel rule can misclassify watches). The re-enrich pre/post gate catches this. For narrow edge cases, prefer the few-shot correction loop (review.ts → corrections fed back as exemplars) over rewriting the global prompt.

Both write evals/runs/<ts>-enrichment-*.{json,md} and print a per-attribute scorecard:

| attribute          | precision | recall |  F1  |
| category           |   94.0%   | 94.0%  | 94.0%|
| gender             |  100.0%   |100.0%  |100.0%|
| colors             |   98.1%   |100.0%  | 99.0%|
| is_apparel_product |   98.0%   | 98.0%  | 98.0%|
| overall (micro)    |   97.6%   | 98.1%  | 97.8%|

The disagreement list is the payoff — it names each product where the pipeline and gold differ (missed vs hallucinated values), so a bad classification (e.g. a non-apparel item that wasn’t gated) is visible and regressable.

In code

evaluate is a method on the Enricher returned by createEnricher — declare evalAttributes at construction time (the Tier-2 samesake() bundle does not wire evalAttributes today, so build the enricher directly for eval, reusing the bundle’s enrichStore from createPostgresBackend):

import { createEnricher } from "@samesake/enrich";
import { createPostgresBackend } from "@samesake/postgres";
import { products } from "./catalog.ts";
import { generate, embed } from "./models.ts";

const backend = createPostgresBackend({ url: process.env.SAMESAKE_DATABASE_URL!, collection: products });
const enricher = createEnricher({
  collection: products,
  generate,
  embed,
  store: backend.enrichStore,
  evalAttributes: [
    { name: "category", kind: "single" },
    { name: "colors", kind: "multi" },
    { name: "is_apparel_product", kind: "single", empty: [] },
  ],
});

const result = await enricher.evaluate([
  { id: "1", labels: { category: "dresses", colors: ["red"], is_apparel_product: true } },
]);
// result.attributes[].{precision,recall,f1}, result.overall.microF1, result.diffs

Gate your enrich-prompt, taxonomy, or FASHION_CONFIDENCE_FLOOR changes on result.overall.microF1 (or a per-attribute floor) the same way ranking changes are gated on retrieval nDCG.