Pipeline lifecycle

Every product row carries a pipeline_status that tracks where it is in the enrich → index → search pipeline. Search only returns rows in ready status (when the collection has an enrich pipeline). The status is explicit, durable, and queryable — not inferred from timestamps alone.

Statuses

Status	Meaning	Searchable?
pending	Ingested but not yet enriched/indexed, or awaiting first successful index	No
ready	Passed the indexing gate and indexed successfully	Yes
quarantined	Enriched but rejected by the indexing gate (low confidence, empty surface, etc.)	No
failed	A pipeline stage threw; row has last_error, attempt_count, and next_attempt_at	No
dead	Exceeded max retry attempts (attempt_count >= maxAttempts)	No

pending ──enrich+gate──► ready (surfaces + fts written; vectors when an embedder is configured)
   │                        │
   │                        └── gate rejects ──► quarantined (vectors nulled)
   │
   └── stage throws ──► failed ──retryFailed──► ready | failed | dead
                              │
                              └── attempt_count >= max ──► dead

Where transitions happen

Enrich builds surfaces and runs the gate

During app.enrich.enrich(...), each row runs the enrich stages, then indexing.surfaces builders persist doc, rerank_doc, and fts_src. The required indexing.gate decides whether the row is indexable:

• Gate returns { index: true } → pipeline_status = 'ready' (vectors are written when an embedder is configured).
• Gate returns { index: false, reason } → pipeline_status = 'quarantined' with gate_reason. Previously indexed vectors are nulled so stale hits cannot leak into search.
• An empty surface text → quarantined with reason empty:<surface-key>.

The fashion template’s gate (fashion.indexing().gate) quarantines non-apparel rows, low-confidence enrichments, uncertain load-bearing fields, and cross-signal disagreements. See Tuning search relevance for the data-quality loop that feeds the gate.

Enrich writes surfaces and optional vectors

In the old API, matcher.index(...) was a distinct step: it embedded persisted doc / image bytes, wrote the vector column, set indexed_at, and set pipeline_status = 'ready'. In the new package graph, Enricher.enrich() (@samesake/enrich’s createEnricher) derives the same surfaces (doc, rerank_doc, fts_src) and status/gate outcome. When createEnricher receives embed, it batch-embeds every ready dense surface, L2-normalizes the vectors, and passes them to EnrichStore.writeEnriched in the same call — no separate index verb. Without embed, the factory remains surfaces-only.

Caution

With @samesake/postgres, writeEnriched persists vectors into the collection’s embedding/emb_* halfvec columns. A collection with a cosine or spaces channel therefore needs an embedder in the bundle’s model configuration; omitting it intentionally produces surfaces without vector write-back.

Failures, backoff, and retry

When enrich throws, recordFailure sets pipeline_status = 'failed', increments attempt_count, stores last_error, and schedules next_attempt_at with exponential backoff (capped at one hour).

app.enrich.retryFailed({ limit }) drains retryable rows (failed, next_attempt_at <= now(), attempt_count < maxAttempts):

• Rows that never enriched → re-run enrich.
• Rows at or above maxAttempts (default 5) → promoted to dead.

Enrich runs also abort when the per-run failure rate exceeds a threshold (default 50% after 10 samples) to avoid hammering a broken upstream.

Image revalidation forces re-embed

Caution

The old matcher.revalidateImages(project, collection) probe-and-clear workflow remains a @samesake/server capability; it has no packaged Enricher or SamesakeBundle equivalent. The underlying mechanism is still present — RawRow carries an imageEtag, and the enrich stage cache key incorporates the image validator (imageValidatorsForUrls in @samesake/enrich’s surfaces.ts) — so a caller that detects an image change and re-upserts the row with an updated image_etag marks it dirty again and enrich.enrich() picks it up with fresh vision output. There is no packaged probe-and-revalidate loop; you own polling image_url yourself. See server-only capabilities.

Declaring the indexing DSL

Surfaces and gate live on the collection config — there is no separate compose step and no string template on the embeddings block:

import { collection, gates } from "@samesake/core";

export const products = collection("products", {
  fields: { /* ... */ },
  enrich: { stages: [] },
  indexing: {
    surfaces: {
      embed_doc: {
        kind: "dense",
        embedding: "doc",
        build: ({ data, enriched }) => `${data.title} ${enriched.color_text}`.trim(),
      },
      rerank_doc: {
        kind: "rerank",
        build: ({ data, enriched }) => `${data.title}. Colors: ${enriched.colors?.join(", ")}`,
      },
      fts_doc: {
        kind: "fts",
        build: ({ data }) => `${data.title} ${data.brand}`,
      },
    },
    gate: gates.always, // or a custom gate like fashion.indexing().gate
  },
  embeddings: { doc: { model: "your-model", dim: 1536 } },
  search: { /* channels, rankingPolicy, rerank */ },
});

Use fashion.indexing() from @samesake/presets for the fashion vertical — it wires embed, rerank, FTS surfaces and the confidence/cross-signal gate. The rerank_doc surface feeds second-stage reranking — see Reranking. The same LLM judge powers eval and optional llmRerank — see Relevance judge.

Operational commands

await app.enrich.enrich();
await app.enrich.retryFailed({ limit: 500 });

Caution

Enricher.retryFailed takes only { limit? }; EnrichStore.markDead is a separate explicit operation rather than something retryFailed triggers automatically. If you need an attempt ceiling, track the attempt count in the store and call markDead once your bar is reached.

Quarantined rows remain visible through the existing review endpoint (max_confidence filter) so operators can inspect low-quality enrichments before fixing data or gate thresholds.

Note

Tune the fashion confidence floor and ranking exponents empirically with the offline eval gate — see Eval gate — tune floor and exponents. Do not hardcode calibrated numbers without running the harness.