Duplicate Detection and Review

Description

Duplicate Detection and Review finds high-similarity row pairs, emits review work, and applies approved consolidation through a contained consumer.

The detection behavior queries compatible embeddings and emits candidates above a configured threshold rather than merging them. Review-queue behavior turns ambiguous evidence into explicit work, and a separate consumer handles only an approved consolidation event. This separation keeps a probabilistic retrieval signal from becoming destructive authority and preserves a durable event boundary around the decision.

User Story

As a curator, I review likely duplicates and consolidate them without manually searching the entire Dotabase. I enable detection with a threshold appropriate to the corpus and receive bounded candidate pairs with similarity evidence. I compare their meaning and provenance, reject unrelated matches, and approve only true duplicates. The consumer then applies the governed consolidation. I walk away with a cleaner corpus without trusting vector proximity as proof.

Extension Surface

Configurable: bindings control thresholds and review behavior; consolidation semantics remain guarded.

packages/dotabases/src/behaviors/dedup-detection/index.ts exposes thresholds and candidate limits. Those settings tune discovery, but packages/dotabases/src/services/consolidation-consumer.ts keeps approved side effects behind a fixed event and service contract.

Capabilities & Limits

• Capability: Executes the scoped workflow described above through the cited live implementation paths.
• Capability: Preserves typed validation and applicable tenant, schema, lifecycle, or service boundaries.
• Limit: Depends on compatible embeddings. Similarity proposes candidates but does not prove two records are the same. The workflow can retrieve likely matches, suppress unsupported outcomes, create review work, and process approved consolidations. It cannot compare rows without compatible vector provenance, resolve nuanced identity automatically, or guarantee that a high score means duplication. Curator judgment and ordinary row permissions remain required.

Implementation Map

• packages/dotabases/src/behaviors/dedup-detection/index.ts
• packages/dotabases/src/behaviors/review-queue/index.ts
• packages/dotabases/src/services/consolidation-consumer.ts

Related Features

Powered by the Behavior Automation Engine and depends on Semantic Row Embeddings.

Operationally, these relationships preserve a clear ownership boundary around Duplicate Detection and Review. Neighboring features can invoke, supply, or consume its results, but `packages/dotabases/src/behaviors/dedup-detection/index.ts` remains the grounded implementation authority for this capability. That distinction prevents the catalog claim from widening into adjacent rendering, transport, or governance work. Teams can therefore combine the feature with the named dependencies while still diagnosing configuration, validation, and lifecycle behavior at the package boundary that owns it.