Description
The Headless Hybrid Search Pipeline is the developer-facing engine underneath Universal Search. SearchService.search() executes six ordered stages: analyze intent and filters, embed the query, retrieve per-entity full-text and vector candidates, rerank the fused pool, expand selected knowledge-graph neighbors, and refine the response with quality and suggestions. It returns a transport-neutral response suitable for UI, CLI, MCP, API, or another in-process consumer.
Retrieval fans out over the live entity registry. Each stream produces raw BM25 and cosine scores that are min-max normalized and blended in JavaScript, with intent-specific weighting and deterministic deduplication. Optional cross-encoder reranking is timeout- and circuit-breaker-protected. Access, company scoping, caches, events, provider clients, and graph behavior are supplied at composition rather than hidden globals.
Selling Points
- One engine can back many product surfaces without duplicating retrieval, ranking, authorization, or pagination rules.
- Full-text and semantic signals are fused rather than forcing developers to choose one retrieval mode for every query.
- Per-stream degradation preserves successful entity types when one backend fails, while an all-stream failure remains loud.
- Reranking and graph expansion improve relevance but have explicit fallbacks, so optional intelligence does not become a reliability cliff.
- Keyset pagination remains stable across concurrent writes and supports both relevance and date-sorted result sets.
User Story
An internal product team needs search inside a new workflow. They construct SearchService with the workspace's executors, access resolver, embedding adapter, and optional reranker, then call search() with the same typed request used elsewhere. They receive ranked hits, facets, quality metadata, degradation flags, and a continuation cursor without rebuilding search infrastructure.
Extension Surface
Extensible is the first true rung for the pipeline itself. packages/search/src/internal/service.ts accepts structural contracts such as EmbeddingProviderLike, RerankerLike, GraphServiceLike, RephraserLike, and executor interfaces through SearchServiceConfig. Consumers can substitute implementations in-repo without editing the engine, but the pipeline stages and their order are not externally registered plugins.
Capabilities & Limits
- Fuses BM25 and vector scores, applies intent-aware boosts, and optionally reranks with a cross-encoder.
- Supports graph expansion, suggestions, facets, exact pins, degradation metadata, and sort-aware cursors.
- Ensures company scope and readable-Dotabase constraints reach every retrieval statement.
- Limits: only registered entity descriptors participate; the package ships with no default entity types.
- Limits: relevance pagination filters the fused candidate pool, while created/updated sorting uses SQL keysets over the full match set.
Implementation Map
packages/search/src/internal/service.tspackages/search/src/internal/pipeline.tspackages/search/src/internal/stages/retrieve.tspackages/search/src/internal/stages/rrf.tspackages/search/src/internal/stages/rerank.ts