Description
Gemini Multimodal Embeddings provides one BYOK adapter for text, image, video, audio, and interleaved content. GeminiE2Adapter satisfies the ordinary EmbeddingProvider text contract and the richer MultiModalEmbeddingProvider interface, so a composition can use it as a text substitute or call media-specific methods directly. Outputs use selectable Matryoshka dimensions of 768, 1,536, or 3,072.
Each media method shapes a provider request appropriate to its input: raw image bytes, bounded video or audio chunks supplied by bytes or URL, or interleaved text-and-media parts. Results report vector, provider, model, dimension, and input-token usage. The adapter validates requested dimensions, media boundaries, batch sizes, credentials, and returned vector length while keeping its transport injectable.
Selling Points
- One vector space can represent text and media, enabling similarity and retrieval workflows that cross content modalities.
- Matryoshka dimensions let callers trade storage and compute against fidelity without changing the provider family.
- The adapter remains compatible with existing text-embedding consumers while exposing explicit media methods to richer hosts.
- Injected transport and constructor-supplied credentials keep network and secret ownership at the composition boundary.
- Typed validation makes malformed dimensions, responses, batches, and rate limits visible to callers.
User Story
A media ingestion workflow receives a video and its transcript. It divides the video into supported chunks, embeds those chunks and selected text into the configured vector dimension, and stores the vectors with their model metadata. Later, a semantic query can retrieve media moments using the same provider family rather than a separate feature extractor.
Extension Surface
Pluggable is the first true rung because packages/embedding-providers/src/internal/multi-modal.ts defines the formal MultiModalEmbeddingProvider contract alongside the base contract in packages/embedding-providers/src/internal/types.ts. GeminiE2Adapter in packages/embedding-providers/src/internal/gemini-e2.ts is a replaceable implementation, and its GeminiE2TransportLike seam permits a caller-controlled transport.
Capabilities & Limits
- Embeds text, text batches, images, video chunks, audio chunks, and interleaved parts.
- Supports three validated Matryoshka dimensions and reports model, dimension, and input usage.
- Applies timeout, retry, key validation, input truncation, and returned-dimension checks.
- Limits: video chunks may not exceed 120 seconds and text batches are capped at six.
- Limits: the adapter supplies low-level embedding calls; chunk orchestration, cost-cap integration, and event emission belong to the consuming host.
Implementation Map
packages/embedding-providers/src/internal/gemini-e2.tspackages/embedding-providers/src/internal/multi-modal.tspackages/embedding-providers/src/internal/types.tspackages/embedding-providers/src/internal/cost.tspackages/embedding-providers/src/internal/provider-registry.ts