Description
Anthropic Vision Description is a dedicated image-to-text sibling of the ordinary Anthropic LLM adapter. Callers provide a base64 image, one of four supported media types, a user instruction, and optional system text, model, and output-token limit. The adapter sends one Anthropic Messages request containing an image block followed by the text instruction and returns the description with model and usage metadata.
Keeping vision as a sibling avoids widening every LLMMessage.content value from a string into a provider-specific multimodal union. It still follows the package's BYOK, timeout, retry, structured-error, and pricing conventions. The default is the Haiku tier and the default output ceiling is 4,096 tokens.
Selling Points
- Gives media workflows a narrow image-description API instead of forcing them through vendor wire formats.
- Preserves a simple text-only
LLMProvidercontract for the much larger population of ordinary callers. - Supports the common web image formats with a type-level media boundary.
- Returns actual usage and model information so downstream provenance and spend records reflect the completed call.
- Uses the same credential, retry, timeout, and error discipline as the production text adapter.
User Story
A media extraction workflow receives a PNG attached to a row. It reads the bytes, encodes them, and calls describe() with an instruction to extract visible facts. The workflow receives a textual description plus the producing model and usage, stores the derived content with provenance, and records spend from the result rather than a guessed flat amount.
Extension Surface
Pluggable is the first true rung because packages/llm-providers/src/anthropic-vision.ts exposes a replaceable AnthropicVisionAdapter behind explicit VisionDescribeInput and VisionDescribeResult contracts, while the wider package provides formal LLM adapter conventions in packages/llm-providers/src/types.ts. A composition can select another vision implementation without changing the consuming media workflow's conceptual boundary.
Capabilities & Limits
- Accepts base64 JPEG, PNG, GIF, or WebP input with separate system and user instructions.
- Supports model and maximum-output overrides and returns text, usage, and model.
- Reuses Anthropic request validation, timeout, retry, key handling, and cost estimation.
- Limits: one call describes one image; it is not a general multi-image conversation or video-analysis loop.
- Limits: media types outside the four Anthropic image formats must be converted or handled by another capability.
Implementation Map
packages/llm-providers/src/anthropic-vision.tspackages/llm-providers/src/anthropic-http.tspackages/llm-providers/src/cost.tspackages/llm-providers/src/errors.tspackages/llm-providers/src/index.ts