Description
The Source Connector Framework is Dots’ developer-facing substrate for bringing structured data from an external system into Dotabases. An adapter implements a provider-neutral contract for authentication, resource discovery, schema description, incremental pulling, row mapping, stable external identity, and optional binaries or relations. Dots then supplies the common machinery: a startup registry, connection and binding records, cursor checkpoints, idempotent row upserts, retries, dead-letter handling, provenance, scrubbing, and post-import enrichment.
The framework deliberately keeps provider logic at the edge. A connector describes the source; the engine remains source-agnostic and targets the Dotabase selected by a binding. This makes a new importer a bounded adapter project instead of a new ingestion system.
Selling Points
- One contract unlocks the whole ingestion stack. Implement the source-specific calls and mappings once, then inherit scheduling, retries, persistence, and enrichment.
- Stable identity prevents duplicate imports. Binding-scoped external ids let repeated and incremental pulls update the same rows.
- Rich sources retain useful structure. Optional binary and relation hooks support attachments and links without hard-coding a provider into the engine.
- Credentials stay out of adapter storage. Authentication resolves a connected-account transport rather than handing raw tokens to connector code.
- Provider failures stay contained. Item errors, fatal runs, resumable cursors, and cooperative cancellation make large imports operationally recoverable.
User Story
An integration developer wants to import records from a new research platform. They implement the SourceConnector methods, register the adapter at the application composition root, and bind one discovered resource to a Dotabase. On the first run Dots creates or maps the schema, imports rows with provenance, and checkpoints progress; later pulls resume incrementally without the developer rebuilding job orchestration.
Extension Surface
This feature is pluggable because packages/connectors/src/contract.ts exports the formal SourceConnector interface and packages/connectors/src/registry.ts accepts registered implementations keyed by provider. The public barrel in packages/connectors/src/index.ts exposes the contract and existing adapter builders. Registration is intentionally performed by a host composition root, so an implementation can be added without teaching the generic engine about its provider.
Capabilities & Limits
Capabilities include schema discovery and mapping, resumable pull cursors, idempotent provenance-keyed persistence, content-aware re-enrichment, binary upload deduplication, and deferred relation resolution. The engine also supports manual backfill, incremental pull, retry, and cadence job definitions.
The honest limit is that two-way source push is declared but not delivered as a general feature; today’s shipped adapters are one-way into Dots. Cross-binding relation upgrades are also deferred, and a connector is not live merely because its code exists: the host must register it and provide connection, storage, jobs, and event seams.
Implementation Map
packages/connectors/src/contract.tspackages/connectors/src/registry.tspackages/connectors/src/engine/orchestrator.tspackages/connectors/src/index.ts