Description
Scheduled Incremental Imports keep configured sources current without requiring a person to launch every run. A binding chooses manual or scheduled cadence and stores its cursor and last-run state. The connector cadence tick scans due provider bindings, distinguishes a never-run backfill from a later incremental pull, and enqueues work under each binding’s owning company. A separate local-source tick applies the same operational model to folders and repositories.
Scheduling is self-healing rather than success-only. Candidate selection includes active bindings plus errored and stale-backfilling bindings, then applies exponential recovery backoff so a crashed or quota-blocked import can re-enter the queue without being retried every minute. Paused and manual bindings are excluded. Successful work restores active state and resets recovery pressure.
Selling Points
- Keep imported knowledge fresh automatically. Cadence settings turn a one-time connection into a maintained data source.
- Pay for changes, not full re-imports. Provider cursors and local content hashes let later runs focus on incremental work.
- Recover from transient failures. Error and stale-backfill states remain eligible under bounded exponential backoff.
- Preserve tenant boundaries during global sweeps. Each queued run executes under the binding’s own company.
- Give operators a real off switch. Manual cadence and paused status are respected by both connector and local-source schedulers.
User Story
A team binds its research database to Dots on a daily cadence. After the initial backfill, each scheduler tick checks whether the binding is due and queues only the next incremental pull. When a provider outage causes one run to fail, the binding moves through recovery backoff and later succeeds without the team recreating it; pausing the binding immediately removes it from cadence candidates.
Extension Surface
This feature is configurable because cadence, pause state, source kind, and binding mappings change runtime behavior without code. Due-time and backoff calculations live in packages/connectors/src/engine/cadence.ts, while packages/connectors/src/services/binding-service.ts persists cadence and recovery state. A connector may define its own cursor shape through the pluggable contract, but the scheduler itself exposes fixed configuration rather than external job plugins.
Capabilities & Limits
Capabilities include backfill-versus-pull selection, per-binding due calculations, company-correct enqueueing, error recovery, stalled-backfill recovery, pause exclusion, manual-cadence exclusion, and local-source scheduling. Incremental efficiency is source-aware: provider adapters own watermarks, while folder ingestion compares stored source paths and content hashes.
The honest limit is that cadence is polling, not a provider webhook tail, so freshness is bounded by the selected interval and job availability. A scheduler can enqueue work but cannot overcome revoked credentials, permanent permission failures, or provider quotas. Recovery backoff reduces repeated failure pressure but does not resolve the underlying cause, and real-time local folder watching is a separate foreground/desktop mode rather than a cadence guarantee.
Implementation Map
packages/connectors/src/engine/cadence.tspackages/connectors/src/engine/orchestrator.tspackages/connectors/src/services/binding-service.tspackages/local-runtime/src/runtime/local-source-sync.ts