Description
Markdown Directory Import turns a local folder tree into rows in a selected Dots Dotabase. It walks Markdown files, normalizes line endings, computes content hashes, compares the scan with the existing import namespace, and classifies files as creates, updates, unchanged entries, or deletions. An injected apply backend then performs the destination writes and returns detailed progress and errors.
The engine is shared across local-daemon and hosted workflows rather than reimplemented per surface. A canonical local-source namespace keeps files from different roots from colliding, while ignore rules and containment checks bound what is read. Unchanged files are skipped by hash, making repeated imports efficient and predictable.
Selling Points
- Imports an entire Markdown knowledge tree without requiring one-file-at-a-time entry.
- Detects additions, edits, unchanged files, and removals deterministically.
- Avoids rewriting unchanged rows by comparing SHA-256 content hashes.
- Supports both local database and hosted API destinations through one diff core.
- Reports per-file progress and read errors instead of hiding partial outcomes.
User Story
As a user with years of Markdown notes, I choose a folder and destination Dotabase, then import it as a maintained collection. Dots tells me what was created, updated, skipped, deleted, or unreadable, and a later rerun processes only the files that actually changed.
Extension Surface
The rating is configurable because callers select the root, destination, exclusions, normalizer, delete behavior, metadata, and apply backend through packages/ingestion/src/markdown/index-directory.ts. packages/ingestion/src/markdown/http-apply.ts and packages/ingestion/src/markdown/daemon-apply.ts provide hosted and local applications. The directory walker remains a governed Markdown-only engine rather than an arbitrary filesystem executor.
Capabilities & Limits
Capabilities include recursive Markdown walking, CRLF normalization, exclusions, content hashing, create-update-delete classification, dry progress reporting, canonical root identity, and backend-neutral application. Per-file read failures can be reported while valid files proceed. Limits are explicit: non-Markdown formats are not ingested by this engine, deleting a file affects only rows owned by the same import namespace, and destination schema mismatches still require a compatible normalizer or mapping.
Implementation Map
packages/ingestion/src/markdown/index-directory.tspackages/ingestion/src/markdown/walk.tspackages/ingestion/src/markdown/classify.tspackages/ingestion/src/markdown/http-apply.ts