Atlas Plan
Plans012 2026 02 23 Pipeline Workflow Unification

Progress

2026-02-23 13:51 - T-001

Overview: Refactored CLI parsing into a dedicated module with month-range parsing, uppercase normalization, and report type flag support.

Completed:

  • refactor(pipeline): extract CLI parsing and shared parsers into @source/cli.ts
  • feat(pipeline): add parseMonth() support for single-month and month-range values
  • test(pipeline): update CLI parser unit tests for normalization and --type parsing

Files:

  • @packages/pipeline/@source/cli.ts
  • @packages/pipeline/@source/index.ts
  • @packages/pipeline/@source/tests/config.test.ts

2026-02-23 13:51 - T-004

Overview: Added DuckDB pipeline metadata table setup and helper APIs for pipeline run tracking.

Completed:

  • feat(pipeline): add _pipeline_runs table bootstrap with sequence and composite index in DuckDB helpers
  • feat(pipeline): implement recordPipelineStart(), recordPipelineComplete(), and getLastRun() metadata helpers

Files:

  • @packages/pipeline/@source/duck.ts
  • @packages/pipeline/@source/meta.ts

2026-02-23 13:58 - T-005

Overview: Added a transform command that executes dbt via uv and tracks run lifecycle in _pipeline_runs.

Completed:

  • feat(pipeline): add transform command module that shells out to uv run dbt run
  • feat(pipeline): capture dbt exit code/stderr and persist success/failure metadata in _pipeline_runs
  • test(pipeline): extend CLI parser tests for --select passthrough support

Files:

  • @packages/pipeline/@source/commands/transform.ts
  • @packages/pipeline/@source/index.ts
  • @packages/pipeline/@source/cli.ts
  • @packages/pipeline/@source/tests/config.test.ts

2026-02-23 13:58 - T-002

Overview: Added a unified sync command to orchestrate extract -> load -> validate -> transform with stage logging and metadata tracking.

Completed:

  • feat(pipeline): add sync command module to run extract/load/validate/transform in sequence
  • feat(pipeline): enforce single-month filtering for load via month-range parser integration
  • feat(pipeline): record sync lifecycle (running/success/failed) in _pipeline_runs

Files:

  • @packages/pipeline/@source/commands/sync.ts
  • @packages/pipeline/@source/index.ts
  • @packages/pipeline/@source/cli.ts

2026-02-23 14:04 - T-006

Overview: Implemented a transactional publish command to sync monthly orders from DuckDB into LibSQL operational tables.

Completed:

  • feat(pipeline): add publish command flow with DuckDB read + LibSQL write transaction
  • feat(pipeline): scope month refresh with transactional delete-then-insert semantics
  • feat(pipeline): track publish lifecycle and row count in _pipeline_runs

Decisions:

  • publish resolves lookup IDs at runtime from LibSQL reference tables to stay compatible with seeded source terms and names.

Files:

  • @packages/pipeline/@source/commands/publish.ts
  • @packages/pipeline/@source/publish/orders.ts
  • @packages/pipeline/@source/index.ts
  • @packages/pipeline/@source/cli.ts

2026-02-23 14:04 - T-010

Overview: Added person dimension resolver for publish flow with in-transaction caching.

Completed:

  • feat(pipeline): resolve/create db_person records from int_orders student identity data
  • feat(pipeline): cache person lookups by organization + name/id to avoid duplicate inserts

Files:

  • @packages/pipeline/@source/publish/dimensions.ts

2026-02-23 14:04 - T-011

Overview: Added organization dimension resolver for publish flow with entity-scoped lookup/create behavior.

Completed:

  • feat(pipeline): resolve/create db_organization by entity and normalized organization name
  • feat(pipeline): cache organization IDs during publish transaction

Files:

  • @packages/pipeline/@source/publish/dimensions.ts

2026-02-23 14:04 - T-012

Overview: Added account resolver to ensure each published order has a person-organization account FK.

Completed:

  • feat(pipeline): resolve/create db_account records linking person and organization
  • feat(pipeline): cache account lookups within publish transaction

Files:

  • @packages/pipeline/@source/publish/dimensions.ts
  • @packages/pipeline/@source/commands/publish.ts

2026-02-23 14:14 - T-007

Overview: Implemented unified report command orchestration that builds report JSON and runs present generation in one flow.

Completed:

  • feat(pipeline): add report command with monthly/quarterly/yearly type support
  • feat(pipeline): wire report lifecycle tracking into _pipeline_runs
  • feat(pipeline): run @services/present automatically after report JSON generation

Files:

  • @packages/pipeline/@source/commands/report.ts
  • @packages/pipeline/@source/index.ts
  • @packages/pipeline/@source/cli.ts

2026-02-23 14:14 - T-008

Overview: Added range-based report aggregation for quarterly/yearly outputs with quarter/range labeling.

Completed:

  • feat(pipeline): add multi-month aggregation for revenue/program/channel sections
  • feat(pipeline): add output naming contract for monthly/quarterly/yearly JSON artifacts
  • feat(pipeline): include period type/range metadata in report output

Files:

  • @packages/pipeline/@source/format/report.ts
  • @packages/pipeline/@source/format/types.ts

2026-02-23 14:14 - T-009

Overview: Implemented report staleness checks and source hash tracking with optional --force bypass.

Completed:

  • feat(pipeline): add staleness validator to warn when transform/extract freshness is outdated
  • feat(pipeline): add source file hashing utility and persist extract hashes in _pipeline_runs
  • test(pipeline): add CLI parsing coverage for --force report flag

Files:

  • @packages/pipeline/@source/staleness.ts
  • @packages/pipeline/@source/source-hash.ts
  • @packages/pipeline/@source/index.ts
  • @packages/pipeline/@source/commands/sync.ts
  • @packages/pipeline/@source/tests/config.test.ts

2026-02-23 14:14 - T-016

Overview: Deprecated run command and wired it as alias to sync with explicit warning output.

Completed:

  • refactor(pipeline): replace legacy run execution path with sync alias behavior
  • docs(pipeline): update CLI usage output to mark run as deprecated

Files:

  • @packages/pipeline/@source/index.ts

2026-02-23 14:14 - T-003

Overview: Updated root package scripts to expose seed, sync, publish, and report shortcuts.

Completed:

  • chore(*): add root-level pipeline workflow shortcuts for bootstrap, sync, publish, and report
  • chore(*): repoint sync to unified pipeline sync command and keep existing debug scripts available

Files:

  • package.json

2026-02-23 14:14 - T-013

Overview: Updated global AGENTS instructions to document unified pipeline workflow and deprecated alias behavior.

Completed:

  • docs(ai): update .ai/SYSTEM.md pipeline command section for seed/sync/publish/report
  • sync(ai): regenerate root AGENTS.md and CLAUDE.md via pnpm ai:sync

Files:

  • .ai/SYSTEM.md
  • AGENTS.md
  • CLAUDE.md

2026-02-23 14:14 - T-014

Overview: Updated architecture deployment workflow with clear bootstrap, recurring refresh, and on-demand report flows.

Completed:

  • docs(plan): rewrite Phase 1 — Local in architecture docs to use new command model
  • docs(plan): align pipeline package description with unified command surface

Files:

  • @plan/architecture.md

2026-02-23 14:14 - T-015

Overview: Updated package-level pipeline AGENTS documentation for new commands and file layout.

Completed:

  • docs(pipeline): refresh command examples and root usage snippets
  • docs(pipeline): update file tree and key patterns for commands/meta/publish/staleness modules

Files:

  • @packages/pipeline/AGENTS.md

Pipeline Workflow Unification

Previous Page

Review

Next Page

On this page

Progress
2026-02-23 13:51 - T-001
2026-02-23 13:51 - T-004
2026-02-23 13:58 - T-005
2026-02-23 13:58 - T-002
2026-02-23 14:04 - T-006
2026-02-23 14:04 - T-010
2026-02-23 14:04 - T-011
2026-02-23 14:04 - T-012
2026-02-23 14:14 - T-007
2026-02-23 14:14 - T-008
2026-02-23 14:14 - T-009
2026-02-23 14:14 - T-016
2026-02-23 14:14 - T-003
2026-02-23 14:14 - T-013
2026-02-23 14:14 - T-014
2026-02-23 14:14 - T-015