Atlas Plan

Phase 1: CLI Restructure

Purpose: Reorganize command parsing, add new commands, normalize flags

T-001 - Refactor CLI argument parser

Add support for month ranges (1-3), case-insensitive entity/unit, and new --type flag.

Status: completed
Priority: P0
Dependencies: none

Acceptance

parseMonth("2") returns { start: 2, end: 2 }
parseMonth("1-3") returns { start: 1, end: 3 }
--entity ions normalized to IONS
--unit tm normalized to TM
--type flag parsed for report command
Unit tests pass

Notes

Month range validation: start <= end, both 1-12
Type values: monthly, quarterly, yearly

Files

@packages/pipeline/@source/index.ts
@packages/pipeline/@source/cli.ts (new, extract parsing logic)

T-002 - Add sync command

Create unified sync command that runs extract → load → validate → transform.

Status: completed
Priority: P0
Dependencies: T-001, T-005

Acceptance

pnpm pipeline -- sync --entity IONS --year 2026 executes full chain
pnpm pipeline -- sync --entity IONS --year 2026 --month 2 filters load to single month
Each stage logs start/completion
Pipeline run recorded in _pipeline_runs

Notes

Replaces current run command behavior
Transform step added (T-005 dependency)

Files

@packages/pipeline/@source/index.ts
@packages/pipeline/@source/commands/sync.ts (new)

T-003 - Update root package.json scripts

Add new root-level shortcuts for seed, sync, publish, report.

Status: completed
Priority: P1
Dependencies: T-002, T-006, T-007

Acceptance

pnpm seed --entity IONS works
pnpm sync --entity IONS --year 2026 works
pnpm publish --entity IONS --year 2026 --month 2 works
pnpm report --entity IONS --year 2026 --month 2 --type monthly works
Existing scripts still work (backward compatible)

Notes

Keep existing extract, validate, format, present as dev/debug commands

Files

package.json

Phase 2: Transform & Publish

Purpose: Implement missing pipeline stages

T-004 - Create _pipeline_runs table

Add DuckDB metadata table for tracking pipeline executions.

Status: completed
Priority: P0
Dependencies: none

Acceptance

Table created on first pipeline run if not exists
Schema includes: id, entity, year, month, stage, started_at, completed_at, status, error_message, source_file_hash, row_count
Index on (entity, year, stage, completed_at)
Helper functions: recordPipelineStart(), recordPipelineComplete(), getLastRun()

Notes

Use DuckDB sequences for auto-increment id

Files

@packages/pipeline/@source/meta.ts (new)
@packages/pipeline/@source/duck.ts (add table setup)

T-005 - Add transform command

Create wrapper for uv run dbt run execution.

Status: completed
Priority: P0
Dependencies: T-004

Acceptance

pnpm pipeline -- transform --entity IONS runs dbt
Shells out to uv run dbt run in @python/analytics directory
Captures exit code and stderr
Records run in _pipeline_runs
Supports --select passthrough for dbt model selection

Notes

dbt doesn't filter by entity at runtime (all data processed)
Entity param used for metadata tracking only

Files

@packages/pipeline/@source/commands/transform.ts (new)
@packages/pipeline/@source/index.ts

T-006 - Add publish command

Create DuckDB → LibSQL sync for operational tables.

Status: completed
Priority: P0
Dependencies: T-004

Acceptance

pnpm pipeline -- publish --entity IONS --year 2026 --month 2 syncs data
Reads from DuckDB int_orders, int_enrollments
Writes to LibSQL commerce_order, finance_transaction, db_person
Uses transaction (BEGIN → DELETE → INSERT → COMMIT)
Delete scoped to entity + year + month
Records run in _pipeline_runs
Handles FK resolution (lookup/create dimension records)

Notes

Delete + insert strategy (not upsert)
Must resolve: organization_name → db_organization, student_name → db_person

Files

@packages/pipeline/@source/commands/publish.ts (new)
@packages/pipeline/@source/publish/orders.ts (new)
@packages/pipeline/@source/publish/dimensions.ts (new)
@packages/pipeline/@source/index.ts

Phase 3: Report Unification

Purpose: Combine format + present into unified report command

T-007 - Create unified report command

Combine format and present into single report command with --type flag.

Status: completed
Priority: P0
Dependencies: T-001, T-004

Acceptance

pnpm pipeline -- report --entity IONS --year 2026 --month 2 --type monthly works
Runs format → present in sequence
Supports --type monthly, --type quarterly, --type yearly
Output paths: IONS-2026-02.json, IONS-2026-Q1.json, IONS-2026.json
Records run in _pipeline_runs