Plans013 2026 02 23 Pipeline Testing and Data Deployment
Phase 1: Local End-to-End Testing
- Purpose: Verify Plan 012 pipeline commands work correctly
T-001 - Test seed command
Test that seed command works with lowercase entity input.
- Status: completed
- Priority: P0
- Dependencies: none
Acceptance
-
pnpm seed --entity ionssucceeds (lowercase normalized to IONS) - LibSQL lookup tables populated
- Console shows
[seed] Completed lookup seeding for IONS
Notes
- Seed is idempotent, safe to re-run
Files
- atlas-ops.db
T-002 - Test sync command (all years)
Load historical data from 2023-2026 into DuckDB.
- Status: completed
- Priority: P0
- Dependencies: T-001
Acceptance
-
pnpm sync --entity IONS --year 2023succeeds -
pnpm sync --entity IONS --year 2024succeeds -
pnpm sync --entity IONS --year 2025succeeds -
pnpm sync --entity IONS --year 2026succeeds - DuckDB int_orders has data for all years
-
_pipeline_runstable shows sync records
Notes
- Sync orchestrates: extract → load → validate → transform
- Each year's sync records metadata in _pipeline_runs
Files
- atlas.db
- @source/clean/*.csv
T-003 - Test publish command (all months)
Publish all available monthly data to local LibSQL.
- Status: completed
- Priority: P0
- Dependencies: T-002
Acceptance
- Publish succeeds for each month with data
- LibSQL commerce_order table populated
- LibSQL finance_transaction table populated
-
_pipeline_runstable shows publish records with row counts
Notes
- Available months: 2025 (6,7,10,11,12), 2026 (1,2,3)
- Month 3 only has 4 orders
Files
- atlas-ops.db
T-004 - Test report command (monthly + quarterly)
Generate monthly and quarterly reports.
- Status: completed
- Priority: P0
- Dependencies: T-003
Acceptance
-
pnpm report --entity IONS --year 2026 --month 2 --type monthlygenerates output/monthly/IONS-2026-02.json - PPTX and PDF files generated for monthly report
-
pnpm report --entity IONS --year 2026 --month 1-3 --type quarterlygenerates output/monthly/IONS-2026-Q1.json - PPTX and PDF files generated for quarterly report
-
_pipeline_runstable shows report records
Notes
- Report orchestrates: format → present
- Quarterly aggregates months 1-3
Files
- output/monthly/IONS-2026-02.json
- output/monthly/IONS-2026-Q1.json
T-005 - Test deprecated run command
Verify deprecated run command shows warning.
- Status: completed
- Priority: P1
- Dependencies: T-001
Acceptance
-
pnpm pipeline -- run --entity IONS --year 2026shows deprecation warning - Warning message says "use
syncinstead" - Command still executes (aliased to sync)
Notes
- Backward compatibility test
Files
- None
T-006 - Verify metadata and outputs
Check _pipeline_runs table and output files.
- Status: completed
- Priority: P1
- Dependencies: T-004
Acceptance
-
_pipeline_runstable has records for all stages (extract, load, validate, transform, publish, report) - Records have correct status, timestamps, row counts
- Output files exist and are valid JSON/PPTX/PDF
Notes
- Verification queries documented in Progress.md
Files
- atlas.db
- output/monthly/*
Phase 2: Documentation Updates
- Purpose: Update @plan/ docs to reflect Plan 012 changes
T-007 - Update state.md
Update project state with Plan 012 completion.
- Status: completed
- Priority: P0
- Dependencies: T-006
Acceptance
- Last Updated and Updated By updated
- Current Phase reflects workflow unification
- @packages/pipeline description updated in Business Packages table
- Pipeline Stages table reflects unified commands
- Plan 012 added to Recent Completed Work
- Key decisions added (frequency-based CLI, metadata tracking, staleness checks)
Files
- @plan/state.md
T-008 - Update workflow.md
Update workflow documentation with new commands.
- Status: completed
- Priority: P0
- Dependencies: T-006
Acceptance
- Quick Reference section uses new commands (seed, sync, publish, report)
- Stage 3 Seed command updated from sync:seed to seed
- Report command examples include --type flag
Files
- @plan/workflow.md
T-009 - Update quickstart.md
Simplify quickstart with new workflow.
- Status: completed
- Priority: P0
- Dependencies: T-006
Acceptance
- "Running the Full Pipeline" section simplified to 4 commands
- Old 7-step process replaced with seed → sync → publish → report
- Plan list updated to include Plan 012
Files
- @plan/quickstart.md
Phase 3: Remote Deployment
- Purpose: Deploy data to Dev and Prod Turso databases
T-010 - Deploy to Dev Turso
Push schema and data to Dev environment.
- Status: completed
- Priority: P0
- Dependencies: T-006
Acceptance
- Schema pushed via drizzle-kit push
- Seed command run against Dev Turso
- All months published to Dev Turso
- Dev dashboard shows data (if accessible)
Notes
- Source credentials from @services/api/.env.dev
- Use DATABASE_URL and TURSO_AUTH_TOKEN env vars
Files
- @services/api/.env.dev
T-011 - Deploy to Prod Turso
Push schema and data to Prod environment.
- Status: completed
- Priority: P0
- Dependencies: T-010
Acceptance
- Schema pushed via drizzle-kit push
- Seed command run against Prod Turso
- All months published to Prod Turso
- Prod dashboard shows data (if accessible)
Notes
- Source credentials from @services/api/.env.prod
- User confirmation before prod deployment
Files
- @services/api/.env.prod
Phase 4: Commit and Wrap-up
- Purpose: Finalize and commit all changes
T-012 - Commit Plan 012 and 013 changes
Create commit with all pipeline and documentation changes.
- Status: pending
- Priority: P0
- Dependencies: T-007, T-008, T-009, T-011
Acceptance
- All Plan 012 code changes staged
- All documentation updates staged
- Plan 012 and 013 artifacts staged
- Commit message follows conventional format
- Commit succeeds
Notes
- Single commit for Plan 012 implementation + testing/docs
Files
- All modified files from git status