Pipeline Testing and Data Deployment

Overview

End-to-end test the new pipeline commands from Plan 012 (seed, sync, publish, report), update documentation to reflect current state, and deploy operational data to all environments (Local, Dev Turso, Prod Turso).

Goals

• Verify all new pipeline commands work correctly (seed, sync, publish, report)
• Load historical data (2023-2026) into DuckDB via sync
• Publish all available monthly data to LibSQL (local, dev, prod)
• Update @plan/ documentation (state.md, workflow.md, quickstart.md) to reflect Plan 012 changes
• Smoke test dashboards in all environments
• Commit Plan 012 changes with proper documentation

Non-Goals

• Modifying the pipeline code (Plan 012 already completed)
• Adding new features
• EPN entity data (IONS only for now)
• Weekly reports (not implemented yet)

Phases

• Phase 1: Local end-to-end testing of new commands
• Phase 2: Documentation updates
• Phase 3: Remote environment setup and data deployment
• Phase 4: Commit and wrap-up

Success

• pnpm seed --entity ions succeeds (case-insensitive)
• pnpm sync loads 2023-2026 data into DuckDB
• pnpm publish populates LibSQL for all months with data
• pnpm report --type monthly generates IONS-2026-02.json + PPTX/PDF
• pnpm report --type quarterly generates IONS-2026-Q1.json + PPTX/PDF
• Deprecated run command shows warning
• _pipeline_runs table populated with execution metadata
• @plan/state.md, workflow.md, quickstart.md updated
• Dev and Prod Turso databases have published data
• Dev and Prod dashboards show data
• All Plan 012 changes committed

Requirements

• Plan 012 code changes (uncommitted, ready to test)
• Source xlsx files in @source/raw/ (2023-2026)
• Local DuckDB and LibSQL databases
• Dev Turso credentials in @services/api/.env.dev
• Prod Turso credentials in @services/api/.env.prod
• Network access to Turso and deployed dashboards

Context

Why This Approach

• Testing before commit ensures Plan 012 implementation is correct
• Loading all historical data provides complete dataset for reports
• Deploying to all environments ensures dashboard shows real data
• Documentation update completes the Plan 012 deliverables

Key Constraints

• 2023-2024 data extracted to CSV but not yet loaded to DuckDB
• Remote Turso databases need schema verification before publish
• Dashboard URLs may have DNS issues from automation host (known issue from Plan 011)

Edge Cases

• Months with no data should be skipped gracefully in publish
• Quarterly report with partial data (month 3 has only 4 orders) should still work
• Remote publish failures should not corrupt local state

Tradeoffs

• Full historical load takes longer but provides complete dataset
• Publishing all months to all environments ensures consistency

Skills

• None required — execution and verification work

Boundaries

• Always: Verify local success before deploying to remote
• Always: Source environment credentials from .env files
• Always: Run drizzle-kit push before remote publish to ensure schema
• Ask first: Before pushing to production Turso
• Never: Modify Plan 012 code (only test it)
• Never: Commit without successful local testing

Questions

• Historical data scope? → Full load (2023-2026)
• Target environments? → Local + Dev + Prod