Files

Database Governance

The repository intentionally keeps SQLite for the current private-team scale. This directory separates four different responsibilities:

Layer Database Grain Owner
L1 L1/L1.db One raw network capture per match Import pipeline
L2 L2/L2.db Normalized match, player, round and event facts L2 Builder
L3 L3/L3.db Roster features and profile marts L3 Builder
Web Web/Web_App.sqlite Lineups, comments, jobs and editorial data Flask app

Rules

  1. Paths are defined only in database/paths.py.
  2. Schemas live next to their owning database.
  3. Builders may read the previous layer and write only their own layer.
  4. User-generated Web data is never restored as part of an ETL rollback.
  5. A full import must run through database/pipeline.py.
  6. Pipeline runs are serialized by database/.pipeline.lock.
  7. L1/L2/L3 are backed up before a full pipeline run.
  8. Missing metrics are stored as NULL, not fabricated zero values.
  9. Admin -> Data Integrity is the operational source of truth.
  10. Web schema changes increment Config.WEB_SCHEMA_VERSION.
  11. YRTV_DATA_DIR owns runtime databases; code schemas always stay in Git.
  12. Container and production deployments must keep runtime data outside the image and source checkout.

Entry Points

make l1          # Import output_arena JSON into L1
make l2          # Rebuild normalized facts
make l3          # Rebuild active-roster features
make pipeline    # Run L1 -> L2 -> L3 with backup and validation
make check       # Compile and run tests
make bootstrap   # Initialize an empty runtime data root
make prepare-data # Copy legacy data into runtime-data

Directory Policy

  • Without YRTV_DATA_DIR, L1/, L2/, L3/, Web/ remain the compatible development runtime.
  • With YRTV_DATA_DIR, those folders provide code and schemas while databases live under the configured external directory.
  • backups/: generated rollback snapshots; ignored by Git.
  • schema_bkp/: historical schema research only; not used at runtime.
  • L1B/: reserved demo-parser integration; not used at runtime.
  • L3/Roadmap/: historical design notes; not used at runtime.

Large-scale directory moves are deliberately deferred until the legacy builders no longer depend on their current module layout.