ETL Pipeline Documentation
1. L1A (Raw Data Ingestion)
Status: ✅ Supports Incremental Update
This script ingests raw JSON files from output_arena/ into database/L1A/L1A.sqlite.
Usage
# Standard Run (Incremental)
# Only processes new files that are not yet in the database.
python ETL/L1A.py
# Force Refresh
# Reprocesses ALL files, overwriting existing records.
python ETL/L1A.py --force
L1B demoparser2 -> L1B.sqlite
L2 L1A.sqlite (+L1b.sqlite) -> L2.sqlite
L3 Deep Dive