2.0.0 Alpha: Data Refinery
This commit is contained in:
+19
-10
@@ -1,16 +1,25 @@
|
||||
L1A 5eplay平台网页爬虫原始数据。
|
||||
# L1 Raw Match Store
|
||||
|
||||
## ETL Step 1:
|
||||
从原始json数据库提取到L1A级数据库中。
|
||||
`output_arena/*/iframe_network.json` -> `database/L1A/L1A.sqlite`
|
||||
L1 stores one complete 5E network capture per match without transforming its
|
||||
payload.
|
||||
|
||||
### 脚本说明
|
||||
- **脚本位置**: `ETL/L1A.py`
|
||||
- **功能**: 自动遍历 `output_arena` 目录下所有的 `iframe_network.json` 文件,提取原始内容并以 `match_id` (文件夹名) 为主键存入 `L1A.sqlite` 数据库的 `raw_iframe_network` 表中。
|
||||
## Runtime Files
|
||||
|
||||
### 运行方式
|
||||
使用项目指定的 Python 环境运行脚本:
|
||||
- Database: `database/L1/L1.db`
|
||||
- Builder: `database/L1/L1_Builder.py`
|
||||
- Input: `output_arena/<match_id>/iframe_network.json`
|
||||
- Primary key: `raw_iframe_network.match_id`
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
C:/ProgramData/anaconda3/python.exe ETL/L1A.py
|
||||
make l1
|
||||
make pipeline
|
||||
```
|
||||
|
||||
Normal ingestion is incremental. `--force` re-reads every capture currently
|
||||
present in `output_arena`.
|
||||
|
||||
`L1A.db` and the historical `database/L1A/L1A.sqlite` path are retired. L1B is
|
||||
reserved for a future demo-parser source and is not part of the runtime
|
||||
pipeline.
|
||||
|
||||
Reference in New Issue
Block a user