Jacky/JKTV-online
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

2.0.0 Alpha: Data Refinery

Browse Source
This commit is contained in:
Jacky
2026-08-08 21:31:56 +08:00
parent fa75081d4d
commit 562775e5db
48 changed files with 4172 additions and 661 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+3
View File
@@ -47,6 +47,7 @@ coverage.xml
local_settings.py
db.sqlite3
*.db.bak
instance/
@@ -66,6 +67,8 @@ venv.bak/
output/
output_arena/
database/backups/
database/.pipeline.lock
arena/
scripts/
experiment
Makefile
+32
View File
@@ -0,0 +1,32 @@
PYTHON := .venv/bin/python
.PHONY: install run test check l1 l2 l3 l3-all pipeline
install:
python3 -m venv .venv
$(PYTHON) -m pip install -r requirements.txt
run:
$(PYTHON) -m web.app
test:
$(PYTHON) -m unittest discover -v
check:
$(PYTHON) -m compileall -q web database tests wsgi.py
$(PYTHON) -m unittest discover -v
l1:
$(PYTHON) database/L1/L1_Builder.py
l2:
$(PYTHON) database/L2/L2_Builder.py
l3:
$(PYTHON) database/L3/L3_Builder.py
l3-all:
$(PYTHON) database/L3/L3_Builder.py --force
pipeline:
$(PYTHON) -c "from web.app import create_app; create_app(); from database.job_store import JobStore; from database.pipeline import run_pipeline; job_id = JobStore().create_job('manual_pipeline', created_by='cli'); print('job_id=', job_id); raise SystemExit(0 if run_pipeline(job_id) else 1)"
README.md
+248 -133
View File
@@ -1,154 +1,269 @@
# YRTV 项目说明 till 1.0.2hotfix
# YRTV 2.0.0 Alpha
## 项目概览
YRTV 是一个基于 CS2 比赛数据的综合分析与战队管理平台。它集成了数据采集、ETL 清洗建模、特征挖掘以及现代化的 Web 交互界面。
核心目标是为战队提供数据驱动的决策支持，包括战术分析、队员表现评估、阵容管理（Clubhouse）以及实时战术板功能。
YRTV 是面向固定 CS2 战队的私人数据站。它不是公共玩家排行榜，而是让队员拥有类似职业选手的个人主页，并为战队提供比赛档案、队内比较、阵容分析、对手情报和战术工具。
---
当前 Alpha 版本已经建立可重复运行的数据流水线、数据库治理和职业主页数据集市，重点服务 active roster。
您可以使用以下命令快速配置环境：
pip install -r requirements.txt
## 当前基线
数据来源与处理核心包括：
- 比赛页面的 iframe JSON 数据（`iframe_network.json`）
- 可选的 demo 文件（`.zip/.dem`）
- L1A/L2/L3 分层数据库建模与校验
- 208 场比赛
- 1,181 名采集到的玩家
- 2,080 条玩家比赛记录
- 4,315 个回合
- 33,560 条回合事件
- 38,423 条经济记录
- 9 名 active roster 队员
- 885 条 roster 逐场历史
- 76 条地图统计
- 236 条武器统计
- 54 条时间窗口统计
- 54 条个人职业纪录
- 24 项自动化测试通过
- 28 项数据完整性检查通过
## v3.0.0 Release 更新要点
- **核心算法升级**: 严格确立 Active Roster (Lineup 1) 为战队平均数据计算基准，修复了雷达图与平均数据的计算偏差。
- **Clubhouse 增强**:
- 布局优化为 3 列网格。
- 新增 **OVR (Overall Score)** 显示，优先展示真实评分 (Real Rating)，直观反映选手综合实力。
- **Tactics 系统**:
- 统一评分逻辑：全站优先采用 L3 `core_avg_rating2` (真实评分)，智能回退至 `basic_avg_rating`。
- Data Center 数据中心现在完整映射了 Utility、Trading 等高阶战术数据。
- **稳定性修复**: 修正了特征服务中的语法错误，增强了对缺失数据的鲁棒性处理。
数据规模会随导入变化，Admin 数据完整性中心显示的结果是运行时事实。
## Web 交互系统 (Core)
基于 Flask + TailwindCSS + Alpine.js 构建的现代化 Web 应用。
## 核心功能
### 核心功能模块
1. **Clubhouse (战队管理)**
- **Roster Management**: 拖拽式管理当前激活阵容 (Active Roster)。
- **Scout System**: 全库模糊搜索玩家，支持按 Rating/Matches/KD 排序筛选。
- **Contract System**: 模拟签约/解约流程 (Sign/Release)，管理战队资产。
- **Identity**: 统一的头像与 ID 显示逻辑 (SteamID/Name)，支持自动生成首字母头像。
### 玩家职业主页
2. **Tactics Board (战术终端)**
- **SPA 架构**: 基于 Alpine.js 的单页应用，无刷新切换四大功能区。
- **Board (战术板)**: 集成 Leaflet.js 的交互式地图，支持战术点位标记。
- **Data (数据中心)**: 实时查看全队近期数据表现，集成 Utility/Trading 等高阶战术指标。
- **Analysis (深度分析)**:
- **Chemistry**: 任意组合 (2-5人) 的共同比赛胜率与数据分析。
- **Depth**: 阵容深度与位置分析。
- **Economy (经济计算)**: 简单的经济局/长枪局计算器。
- Rating、K/D、ADR、KAST 等生涯数据
- Aim、Clutch、Pistol、Defense、Utility、Stability、Economy、Pace 八维能力
- 生涯、最近 10/20/30 场、最近 30/90 天阶段统计
- 可切换时间窗口的 Rating 趋势
- 最高 Rating、最多击杀、最高 ADR、最高 K/D、最多爆头和最长连胜
- 每项个人纪录可追溯到具体比赛
- 地图表现、比赛历史、Party 信息、队内排名和留言板
- 缺失或尚未实现的指标显示为 `N/A`，不使用伪造分数
3. **Match Center (比赛中心)**
- **List View**:
- 显示比赛平均 ELO。
- **Party Identification**: 自动识别组排车队 (👥 2-5)，并用颜色区分规模 (Indigo/Blue/Purple/Orange)。
- **Result Tracking**: 基于 "Our Team" (Active Roster) 的胜负判定 (VICTORY/DEFEAT/CIVIL WAR)。
- **Detail View**:
- 按 Rating 降序排列双方队员。
- 高亮显示组排关系。
- 集成 Round-by-Round 经济与事件详情。
### 比赛中心
4. **Player Profile (玩家档案)**
- 综合能力雷达图 (八维数据: Aim, Clutch, Pistol, Defense, Util, Stability, Economy, Pace)。
- 近期 Rating/KD/ADR 趋势折线图。
- 详细的历史比赛记录（含 Party info 与 Result）。
- 头像上传与管理。
- 比赛列表、地图、比分、平均 ELO 和己方结果
- Active roster 与 Party 识别
- 双方玩家表现和 Rating 排序
- Head-to-head 击杀矩阵
- 回合事件、经济和装备信息
- 原始比赛数据查看
## 自动化与运维
新增 `ETL/refresh.py` 自动化脚本，用于一键执行全量数据刷新：
- 自动清理旧数据库。
- 顺序执行 L1A -> L2 -> L3 构建。
- 自动处理 schema 迁移。
### 战队与战术
## 数据流程
1. **下载与落盘**
通过 `downloader/downloader.py` 抓取比赛页面数据，生成 `output_arena/<match_id>/iframe_network.json`，并可同时下载 demo 文件。
2. **L1A 入库（原始 JSON）**
`ETL/L1A.py` 将 `output_arena/*/iframe_network.json` 批量写入 `database/L1A/L1A.sqlite`。
3. **L2 入库（结构化事实表/维度表）**
`ETL/L2_Builder.py` 读取 L1A 数据，按 `database/L2/schema.sql` 构建维度表与事实表，生成 `database/L2/L2_Main.sqlite`。
4. **L3 入库（特征集市）**
`ETL/L3_Builder.py` 读取 L2 数据，计算 Basic 及 6 大挖掘能力维度特征，生成 `database/L3/L3_Features.sqlite`。
5. **质量校验与覆盖分析**
`ETL/verify/verify_L2.py` 与 `ETL/verify/verify_deep.py` 用于 L2 字段覆盖与逻辑检查。
- Active roster 管理
- 玩家搜索、签入和移出
- 2-5 人同队比赛与 Chemistry 分析
- 对手档案和真实交手记录
- 地图战术板、阵容数据中心和经济工具
- Wiki、玩家标签、备注和评论
### 数据运营
- Admin 上传 `iframe_network.json`
- 自动识别唯一 `g161-*` 比赛 ID
- JSON 结构、必要接口、哈希和重复比赛校验
- 后台执行 L1 → L2 → L3
- 实时查看作业阶段、进度、日志和耗时
- 数据完整性中心与 JSON 报告
## 快速开始
环境要求：
- macOS/Linux
- Python 3.9+
- SQLite 3
安装并启动：
```bash
make install
export SECRET_KEY='replace-with-a-random-secret'
export ADMIN_TOKEN='replace-with-an-admin-token'
make run
```
默认地址：
- 应用：`http://127.0.0.1:5000`
- Admin：`/admin/`
- 比赛导入：`/admin/import-match`
- 数据完整性：`/admin/data-integrity`
生产进程入口：
```bash
.venv/bin/gunicorn wsgi:app
```
## 常用命令
```bash
make run # 启动 Flask
make check # 编译检查 + 自动化测试
make pipeline # 备份后执行完整 L1 -> L2 -> L3
make l1 # 仅构建 L1
make l2 # 仅构建 L2
make l3 # 仅构建 active roster L3
make l3-all # 为全部采集玩家构建 L3
```
正常维护优先使用 `make pipeline`。单层命令主要用于开发和排错。
## 比赛导入
推荐从 Admin 页面上传完整的 `iframe_network.json`。
导入流程：
1. 验证 UTF-8 和 JSON 结构。
2. 从网络 URL 中提取唯一比赛 ID。
3. 检查 match 和 round 必要接口。
4. 计算 SHA256，拒绝相同数据重复导入。
5. 保存到 `output_arena/<match_id>/iframe_network.json`。
6. 创建 `etl_jobs` 作业。
7. 备份 L1/L2/L3。
8. 串行执行三个 Builder。
9. 验证目标比赛具有 10 名玩家和回合事实。
10. 成功提交；失败自动恢复备份。
仓库当前不包含自动访问 5E 网页的下载器，因此首页 URL 输入不会抓取数据。
## 数据架构
```text
iframe_network.json
|
v
L1 raw capture
|
v
L2 normalized facts
|
v
L3 roster features and profile marts
|
v
Flask services and player profiles
```
### L1：原始层
- 数据库：`database/L1/L1.db`
- Builder：`database/L1/L1_Builder.py`
- Grain：每场比赛一份完整网络抓包
- 核心表：`raw_iframe_network`
### L2：事实层
- 数据库：`database/L2/L2.db`
- Schema：`database/L2/schema.sql`
- Builder：`database/L2/L2_Builder.py`
- 核心表：
- `dim_players`
- `dim_maps`
- `fact_matches`
- `fact_match_teams`
- `fact_match_players`
- `fact_match_players_t`
- `fact_match_players_ct`
- `fact_rounds`
- `fact_round_events`
- `fact_round_player_economy`
### L3：特征与主页集市
- 数据库：`database/L3/L3.db`
- Schema：`database/L3/schema.sql`
- Builder：`database/L3/L3_Builder.py`
- 核心表：
- `dm_player_features`
- `dm_player_match_history`
- `dm_player_map_stats`
- `dm_player_weapon_stats`
- `dm_player_period_stats`
- `dm_player_records`
### Web：应用状态
- 数据库：`database/Web/Web_App.sqlite`
- Schema：`database/Web/schema.sql`
- 当前 schema version：2
- 保存 lineup、玩家备注、评论、Wiki、战术板、导入登记和 ETL 作业
## 数据库治理
- 所有运行路径集中定义在 `database/paths.py`
- 完整编排入口为 `database/pipeline.py`
- 同一时间只允许一个 pipeline
- Pipeline 运行前备份 L1/L2/L3
- 失败时恢复三层数据库，Web 作业日志继续保留
- 备份位于 `database/backups/`
- 自动保留最近 3 组备份
- Web schema 使用 `schema_migrations` 记录版本
- 高频玩家历史、Party、事件和经济查询具有专用索引
- 数据库和目录规则详见 `database/README.md`
## 数据质量
Admin 数据完整性中心检查：
- 四个 SQLite 数据库的 `quick_check`
- 必要表和 Web schema version
- 玩家比赛、回合事件的引用完整性
- 每场比赛玩家数量
- 玩家身份覆盖
- 高频查询索引
- Active roster 的 L3 特征覆盖
- 逐场历史与总场次一致性
- 真实队内 percentile
- 地图、武器、时间窗口和职业纪录集市
- 占位空间指标
- Web 外键、active lineup 和 pipeline 并发
- 备份数量与存储规模
运行测试：
```bash
make check
```
## 目录结构
```
```text
yrtv/
├── downloader/ # 下载器（抓取 iframe JSON 与 demo）
├── ETL/ # ETL 脚本
│ ├── L1A.py
│ ├── L2_Builder.py
│ ├── L3_Builder.py
│ ├── refresh.py # [NEW] 一键刷新脚本
│ └── verify/
├── database/ # SQLite 数据库存储
│ ├── L1A/
│ ├── L2/
│ ├── L3/
│ └── original_json_schema/
├── web/ # [NEW] Web 应用程序
│ ├── app.py # 应用入口
│ ├── routes/ # 路由 (matches, players, teams, tactics)
│ ├── services/ # 业务逻辑 (stats, web)
│ ├── templates/ # Jinja2 模板 (TailwindCSS + Alpine.js)
│ └── static/ # 静态资源 (CSS, JS, Uploads)
└── utils/
└── json_extractor/ # JSON Schema 抽取工具
├── database/
│ ├── L1/ # 原始抓包与 Builder
│ ├── L2/ # 事实层、Schema、Processor
│ ├── L3/ # 特征层、Schema、Processor
│ ├── Web/ # 应用数据库 Schema
│ ├── paths.py # 统一路径
│ ├── maintenance.py # 备份、恢复、健康检查
│ ├── job_store.py # ETL 作业状态
│ └── pipeline.py # 完整流水线
├── tests/ # 自动化测试
├── utils/ # JSON 结构分析工具
├── web/
│ ├── routes/
│ ├── services/
│ ├── templates/
│ └── static/
├── Makefile
├── requirements.txt
└── wsgi.py
```
## 环境要求
- Python 3.11.4+
- Flask, Jinja2
- Playwright（下载器依赖）
- pandas, numpy（数据处理依赖）
## Alpha 限制
## 数据库层级说明
### L1A
- **用途**：保存原始 iframe JSON
- **输入**：`output_arena/*/iframe_network.json`
- **输出**：`database/L1A/L1A.sqlite`
- **脚本**：`ETL/L1A.py`
- 当前主要数据源为 5E iframe 网络响应。
- 不包含自动网页下载器和 Demo parser。
- 认证仍是单一 Admin Token，适合私人部署，不适合开放注册。
- SQLite 适合当前单战队规模，不面向高并发多租户。
- 部分高级空间能力需要地图边界、路径和 Demo 数据，当前显示 `N/A`。
- `StatsService` 仍保留部分兼容逻辑，后续会继续按领域拆分。
### L1B
- **用途**：保存 demo 解析后的原始数据（由 demoparser2 产出）
- **输出**：`database/L1B/L1B.sqlite`
- 当前仓库提供目录与说明，解析流程需结合外部工具执行
## 版本
### L2
结构化事实表/维度表数据库，覆盖比赛、玩家、回合与经济等数据：
- **Schema**：`database/L2/schema.sql`
- **输出**：`database/L2/L2_Main.sqlite`
- **核心表**：
- `dim_players`、`dim_maps`
- `fact_matches`、`fact_match_teams`
- `fact_match_players`、`fact_match_players_t`、`fact_match_players_ct`
- `fact_rounds`、`fact_round_events`、`fact_round_player_economy`
当前版本：`2.0.0 Alpha`
### L3
玩家特征集市 (Player Features Data Mart)，聚合 Basic 及 6 大挖掘能力维度 (STA, BAT, HPS, PTL, T/CT, UTIL)。
- **Schema**：`database/L3/schema.sql`
- **输出**：`database/L3/L3_Features.sqlite`
- **脚本**：`ETL/L3_Builder.py`
- **核心表**：`dm_player_features` (玩家聚合画像)
## JSON Schema 抽取工具
用于分析大量 `iframe_network.json` 的字段结构与覆盖情况，支持动态 Key 归并与多格式输出。
输出内容通常位于 `output_reports/` 或 `database/original_json_schema/`，包括：
- `schema_summary.md`：结构概览
- `schema_flat.csv`：扁平字段列表
- `uncovered_features.csv`：未覆盖字段清单
## 数据源互斥说明
L2 中 `fact_matches.data_source_type` 用于区分数据来源与字段覆盖范围：
- `classic`：含 round_list 详细回合与坐标信息
- `leetify`：含 leetify 评分与经济信息
- `unknown`：无法识别来源
入库逻辑保持互斥：同一场比赛只会按其来源覆盖相应字段，避免重复或冲突。
这一版本的目标是建立可信、可恢复、可持续导入的私人战队 HLTV 基线，而不是冻结产品功能。
database/L1/L1A.db
View File
database/L1/L1_Builder.py
+15 -6
View File
@@ -14,13 +14,18 @@ import os
import json
import sqlite3
import glob
import argparse # Added
import argparse
import sys
# Paths
BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
OUTPUT_ARENA_DIR = os.path.join(BASE_DIR, 'output_arena')
DB_DIR = os.path.join(BASE_DIR, 'database', 'L1')
DB_PATH = os.path.join(DB_DIR, 'L1.db')
if BASE_DIR not in sys.path:
sys.path.insert(0, BASE_DIR)
from database.paths import L1_DB, L1_DIR, OUTPUT_ARENA
OUTPUT_ARENA_DIR = str(OUTPUT_ARENA)
DB_DIR = str(L1_DIR)
DB_PATH = str(L1_DB)
def init_db():
if not os.path.exists(DB_DIR):
@@ -65,6 +70,7 @@ def process_files():
count = 0
skipped = 0
errors = 0
for file_path in files:
try:
@@ -92,11 +98,14 @@ def process_files():
conn.commit()
except Exception as e:
errors += 1
print(f"Error processing {file_path}: {e}")
conn.commit()
conn.close()
print(f"Finished. Processed: {count}, Skipped: {skipped}.")
print(f"Finished. Processed: {count}, Skipped: {skipped}, Errors: {errors}.")
if errors:
raise RuntimeError(f"L1 ingestion failed for {errors} file(s)")
if __name__ == '__main__':
process_files()
database/L1/README.md
+19 -10
View File
@@ -1,16 +1,25 @@
L1A 5eplay平台网页爬虫原始数据。
# L1 Raw Match Store
## ETL Step 1:
从原始json数据库提取到L1A级数据库中。
`output_arena/*/iframe_network.json` -> `database/L1A/L1A.sqlite`
L1 stores one complete 5E network capture per match without transforming its
payload.
### 脚本说明
- **脚本位置**: `ETL/L1A.py`
- **功能**: 自动遍历 `output_arena` 目录下所有的 `iframe_network.json` 文件，提取原始内容并以 `match_id` (文件夹名) 为主键存入 `L1A.sqlite` 数据库的 `raw_iframe_network` 表中。
## Runtime Files
### 运行方式
使用项目指定的 Python 环境运行脚本：
- Database: `database/L1/L1.db`
- Builder: `database/L1/L1_Builder.py`
- Input: `output_arena/<match_id>/iframe_network.json`
- Primary key: `raw_iframe_network.match_id`
## Commands
```bash
C:/ProgramData/anaconda3/python.exe ETL/L1A.py
make l1
make pipeline
```
Normal ingestion is incremental. `--force` re-reads every capture currently
present in `output_arena`.
`L1A.db` and the historical `database/L1A/L1A.sqlite` path are retired. L1B is
reserved for a future demo-parser source and is not part of the runtime
pipeline.
database/L2/L2.db
BIN
View File
Binary file not shown.
database/L2/L2_Builder.py
+13 -3
View File
@@ -7,14 +7,20 @@ from dataclasses import dataclass, field
from typing import List, Dict, Optional, Any, Tuple
from datetime import datetime
PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
if PROJECT_ROOT not in sys.path:
sys.path.insert(0, PROJECT_ROOT)
from database.paths import L1_DB, L2_DB, L2_SCHEMA
# Setup logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
# Constants
L1A_DB_PATH = 'database/L1/L1.db'
L2_DB_PATH = 'database/L2/L2.db'
SCHEMA_PATH = 'database/L2/schema.sql'
L1A_DB_PATH = str(L1_DB)
L2_DB_PATH = str(L2_DB)
SCHEMA_PATH = str(L2_SCHEMA)
# --- Data Structures for Unification ---
@@ -1238,6 +1244,10 @@ def process_matches():
l1_conn.close()
l2_conn.close()
logger.info(f"\nDone. Processed {count} matches ({success_count} success, {error_count} errors).")
if error_count:
raise RuntimeError(
f"L2 build failed for {error_count}/{count} matches"
)
if __name__ == "__main__":
process_matches()
database/L2/schema.sql
+21
View File
@@ -636,3 +636,24 @@ SELECT
FROM fact_match_players fmp
JOIN fact_matches fm ON fmp.match_id = fm.match_id
GROUP BY fmp.steam_id_64, fm.map_name;
-- ==========================================
-- Operational query indexes
-- ==========================================
CREATE INDEX IF NOT EXISTS idx_match_players_player_match
ON fact_match_players(steam_id_64, match_id);
CREATE INDEX IF NOT EXISTS idx_match_players_match_team
ON fact_match_players(match_id, team_id, steam_id_64);
CREATE INDEX IF NOT EXISTS idx_match_players_party
ON fact_match_players(match_id, match_team_id, steam_id_64);
CREATE INDEX IF NOT EXISTS idx_round_events_victim
ON fact_round_events(victim_steam_id, match_id);
CREATE INDEX IF NOT EXISTS idx_economy_player_match
ON fact_round_player_economy(steam_id_64, match_id, round_num);
CREATE INDEX IF NOT EXISTS idx_matches_map_time
ON fact_matches(map_name, start_time DESC);
database/L3/L3.db
BIN
View File
Binary file not shown.
database/L3/L3_Builder.py
+489 -11
View File
@@ -6,6 +6,8 @@ import sqlite3
import json
import argparse
import concurrent.futures
from collections import defaultdict, deque
from typing import Optional
# Setup logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
@@ -15,10 +17,14 @@ logger = logging.getLogger(__name__)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) # Points to database/ directory
PROJECT_ROOT = os.path.dirname(BASE_DIR) # Points to project root
sys.path.insert(0, PROJECT_ROOT) # Add project root to Python path
L2_DB_PATH = os.path.join(BASE_DIR, 'L2', 'L2.db')
L3_DB_PATH = os.path.join(BASE_DIR, 'L3', 'L3.db')
WEB_DB_PATH = os.path.join(BASE_DIR, 'Web', 'Web_App.sqlite')
SCHEMA_PATH = os.path.join(BASE_DIR, 'L3', 'schema.sql')
from database.paths import L2_DB, L3_DB, L3_SCHEMA, WEB_DB
L2_DB_PATH = str(L2_DB)
L3_DB_PATH = str(L3_DB)
L3_BACKUP_PATH = f"{L3_DB_PATH}.bak"
WEB_DB_PATH = str(WEB_DB)
SCHEMA_PATH = str(L3_SCHEMA)
def _get_existing_columns(conn, table_name):
cur = conn.execute(f"PRAGMA table_info({table_name})")
@@ -76,7 +82,28 @@ def _get_team_players():
try:
conn = sqlite3.connect(WEB_DB_PATH)
cursor = conn.cursor()
cursor.execute("SELECT player_ids_json FROM team_lineups")
columns = {
row[1] for row in cursor.execute("PRAGMA table_info(team_lineups)")
}
if 'is_active' in columns:
cursor.execute(
"""
SELECT player_ids_json
FROM team_lineups
WHERE is_active = 1
ORDER BY created_at DESC, id DESC
LIMIT 1
"""
)
else:
cursor.execute(
"""
SELECT player_ids_json
FROM team_lineups
ORDER BY created_at DESC, id DESC
LIMIT 1
"""
)
rows = cursor.fetchall()
steam_ids = set()
@@ -150,7 +177,25 @@ def _build_player_record(steam_id: str):
"error": str(e),
}
def main(force_all: bool = False, workers: int = 1):
def _backup_l3_database(source_path=L3_DB_PATH, backup_path=L3_BACKUP_PATH):
if not os.path.exists(source_path):
return None
source = sqlite3.connect(source_path)
backup = sqlite3.connect(backup_path)
try:
source.backup(backup)
result = backup.execute("PRAGMA quick_check").fetchone()[0]
if result != 'ok':
raise RuntimeError(f"L3 backup quick_check failed: {result}")
finally:
source.close()
backup.close()
logger.info("L3 backup created at %s", backup_path)
return backup_path
def main(force_all: bool = False, workers: int = 1, create_backup: bool = True):
"""
Main L3 feature building pipeline using modular processors
"""
@@ -158,6 +203,9 @@ def main(force_all: bool = False, workers: int = 1):
logger.info("Starting L3 Builder with 5-Tier Architecture")
logger.info("========================================")
if create_backup:
_backup_l3_database()
# 1. Ensure Schema is up to date
init_db()
@@ -181,6 +229,7 @@ def main(force_all: bool = False, workers: int = 1):
conn_l3 = sqlite3.connect(L3_DB_PATH)
try:
conn_l3.execute("BEGIN IMMEDIATE")
cursor_l2 = conn_l2.cursor()
if force_all:
logger.info("Force mode enabled: building L3 for all players in L2.")
@@ -240,7 +289,6 @@ def main(force_all: bool = False, workers: int = 1):
)
success_count += 1
if processed_count % 2 == 0:
conn_l3.commit()
logger.info(f"Progress: {processed_count}/{total_players} ({success_count} success, {error_count} errors)")
else:
for idx, row in enumerate(players, 1):
@@ -268,10 +316,20 @@ def main(force_all: bool = False, workers: int = 1):
processed_count = idx
if processed_count % 2 == 0:
conn_l3.commit()
logger.info(f"Progress: {processed_count}/{total_players} ({success_count} success, {error_count} errors)")
# Final commit
if error_count:
raise RuntimeError(
f"L3 feature build failed for {error_count}/{total_players} players"
)
processed_ids = [str(row[0]) for row in players]
_update_percentiles(conn_l3, processed_ids)
_rebuild_auxiliary_marts(conn_l2, conn_l3, processed_ids)
quick_check = conn_l3.execute("PRAGMA quick_check").fetchone()[0]
if quick_check != 'ok':
raise RuntimeError(f"L3 quick_check failed before commit: {quick_check}")
conn_l3.commit()
logger.info("========================================")
@@ -283,9 +341,11 @@ def main(force_all: bool = False, workers: int = 1):
logger.info("========================================")
except Exception as e:
conn_l3.rollback()
logger.error(f"Fatal error during L3 build: {e}")
import traceback
traceback.print_exc()
raise
finally:
conn_l2.close()
@@ -313,7 +373,7 @@ def _get_round_count(steam_id: str, conn_l2: sqlite3.Connection) -> int:
def _upsert_features(conn_l3: sqlite3.Connection, steam_id: str, features: dict,
match_count: int, round_count: int, conn_l2: sqlite3.Connection | None,
match_count: int, round_count: int, conn_l2: Optional[sqlite3.Connection],
first_match_date=None, last_match_date=None):
"""
Insert or update player features in dm_player_features
@@ -353,12 +413,430 @@ def _upsert_features(conn_l3: sqlite3.Connection, steam_id: str, features: dict,
cursor_l3.execute(sql, values)
def _rebuild_auxiliary_marts(conn_l2, conn_l3, steam_ids):
"""Rebuild player-grain marts used by profiles and trend APIs."""
if not steam_ids:
return
logger.info("Rebuilding L3 match, map and weapon marts")
total_history = 0
total_maps = 0
total_weapons = 0
total_periods = 0
total_records = 0
for start in range(0, len(steam_ids), 400):
chunk = steam_ids[start:start + 400]
placeholders = ','.join('?' for _ in chunk)
for table in (
'dm_player_match_history',
'dm_player_map_stats',
'dm_player_weapon_stats',
'dm_player_period_stats',
'dm_player_records',
):
conn_l3.execute(
f"DELETE FROM {table} WHERE steam_id_64 IN ({placeholders})",
chunk,
)
history_rows = conn_l2.execute(
f"""
SELECT
mp.steam_id_64,
mp.match_id,
m.start_time,
mp.rating,
mp.kd_ratio,
mp.adr,
mp.kast,
mp.is_win,
m.map_name,
mp.kills,
mp.deaths,
mp.headshot_count,
(
SELECT AVG(teammate.rating)
FROM fact_match_players teammate
WHERE teammate.match_id = mp.match_id
AND teammate.team_id = mp.team_id
AND teammate.steam_id_64 != mp.steam_id_64
) AS teammate_avg_rating
FROM fact_match_players mp
JOIN fact_matches m ON m.match_id = mp.match_id
WHERE mp.steam_id_64 IN ({placeholders})
ORDER BY mp.steam_id_64, m.start_time, mp.match_id
""",
chunk,
).fetchall()
history_values = []
player_state = defaultdict(lambda: {
'sequence': 0,
'rating_sum': 0.0,
'recent': deque(maxlen=10),
})
for row in history_rows:
steam_id = str(row[0])
state = player_state[steam_id]
rating = float(row[3] or 0.0)
state['sequence'] += 1
state['rating_sum'] += rating
state['recent'].append(rating)
history_values.append((
steam_id,
row[1],
row[2],
state['sequence'],
row[3],
row[4],
row[5],
row[6],
row[7],
row[8],
None,
row[12],
state['rating_sum'] / state['sequence'],
sum(state['recent']) / len(state['recent']),
))
conn_l3.executemany(
"""
INSERT INTO dm_player_match_history (
steam_id_64, match_id, match_date, match_sequence,
rating, kd_ratio, adr, kast, is_win, map_name,
opponent_avg_elo, teammate_avg_rating,
cumulative_rating, rolling_10_rating
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
history_values,
)
total_history += len(history_values)
map_rows = conn_l2.execute(
f"""
SELECT
mp.steam_id_64,
m.map_name,
COUNT(*) AS matches,
SUM(CASE WHEN mp.is_win = 1 THEN 1 ELSE 0 END) AS wins,
AVG(mp.rating) AS avg_rating,
AVG(mp.kd_ratio) AS avg_kd,
AVG(mp.adr) AS avg_adr,
AVG(mp.kast) AS avg_kast,
MAX(mp.rating) AS best_rating,
MIN(mp.rating) AS worst_rating
FROM fact_match_players mp
JOIN fact_matches m ON m.match_id = mp.match_id
WHERE mp.steam_id_64 IN ({placeholders})
AND m.map_name IS NOT NULL
AND m.map_name != ''
GROUP BY mp.steam_id_64, m.map_name
""",
chunk,
).fetchall()
map_values = [
tuple(row[:4]) + (
(row[3] or 0) / row[2] if row[2] else 0.0,
) + tuple(row[4:])
for row in map_rows
]
conn_l3.executemany(
"""
INSERT INTO dm_player_map_stats (
steam_id_64, map_name, matches, wins, win_rate,
avg_rating, avg_kd, avg_adr, avg_kast,
best_rating, worst_rating
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
map_values,
)
total_maps += len(map_values)
round_counts = {
str(row[0]): int(row[1] or 0)
for row in conn_l2.execute(
f"""
SELECT steam_id_64, SUM(round_total)
FROM fact_match_players
WHERE steam_id_64 IN ({placeholders})
GROUP BY steam_id_64
""",
chunk,
)
}
weapon_rows = conn_l2.execute(
f"""
SELECT
attacker_steam_id,
weapon,
COUNT(*) AS total_kills,
SUM(CASE WHEN is_headshot = 1 THEN 1 ELSE 0 END) AS total_headshots,
COUNT(DISTINCT match_id || ':' || round_num) AS usage_rounds
FROM fact_round_events
WHERE event_type = 'kill'
AND attacker_steam_id IN ({placeholders})
AND weapon IS NOT NULL
AND weapon != ''
GROUP BY attacker_steam_id, weapon
""",
chunk,
).fetchall()
weapon_values = []
for row in weapon_rows:
rounds = round_counts.get(str(row[0]), 0)
kills = int(row[2] or 0)
headshots = int(row[3] or 0)
usage_rounds = int(row[4] or 0)
hs_rate = headshots / kills if kills else 0.0
usage_rate = usage_rounds / rounds if rounds else 0.0
kills_per_round = kills / rounds if rounds else 0.0
effectiveness = kills / usage_rounds if usage_rounds else 0.0
weapon_values.append((
str(row[0]),
row[1],
kills,
headshots,
hs_rate,
usage_rounds,
usage_rate,
kills_per_round,
effectiveness,
))
conn_l3.executemany(
"""
INSERT INTO dm_player_weapon_stats (
steam_id_64, weapon_name, total_kills, total_headshots,
hs_rate, usage_rounds, usage_rate,
avg_kills_per_round, effectiveness_score
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
weapon_values,
)
total_weapons += len(weapon_values)
period_values = _calculate_period_rows(history_rows)
conn_l3.executemany(
"""
INSERT INTO dm_player_period_stats (
steam_id_64, period_key, period_label,
period_start, period_end, matches, wins, win_rate,
avg_rating, avg_kd, avg_adr, avg_kast,
total_kills, total_deaths, sample_reliable
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
period_values,
)
total_periods += len(period_values)
record_values = _calculate_record_rows(history_rows)
conn_l3.executemany(
"""
INSERT INTO dm_player_records (
steam_id_64, record_key, record_label, record_value,
match_id, map_name, match_date
) VALUES (?, ?, ?, ?, ?, ?, ?)
""",
record_values,
)
total_records += len(record_values)
logger.info(
"Auxiliary marts rebuilt: %s history, %s map, %s weapon, "
"%s period, %s record rows",
total_history,
total_maps,
total_weapons,
total_periods,
total_records,
)
def _group_player_match_rows(history_rows):
grouped = defaultdict(list)
for row in history_rows:
grouped[str(row['steam_id_64'])].append(row)
for rows in grouped.values():
rows.sort(key=lambda row: (row['start_time'] or 0, row['match_id']))
return grouped
def _safe_average(rows, key):
values = [float(row[key]) for row in rows if row[key] is not None]
return sum(values) / len(values) if values else None
def _calculate_period_rows(history_rows):
result = []
for steam_id, all_rows in _group_player_match_rows(history_rows).items():
latest_time = max(int(row['start_time'] or 0) for row in all_rows)
period_groups = [
('career', '生涯', all_rows),
('last_10', '最近 10 场', all_rows[-10:]),
('last_20', '最近 20 场', all_rows[-20:]),
('last_30', '最近 30 场', all_rows[-30:]),
(
'days_30',
'最近 30 天',
[
row for row in all_rows
if int(row['start_time'] or 0) >= latest_time - 30 * 86400
],
),
(
'days_90',
'最近 90 天',
[
row for row in all_rows
if int(row['start_time'] or 0) >= latest_time - 90 * 86400
],
),
]
for period_key, period_label, rows in period_groups:
if not rows:
continue
matches = len(rows)
wins = sum(1 for row in rows if row['is_win'])
kills = sum(int(row['kills'] or 0) for row in rows)
deaths = sum(int(row['deaths'] or 0) for row in rows)
result.append((
steam_id,
period_key,
period_label,
min(int(row['start_time'] or 0) for row in rows),
max(int(row['start_time'] or 0) for row in rows),
matches,
wins,
wins / matches,
_safe_average(rows, 'rating'),
kills / deaths if deaths else float(kills),
_safe_average(rows, 'adr'),
_safe_average(rows, 'kast'),
kills,
deaths,
1 if matches >= 10 else 0,
))
return result
def _calculate_record_rows(history_rows):
result = []
metric_definitions = (
('highest_rating', '最高 Rating', 'rating'),
('most_kills', '单场最多击杀', 'kills'),
('highest_adr', '单场最高 ADR', 'adr'),
('highest_kd', '单场最高 K/D', 'kd_ratio'),
('most_headshots', '单场最多爆头', 'headshot_count'),
)
for steam_id, rows in _group_player_match_rows(history_rows).items():
for record_key, record_label, field in metric_definitions:
candidates = [row for row in rows if row[field] is not None]
if not candidates:
continue
best = max(
candidates,
key=lambda row: (
float(row[field]),
int(row['start_time'] or 0),
),
)
result.append((
steam_id,
record_key,
record_label,
float(best[field]),
best['match_id'],
best['map_name'],
best['start_time'],
))
longest_streak = 0
current_streak = 0
streak_end = None
for row in rows:
if row['is_win']:
current_streak += 1
if current_streak >= longest_streak:
longest_streak = current_streak
streak_end = row
else:
current_streak = 0
if streak_end is not None:
result.append((
steam_id,
'longest_win_streak',
'最长连胜',
float(longest_streak),
streak_end['match_id'],
streak_end['map_name'],
streak_end['start_time'],
))
return result
def _update_percentiles(conn_l3, steam_ids):
"""Calculate a real percentile among eligible players in this build."""
if not steam_ids:
return
score_rows = []
for start in range(0, len(steam_ids), 400):
chunk = steam_ids[start:start + 400]
placeholders = ','.join('?' for _ in chunk)
conn_l3.execute(
f"""
UPDATE dm_player_features
SET tier_percentile = NULL
WHERE steam_id_64 IN ({placeholders})
""",
chunk,
)
score_rows.extend(conn_l3.execute(
f"""
SELECT steam_id_64, score_overall
FROM dm_player_features
WHERE steam_id_64 IN ({placeholders})
AND score_overall > 0
""",
chunk,
).fetchall())
if not score_rows:
return
scores = [float(row[1]) for row in score_rows]
percentile_values = []
for row in score_rows:
score = float(row[1])
percentile = sum(value <= score for value in scores) / len(scores) * 100
percentile_values.append((round(percentile, 2), str(row[0])))
conn_l3.executemany(
"""
UPDATE dm_player_features
SET tier_percentile = ?
WHERE steam_id_64 = ?
""",
percentile_values,
)
logger.info("Updated percentiles for %s eligible players", len(score_rows))
def _parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("--force", action="store_true")
parser.add_argument("--workers", type=int, default=1)
parser.add_argument("--no-backup", action="store_true")
return parser.parse_args()
if __name__ == "__main__":
args = _parse_args()
main(force_all=args.force, workers=args.workers)
main(
force_all=args.force,
workers=args.workers,
create_backup=not args.no_backup,
)
database/L3/processors/composite_processor.py
+20 -26
View File
@@ -65,8 +65,8 @@ class CompositeProcessor(BaseFeatureProcessor):
# Classify tier based on overall score
features['tier_classification'] = CompositeProcessor._classify_tier(features['score_overall'])
# Percentile rank (placeholder - requires all players)
features['tier_percentile'] = min(features['score_overall'], 100.0)
# Filled by L3_Builder after every eligible player has been calculated.
features['tier_percentile'] = None
return features
@@ -266,13 +266,13 @@ class CompositeProcessor(BaseFeatureProcessor):
STABILITY Score (0-100) | 8%
"""
# Extract features
volatility = features.get('meta_rating_volatility', 0.0)
loss_rating = features.get('meta_loss_rating', 0.0)
consistency = features.get('meta_rating_consistency', 0.0)
tilt_resilience = features.get('int_pressure_tilt_resistance', 0.0)
map_stable = features.get('meta_map_stability', 0.0)
elo_stable = features.get('meta_elo_tier_stability', 0.0)
recent_form = features.get('meta_recent_form_rating', 0.0)
volatility = features.get('meta_rating_volatility') or 0.0
loss_rating = features.get('meta_loss_rating') or 0.0
consistency = features.get('meta_rating_consistency') or 0.0
tilt_resilience = features.get('int_pressure_tilt_resistance') or 0.0
map_stable = features.get('meta_map_stability') or 0.0
elo_stable = features.get('meta_elo_tier_stability') or 0.0
recent_form = features.get('meta_recent_form_rating') or 0.0
# Normalize
# Volatility: Reverse score. 100 - (Vol * 220)
@@ -281,8 +281,8 @@ class CompositeProcessor(BaseFeatureProcessor):
loss_score = min((loss_rating / 1.00) * 100, 100)
cons_score = min((consistency / 70) * 100, 100)
tilt_score = min((tilt_resilience / 0.80) * 100, 100)
map_score = min((map_stable / 0.25) * 100, 100)
elo_score = min((elo_stable / 0.48) * 100, 100)
map_score = max(0, min(100, 100 - (map_stable / 0.25) * 100))
elo_score = max(0, min(100, 100 - (elo_stable / 0.48) * 100))
recent_score = min((recent_form / 1.15) * 100, 100)
# Weighted Sum
@@ -337,12 +337,12 @@ class CompositeProcessor(BaseFeatureProcessor):
PACE Score (0-100) | 5%
"""
# Extract features
early_kill_pct = features.get('int_timing_early_kill_share', 0.0)
aggression = features.get('int_timing_aggression_index', 0.0)
trade_speed = features.get('int_trade_response_time', 0.0)
trade_kill = features.get('int_trade_kill_count', 0)
teamwork = features.get('int_teamwork_score', 0.0)
first_contact = features.get('int_timing_first_contact_time', 0.0)
early_kill_pct = features.get('int_timing_early_kill_share') or 0.0
aggression = features.get('int_timing_aggression_index') or 0.0
trade_speed = features.get('int_trade_response_time') or 0.0
trade_kill = features.get('int_trade_kill_count') or 0
teamwork = features.get('int_teamwork_score') or 0.0
first_contact = features.get('int_timing_first_contact_time') or 0.0
# Normalize
early_score = min((early_kill_pct / 0.44) * 100, 100)
@@ -353,7 +353,7 @@ class CompositeProcessor(BaseFeatureProcessor):
if trade_speed > 0.01:
trade_speed_score = min((2.0 / trade_speed) * 100, 100)
else:
trade_speed_score = 100 # Instant trade
trade_speed_score = 0
trade_kill_score = min((trade_kill / 650) * 100, 100)
teamwork_score = min((teamwork / 29) * 100, 100)
@@ -362,13 +362,7 @@ class CompositeProcessor(BaseFeatureProcessor):
if first_contact > 0.01:
first_contact_score = min((30 / first_contact) * 100, 100)
else:
first_contact_score = 0 # If 0, probably no data, safe to say 0? Or 100?
# 0 first contact time means instant damage.
# But "30 / Contact" means smaller contact time gives higher score.
# If contact time is 0, score explodes.
# Realistically first contact time is > 0.
# I will clamp it.
first_contact_score = 100 # Assume very fast
first_contact_score = 0
# Weighted Sum
pace_score = (
@@ -416,5 +410,5 @@ def _get_default_composite_features() -> Dict[str, Any]:
'score_pace': 0.0,
'score_overall': 0.0,
'tier_classification': 'Beginner',
'tier_percentile': 0.0,
'tier_percentile': None,
}
database/L3/processors/intelligence_processor.py
+40 -52
View File
@@ -466,7 +466,8 @@ class IntelligenceProcessor(BaseFeatureProcessor):
- int_pos_spatial_iq_score
- int_pos_avg_distance_from_teammates
Note: Simplified implementation - full version requires DBSCAN clustering
Only geometry-independent values are calculated here. Metrics that
require map boundaries, paths or teammate positions remain NULL.
"""
cursor = conn_l2.cursor()
@@ -481,26 +482,23 @@ class IntelligenceProcessor(BaseFeatureProcessor):
has_position_data = cursor.fetchone()[0] > 0
if not has_position_data:
# Return placeholder values if no position data
return {
'int_pos_site_a_control_rate': 0.0,
'int_pos_site_b_control_rate': 0.0,
'int_pos_mid_control_rate': 0.0,
'int_pos_favorite_position': 'unknown',
'int_pos_position_diversity': 0.0,
'int_pos_rotation_speed': 0.0,
'int_pos_map_coverage': 0.0,
'int_pos_lurk_tendency': 0.0,
'int_pos_site_anchor_score': 0.0,
'int_pos_entry_route_diversity': 0.0,
'int_pos_retake_positioning': 0.0,
'int_pos_postplant_positioning': 0.0,
'int_pos_spatial_iq_score': 0.0,
'int_pos_avg_distance_from_teammates': 0.0,
'int_pos_site_a_control_rate': None,
'int_pos_site_b_control_rate': None,
'int_pos_mid_control_rate': None,
'int_pos_favorite_position': None,
'int_pos_position_diversity': None,
'int_pos_rotation_speed': None,
'int_pos_map_coverage': None,
'int_pos_lurk_tendency': None,
'int_pos_site_anchor_score': None,
'int_pos_entry_route_diversity': None,
'int_pos_retake_positioning': None,
'int_pos_postplant_positioning': None,
'int_pos_spatial_iq_score': None,
'int_pos_avg_distance_from_teammates': None,
}
# Simplified position analysis (proper implementation needs clustering)
# Calculate basic position variance as proxy for mobility
cursor.execute("""
SELECT
AVG(attacker_pos_x) as avg_x,
@@ -515,34 +513,24 @@ class IntelligenceProcessor(BaseFeatureProcessor):
pos_row = cursor.fetchone()
position_count = pos_row[3] if pos_row[3] else 1
# Position diversity based on unique grid cells visited
position_diversity = min(position_count / 50.0, 1.0) # Normalize to 0-1
# Map coverage (simplified)
map_coverage = position_diversity
# Site control rates CANNOT be calculated without map-specific geometry data
# Each map (Dust2, Mirage, Nuke, etc.) has different site boundaries
# Would require: CREATE TABLE map_boundaries (map_name, site_name, min_x, max_x, min_y, max_y)
# Commenting out these 3 features:
# - int_pos_site_a_control_rate
# - int_pos_site_b_control_rate
# - int_pos_mid_control_rate
return {
'int_pos_site_a_control_rate': 0.33, # Placeholder
'int_pos_site_b_control_rate': 0.33, # Placeholder
'int_pos_mid_control_rate': 0.34, # Placeholder
'int_pos_favorite_position': 'mid',
'int_pos_site_a_control_rate': None,
'int_pos_site_b_control_rate': None,
'int_pos_mid_control_rate': None,
'int_pos_favorite_position': None,
'int_pos_position_diversity': round(position_diversity, 3),
'int_pos_rotation_speed': 50.0,
'int_pos_rotation_speed': None,
'int_pos_map_coverage': round(map_coverage, 3),
'int_pos_lurk_tendency': 0.25,
'int_pos_site_anchor_score': 50.0,
'int_pos_lurk_tendency': None,
'int_pos_site_anchor_score': None,
'int_pos_entry_route_diversity': round(position_diversity, 3),
'int_pos_retake_positioning': 50.0,
'int_pos_postplant_positioning': 50.0,
'int_pos_retake_positioning': None,
'int_pos_postplant_positioning': None,
'int_pos_spatial_iq_score': round(position_diversity * 100, 2),
'int_pos_avg_distance_from_teammates': 500.0,
'int_pos_avg_distance_from_teammates': None,
}
@staticmethod
@@ -706,20 +694,20 @@ def _get_default_intelligence_features() -> Dict[str, Any]:
'int_pressure_big_moment_score': 0.0,
'int_pressure_tilt_resistance': 0.0,
# Position Mastery (14)
'int_pos_site_a_control_rate': 0.0,
'int_pos_site_b_control_rate': 0.0,
'int_pos_mid_control_rate': 0.0,
'int_pos_favorite_position': 'unknown',
'int_pos_position_diversity': 0.0,
'int_pos_rotation_speed': 0.0,
'int_pos_map_coverage': 0.0,
'int_pos_lurk_tendency': 0.0,
'int_pos_site_anchor_score': 0.0,
'int_pos_entry_route_diversity': 0.0,
'int_pos_retake_positioning': 0.0,
'int_pos_postplant_positioning': 0.0,
'int_pos_spatial_iq_score': 0.0,
'int_pos_avg_distance_from_teammates': 0.0,
'int_pos_site_a_control_rate': None,
'int_pos_site_b_control_rate': None,
'int_pos_mid_control_rate': None,
'int_pos_favorite_position': None,
'int_pos_position_diversity': None,
'int_pos_rotation_speed': None,
'int_pos_map_coverage': None,
'int_pos_lurk_tendency': None,
'int_pos_site_anchor_score': None,
'int_pos_entry_route_diversity': None,
'int_pos_retake_positioning': None,
'int_pos_postplant_positioning': None,
'int_pos_spatial_iq_score': None,
'int_pos_avg_distance_from_teammates': None,
# Trade Network (8)
'int_trade_kill_count': 0,
'int_trade_kill_rate': 0.0,
database/L3/processors/meta_processor.py
+34 -6
View File
@@ -60,10 +60,11 @@ class MetaProcessor(BaseFeatureProcessor):
# Get recent matches for volatility
cursor.execute("""
SELECT rating
FROM fact_match_players
WHERE steam_id_64 = ?
ORDER BY match_id DESC
SELECT p.rating
FROM fact_match_players p
JOIN fact_matches m ON m.match_id = p.match_id
WHERE p.steam_id_64 = ?
ORDER BY m.start_time DESC, p.match_id DESC
LIMIT 20
""", (steam_id,))
@@ -141,8 +142,35 @@ class MetaProcessor(BaseFeatureProcessor):
map_ratings = [row[1] for row in cursor.fetchall() if row[1] is not None]
map_stability = SafeAggregator.safe_stddev(map_ratings, 0.0)
# ELO tier stability (placeholder)
elo_tier_stability = rating_volatility # Simplified
cursor.execute("""
SELECT
CASE
WHEN p.origin_elo - opponent.avg_elo > 200 THEN 'lower'
WHEN p.origin_elo - opponent.avg_elo < -200 THEN 'higher'
ELSE 'similar'
END AS opponent_tier,
AVG(p.rating) AS avg_rating
FROM fact_match_players p
JOIN (
SELECT match_id, team_id, AVG(origin_elo) AS avg_elo
FROM fact_match_players
WHERE origin_elo IS NOT NULL
GROUP BY match_id, team_id
) opponent
ON opponent.match_id = p.match_id
AND opponent.team_id != p.team_id
WHERE p.steam_id_64 = ?
AND p.origin_elo IS NOT NULL
AND p.rating IS NOT NULL
GROUP BY opponent_tier
""", (steam_id,))
elo_tier_ratings = [
row[1] for row in cursor.fetchall() if row[1] is not None
]
elo_tier_stability = SafeAggregator.safe_stddev(
elo_tier_ratings,
0.0,
)
return {
'meta_rating_volatility': round(rating_volatility, 3),
database/L3/schema.sql
+52
View File
@@ -378,6 +378,56 @@ CREATE TABLE IF NOT EXISTS dm_player_weapon_stats (
CREATE INDEX IF NOT EXISTS idx_player_weapon_stats_player ON dm_player_weapon_stats(steam_id_64);
CREATE INDEX IF NOT EXISTS idx_player_weapon_stats_weapon ON dm_player_weapon_stats(weapon_name);
-- ============================================================================
-- Profile Mart: Time-window statistics
-- ============================================================================
CREATE TABLE IF NOT EXISTS dm_player_period_stats (
steam_id_64 TEXT NOT NULL,
period_key TEXT NOT NULL,
period_label TEXT NOT NULL,
period_start INTEGER,
period_end INTEGER,
matches INTEGER NOT NULL DEFAULT 0,
wins INTEGER NOT NULL DEFAULT 0,
win_rate REAL,
avg_rating REAL,
avg_kd REAL,
avg_adr REAL,
avg_kast REAL,
total_kills INTEGER NOT NULL DEFAULT 0,
total_deaths INTEGER NOT NULL DEFAULT 0,
sample_reliable BOOLEAN NOT NULL DEFAULT 0,
last_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (steam_id_64, period_key),
FOREIGN KEY (steam_id_64)
REFERENCES dm_player_features(steam_id_64) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_player_period_player
ON dm_player_period_stats(steam_id_64, period_key);
-- ============================================================================
-- Profile Mart: Career records linked to the source match
-- ============================================================================
CREATE TABLE IF NOT EXISTS dm_player_records (
steam_id_64 TEXT NOT NULL,
record_key TEXT NOT NULL,
record_label TEXT NOT NULL,
record_value REAL,
match_id TEXT,
map_name TEXT,
match_date INTEGER,
last_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (steam_id_64, record_key),
FOREIGN KEY (steam_id_64)
REFERENCES dm_player_features(steam_id_64) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_player_records_player
ON dm_player_records(steam_id_64, record_key);
-- ============================================================================
-- Schema Summary
-- ============================================================================
@@ -391,4 +441,6 @@ CREATE INDEX IF NOT EXISTS idx_player_weapon_stats_weapon ON dm_player_weapon_st
-- dm_player_match_history: Per-match snapshots for trend analysis
-- dm_player_map_stats: Map-level aggregations
-- dm_player_weapon_stats: Weapon usage statistics
-- dm_player_period_stats: Career/recent time-window aggregations
-- dm_player_records: Career record values and source matches
-- ============================================================================
database/README.md
+45
View File
@@ -0,0 +1,45 @@
# Database Governance
The repository intentionally keeps SQLite for the current private-team scale.
This directory separates four different responsibilities:
| Layer | Database | Grain | Owner |
|---|---|---|---|
| L1 | `L1/L1.db` | One raw network capture per match | Import pipeline |
| L2 | `L2/L2.db` | Normalized match, player, round and event facts | L2 Builder |
| L3 | `L3/L3.db` | Roster features and profile marts | L3 Builder |
| Web | `Web/Web_App.sqlite` | Lineups, comments, jobs and editorial data | Flask app |
## Rules
1. Paths are defined only in `database/paths.py`.
2. Schemas live next to their owning database.
3. Builders may read the previous layer and write only their own layer.
4. User-generated Web data is never restored as part of an ETL rollback.
5. A full import must run through `database/pipeline.py`.
6. Pipeline runs are serialized by `database/.pipeline.lock`.
7. L1/L2/L3 are backed up before a full pipeline run.
8. Missing metrics are stored as `NULL`, not fabricated zero values.
9. `Admin -> Data Integrity` is the operational source of truth.
10. Web schema changes increment `Config.WEB_SCHEMA_VERSION`.
## Entry Points
```bash
make l1 # Import output_arena JSON into L1
make l2 # Rebuild normalized facts
make l3 # Rebuild active-roster features
make pipeline # Run L1 -> L2 -> L3 with backup and validation
make check # Compile and run tests
```
## Directory Policy
- `L1/`, `L2/`, `L3/`, `Web/`: active code, schema and database.
- `backups/`: generated rollback snapshots; ignored by Git.
- `schema_bkp/`: historical schema research only; not used at runtime.
- `L1B/`: reserved demo-parser integration; not used at runtime.
- `L3/Roadmap/`: historical design notes; not used at runtime.
Large-scale directory moves are deliberately deferred until the legacy
builders no longer depend on their current module layout.
database/Web/Web_App.sqlite
BIN
View File
Binary file not shown.
database/Web/schema.sql
+95
View File
@@ -0,0 +1,95 @@
CREATE TABLE IF NOT EXISTS schema_migrations (
version INTEGER PRIMARY KEY,
description TEXT NOT NULL,
applied_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS comments (
id INTEGER PRIMARY KEY AUTOINCREMENT,
user_id TEXT,
username TEXT,
target_type TEXT NOT NULL,
target_id TEXT NOT NULL,
content TEXT NOT NULL,
likes INTEGER NOT NULL DEFAULT 0,
is_hidden INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_comments_target
ON comments(target_type, target_id, is_hidden, created_at DESC);
CREATE TABLE IF NOT EXISTS player_metadata (
steam_id_64 TEXT PRIMARY KEY,
notes TEXT,
tags TEXT NOT NULL DEFAULT '[]',
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS strategy_boards (
id INTEGER PRIMARY KEY AUTOINCREMENT,
title TEXT NOT NULL,
map_name TEXT NOT NULL,
data_json TEXT NOT NULL,
created_by TEXT,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS team_lineups (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
description TEXT,
player_ids_json TEXT NOT NULL DEFAULT '[]',
is_active INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_team_lineups_single_active
ON team_lineups(is_active)
WHERE is_active = 1;
CREATE TABLE IF NOT EXISTS wiki_pages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
path TEXT NOT NULL UNIQUE,
title TEXT NOT NULL,
content TEXT NOT NULL DEFAULT '',
updated_by TEXT,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS etl_jobs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
job_type TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'queued'
CHECK (status IN ('queued', 'running', 'succeeded', 'failed')),
match_id TEXT,
input_path TEXT,
current_stage TEXT,
progress INTEGER NOT NULL DEFAULT 0
CHECK (progress >= 0 AND progress <= 100),
message TEXT,
log_text TEXT NOT NULL DEFAULT '',
created_by TEXT,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
started_at TIMESTAMP,
finished_at TIMESTAMP,
duration_seconds REAL
);
CREATE INDEX IF NOT EXISTS idx_etl_jobs_created
ON etl_jobs(created_at DESC, id DESC);
CREATE INDEX IF NOT EXISTS idx_etl_jobs_status
ON etl_jobs(status, created_at);
CREATE TABLE IF NOT EXISTS match_imports (
match_id TEXT PRIMARY KEY,
content_sha256 TEXT NOT NULL,
source_path TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'queued',
job_id INTEGER,
imported_at TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (job_id) REFERENCES etl_jobs(id) ON DELETE SET NULL
);
database/__init__.py
+2
View File
@@ -0,0 +1,2 @@
"""Database builders, schemas, maintenance tools and local data stores."""
database/job_store.py
+198
View File
@@ -0,0 +1,198 @@
import sqlite3
from typing import Any, Dict, List, Optional
from database.paths import WEB_DB
class JobStore:
def __init__(self, database_path=WEB_DB):
self.database_path = str(database_path)
def _connect(self):
db = sqlite3.connect(self.database_path, timeout=30)
db.row_factory = sqlite3.Row
db.execute('PRAGMA busy_timeout = 30000')
db.execute('PRAGMA foreign_keys = ON')
return db
def create_job(
self,
job_type: str,
match_id: Optional[str] = None,
input_path: Optional[str] = None,
created_by: Optional[str] = None,
) -> int:
db = self._connect()
try:
cursor = db.execute(
"""
INSERT INTO etl_jobs (
job_type, match_id, input_path, created_by
) VALUES (?, ?, ?, ?)
""",
[job_type, match_id, input_path, created_by],
)
db.commit()
return int(cursor.lastrowid)
finally:
db.close()
def get_job(self, job_id: int) -> Optional[Dict[str, Any]]:
db = self._connect()
try:
row = db.execute(
'SELECT * FROM etl_jobs WHERE id = ?',
[job_id],
).fetchone()
return dict(row) if row else None
finally:
db.close()
def list_jobs(self, limit: int = 20) -> List[Dict[str, Any]]:
db = self._connect()
try:
rows = db.execute(
"""
SELECT *
FROM etl_jobs
ORDER BY created_at DESC, id DESC
LIMIT ?
""",
[max(1, min(int(limit), 100))],
).fetchall()
return [dict(row) for row in rows]
finally:
db.close()
def start_job(self, job_id: int, stage: str, message: str):
db = self._connect()
try:
db.execute(
"""
UPDATE etl_jobs
SET status = 'running',
current_stage = ?,
progress = 1,
message = ?,
started_at = CURRENT_TIMESTAMP
WHERE id = ? AND status = 'queued'
""",
[stage, message, job_id],
)
db.commit()
finally:
db.close()
def update_progress(
self,
job_id: int,
stage: str,
progress: int,
message: str,
):
db = self._connect()
try:
db.execute(
"""
UPDATE etl_jobs
SET current_stage = ?, progress = ?, message = ?
WHERE id = ?
""",
[stage, max(0, min(int(progress), 100)), message, job_id],
)
db.commit()
finally:
db.close()
def append_log(self, job_id: int, text: str):
if not text:
return
db = self._connect()
try:
db.execute(
"""
UPDATE etl_jobs
SET log_text = substr(log_text || ?, -100000)
WHERE id = ?
""",
[text, job_id],
)
db.commit()
finally:
db.close()
def finish_job(
self,
job_id: int,
succeeded: bool,
message: str,
duration_seconds: float,
):
status = 'succeeded' if succeeded else 'failed'
db = self._connect()
try:
db.execute(
"""
UPDATE etl_jobs
SET status = ?,
current_stage = ?,
progress = ?,
message = ?,
finished_at = CURRENT_TIMESTAMP,
duration_seconds = ?
WHERE id = ?
""",
[
status,
'complete' if succeeded else 'failed',
100 if succeeded else 0,
message,
round(float(duration_seconds), 3),
job_id,
],
)
db.execute(
"""
UPDATE match_imports
SET status = ?,
imported_at = CASE
WHEN ? = 'succeeded' THEN CURRENT_TIMESTAMP
ELSE imported_at
END,
updated_at = CURRENT_TIMESTAMP
WHERE job_id = ?
""",
[status, status, job_id],
)
db.commit()
finally:
db.close()
def upsert_match_import(
self,
match_id: str,
content_sha256: str,
source_path: str,
status: str,
job_id: int,
):
db = self._connect()
try:
db.execute(
"""
INSERT INTO match_imports (
match_id, content_sha256, source_path, status, job_id
) VALUES (?, ?, ?, ?, ?)
ON CONFLICT(match_id) DO UPDATE SET
content_sha256 = excluded.content_sha256,
source_path = excluded.source_path,
status = excluded.status,
job_id = excluded.job_id,
updated_at = CURRENT_TIMESTAMP
""",
[match_id, content_sha256, source_path, status, job_id],
)
db.commit()
finally:
db.close()
database/maintenance.py
+140
View File
@@ -0,0 +1,140 @@
from datetime import datetime, timezone
import json
from pathlib import Path
import shutil
import sqlite3
from typing import Dict
from database.paths import BACKUP_ROOT, L1_DB, L2_DB, L3_DB
MANAGED_DATABASES = {
'l1': L1_DB,
'l2': L2_DB,
'l3': L3_DB,
}
def quick_check(path: Path) -> str:
if not path.exists():
return 'missing'
db = sqlite3.connect(str(path))
try:
return str(db.execute('PRAGMA quick_check').fetchone()[0])
finally:
db.close()
def backup_database(source_path: Path, backup_path: Path):
if not source_path.exists():
raise FileNotFoundError(f'Database does not exist: {source_path}')
backup_path.parent.mkdir(parents=True, exist_ok=True)
source = sqlite3.connect(str(source_path))
destination = sqlite3.connect(str(backup_path))
try:
source.backup(destination)
result = destination.execute('PRAGMA quick_check').fetchone()[0]
if result != 'ok':
raise RuntimeError(
f'Backup quick_check failed for {source_path.name}: {result}'
)
finally:
source.close()
destination.close()
def restore_database(backup_path: Path, target_path: Path):
if not backup_path.exists():
raise FileNotFoundError(f'Backup does not exist: {backup_path}')
source = sqlite3.connect(str(backup_path))
target = sqlite3.connect(str(target_path), timeout=30)
try:
source.backup(target)
result = target.execute('PRAGMA quick_check').fetchone()[0]
if result != 'ok':
raise RuntimeError(
f'Restored quick_check failed for {target_path.name}: {result}'
)
finally:
source.close()
target.close()
def create_backup_set(label: str) -> Path:
safe_label = ''.join(
character for character in str(label)
if character.isalnum() or character in {'-', '_'}
)
if not safe_label:
raise ValueError('Backup label is empty after sanitization')
backup_dir = BACKUP_ROOT / safe_label
backup_dir.mkdir(parents=True, exist_ok=True)
manifest: Dict[str, object] = {
'label': safe_label,
'created_at': datetime.now(timezone.utc).isoformat(),
'databases': {},
}
for name, source_path in MANAGED_DATABASES.items():
backup_path = backup_dir / source_path.name
backup_database(source_path, backup_path)
manifest['databases'][name] = {
'source': str(source_path),
'backup': str(backup_path),
'size_bytes': backup_path.stat().st_size,
'quick_check': quick_check(backup_path),
}
with (backup_dir / 'manifest.json').open('w', encoding='utf-8') as file:
json.dump(manifest, file, ensure_ascii=True, indent=2)
return backup_dir
def restore_backup_set(backup_dir: Path):
for name, target_path in MANAGED_DATABASES.items():
backup_path = backup_dir / target_path.name
restore_database(backup_path, target_path)
def prune_backup_sets(keep: int = 3):
BACKUP_ROOT.mkdir(parents=True, exist_ok=True)
backup_dirs = sorted(
[path for path in BACKUP_ROOT.iterdir() if path.is_dir()],
key=lambda path: path.stat().st_mtime,
reverse=True,
)
removed = []
for path in backup_dirs[max(int(keep), 0):]:
shutil.rmtree(path)
removed.append(str(path))
return removed
def backup_storage_status():
BACKUP_ROOT.mkdir(parents=True, exist_ok=True)
backup_dirs = [path for path in BACKUP_ROOT.iterdir() if path.is_dir()]
total_bytes = sum(
file.stat().st_size
for directory in backup_dirs
for file in directory.rglob('*')
if file.is_file()
)
return {
'sets': len(backup_dirs),
'total_bytes': total_bytes,
}
def check_managed_databases():
return {
name: {
'path': str(path),
'exists': path.exists(),
'size_bytes': path.stat().st_size if path.exists() else 0,
'quick_check': quick_check(path),
}
for name, path in MANAGED_DATABASES.items()
}
database/paths.py
+36
View File
@@ -0,0 +1,36 @@
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent
DATABASE_ROOT = PROJECT_ROOT / 'database'
L1_DIR = DATABASE_ROOT / 'L1'
L2_DIR = DATABASE_ROOT / 'L2'
L3_DIR = DATABASE_ROOT / 'L3'
WEB_DIR = DATABASE_ROOT / 'Web'
L1_DB = L1_DIR / 'L1.db'
L2_DB = L2_DIR / 'L2.db'
L3_DB = L3_DIR / 'L3.db'
WEB_DB = WEB_DIR / 'Web_App.sqlite'
L2_SCHEMA = L2_DIR / 'schema.sql'
L3_SCHEMA = L3_DIR / 'schema.sql'
WEB_SCHEMA = WEB_DIR / 'schema.sql'
OUTPUT_ARENA = PROJECT_ROOT / 'output_arena'
BACKUP_ROOT = DATABASE_ROOT / 'backups'
PIPELINE_LOCK = DATABASE_ROOT / '.pipeline.lock'
def ensure_runtime_directories():
for path in (
L1_DIR,
L2_DIR,
L3_DIR,
WEB_DIR,
OUTPUT_ARENA,
BACKUP_ROOT,
):
path.mkdir(parents=True, exist_ok=True)
database/pipeline.py
+212
View File
@@ -0,0 +1,212 @@
import argparse
import fcntl
import os
import sqlite3
import subprocess
import sys
import time
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
from database.job_store import JobStore
from database.maintenance import (
check_managed_databases,
create_backup_set,
prune_backup_sets,
restore_backup_set,
)
from database.paths import L1_DB, L2_DB, PIPELINE_LOCK
STAGES = (
('l1', 15, Path('database/L1/L1_Builder.py')),
('l2', 55, Path('database/L2/L2_Builder.py')),
('l3', 85, Path('database/L3/L3_Builder.py')),
)
class PipelineError(RuntimeError):
pass
def _run_stage(store, job_id, stage, progress, script_path, replace=False):
store.update_progress(
job_id,
stage,
progress,
f'Running {stage.upper()} builder',
)
command = [sys.executable, str(PROJECT_ROOT / script_path)]
if stage == 'l1' and replace:
command.append('--force')
if stage == 'l3':
command.append('--no-backup')
started = time.monotonic()
result = subprocess.run(
command,
cwd=str(PROJECT_ROOT),
capture_output=True,
text=True,
timeout=1200,
)
duration = time.monotonic() - started
store.append_log(
job_id,
(
f'\n===== {stage.upper()} ({duration:.2f}s) =====\n'
f'{result.stdout}\n{result.stderr}'
),
)
if result.returncode != 0:
raise PipelineError(
f'{stage.upper()} builder exited with code {result.returncode}'
)
def _validate_pipeline_output(match_id=None):
database_status = check_managed_databases()
failures = [
f"{name}: {status['quick_check']}"
for name, status in database_status.items()
if status['quick_check'] != 'ok'
]
if failures:
raise PipelineError(
'Database quick_check failed: ' + ', '.join(failures)
)
if not match_id:
return
l1 = sqlite3.connect(str(L1_DB))
l2 = sqlite3.connect(str(L2_DB))
try:
raw_count = l1.execute(
'SELECT COUNT(*) FROM raw_iframe_network WHERE match_id = ?',
[match_id],
).fetchone()[0]
match_count = l2.execute(
'SELECT COUNT(*) FROM fact_matches WHERE match_id = ?',
[match_id],
).fetchone()[0]
player_count = l2.execute(
'SELECT COUNT(*) FROM fact_match_players WHERE match_id = ?',
[match_id],
).fetchone()[0]
round_count = l2.execute(
'SELECT COUNT(*) FROM fact_rounds WHERE match_id = ?',
[match_id],
).fetchone()[0]
finally:
l1.close()
l2.close()
if raw_count != 1:
raise PipelineError(f'L1 does not contain imported match {match_id}')
if match_count != 1:
raise PipelineError(f'L2 does not contain imported match {match_id}')
if player_count != 10:
raise PipelineError(
f'Imported match has {player_count} players; expected 10'
)
if round_count <= 0:
raise PipelineError('Imported match has no round facts')
def run_pipeline(job_id, match_id=None, replace=False):
store = JobStore()
job = store.get_job(job_id)
if not job:
raise PipelineError(f'Unknown ETL job: {job_id}')
started = time.monotonic()
backup_dir = None
PIPELINE_LOCK.parent.mkdir(parents=True, exist_ok=True)
lock_file = PIPELINE_LOCK.open('w')
try:
try:
fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError as exc:
raise PipelineError('Another database pipeline is already running') from exc
store.start_job(job_id, 'backup', 'Creating rollback snapshot')
backup_dir = create_backup_set(f'job-{job_id}')
store.append_log(job_id, f'Backup created: {backup_dir}\n')
for stage, progress, script_path in STAGES:
_run_stage(
store,
job_id,
stage,
progress,
script_path,
replace=replace,
)
store.update_progress(
job_id,
'validation',
95,
'Validating imported data',
)
_validate_pipeline_output(match_id)
removed_backups = prune_backup_sets(keep=3)
if removed_backups:
store.append_log(
job_id,
f"Pruned old backups: {', '.join(removed_backups)}\n",
)
duration = time.monotonic() - started
store.finish_job(
job_id,
True,
'Pipeline completed and validated',
duration,
)
return True
except Exception as exc:
store.append_log(job_id, f'\nPIPELINE FAILED: {exc}\n')
if backup_dir:
try:
restore_backup_set(backup_dir)
store.append_log(job_id, 'Rollback snapshot restored successfully\n')
except Exception as restore_exc:
store.append_log(
job_id,
f'ROLLBACK FAILED: {restore_exc}\n',
)
exc = PipelineError(f'{exc}; rollback also failed: {restore_exc}')
store.finish_job(
job_id,
False,
str(exc),
time.monotonic() - started,
)
return False
finally:
try:
fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN)
finally:
lock_file.close()
def _parse_args():
parser = argparse.ArgumentParser()
parser.add_argument('--job-id', type=int, required=True)
parser.add_argument('--match-id')
parser.add_argument('--replace', action='store_true')
return parser.parse_args()
if __name__ == '__main__':
args = _parse_args()
succeeded = run_pipeline(
args.job_id,
match_id=args.match_id,
replace=args.replace,
)
raise SystemExit(0 if succeeded else 1)
requirements.txt
+3 -7
View File
@@ -1,7 +1,3 @@
Flask
pandas
numpy
playwright
gunicorn
gevent
matplotlib
Flask>=3.0,<4
gunicorn>=21,<24
pytest>=8,<9
tests/__init__.py
+1
View File
@@ -0,0 +1 @@
tests/test_integration.py
+620
View File
@@ -0,0 +1,620 @@
import json
import io
import os
import shutil
import sqlite3
import tempfile
import unittest
from pathlib import Path
from web.config import Config
class ApplicationIntegrationTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.temp_dir = tempfile.TemporaryDirectory()
cls.original_web_path = Config.DB_WEB_PATH
Config.DB_WEB_PATH = os.path.join(cls.temp_dir.name, 'Web_App.sqlite')
shutil.copy2(cls.original_web_path, Config.DB_WEB_PATH)
from web.app import create_app
cls.app = create_app()
cls.app.config.update(TESTING=True)
cls.client = cls.app.test_client()
with sqlite3.connect(Config.DB_WEB_PATH) as db:
raw_ids = db.execute(
"""
SELECT player_ids_json
FROM team_lineups
WHERE is_active = 1
LIMIT 1
"""
).fetchone()[0]
cls.roster_ids = [str(value) for value in json.loads(raw_ids)]
@classmethod
def tearDownClass(cls):
Config.DB_WEB_PATH = cls.original_web_path
cls.temp_dir.cleanup()
def test_primary_pages_render(self):
paths = [
'/',
'/matches/',
'/players/',
f'/players/{self.roster_ids[0]}',
'/teams/',
'/tactics/',
'/opponents/',
]
for path in paths:
with self.subTest(path=path):
response = self.client.get(path)
self.assertEqual(response.status_code, 200)
self.assertGreater(len(response.data), 100)
def test_admin_integrity_page_and_json_render(self):
with self.client.session_transaction() as session:
session['is_admin'] = True
response = self.client.get('/admin/data-integrity')
self.assertEqual(response.status_code, 200)
self.assertIn('数据完整性中心'.encode('utf-8'), response.data)
response = self.client.get('/admin/data-integrity?format=json')
self.assertEqual(response.status_code, 200)
report = response.get_json()
self.assertIn(report['overall_status'], {'pass', 'warn', 'fail'})
self.assertGreater(report['counts']['matches'], 0)
self.assertEqual(
report['counts']['web_schema_version'],
Config.WEB_SCHEMA_VERSION,
)
response = self.client.get('/admin/import-match')
self.assertEqual(response.status_code, 200)
self.assertIn('比赛数据导入'.encode('utf-8'), response.data)
def test_duplicate_match_upload_is_rejected_without_starting_job(self):
from database.paths import L1_DB
with self.client.session_transaction() as session:
session['is_admin'] = True
with sqlite3.connect(str(L1_DB)) as db:
raw = db.execute(
"""
SELECT content
FROM raw_iframe_network
ORDER BY match_id
LIMIT 1
"""
).fetchone()[0]
response = self.client.post(
'/admin/import-match',
data={
'capture': (
io.BytesIO(raw.encode('utf-8')),
'iframe_network.json',
),
},
content_type='multipart/form-data',
follow_redirects=True,
)
self.assertEqual(response.status_code, 200)
self.assertIn(b'already imported with identical data', response.data)
def test_player_search_works_across_l2_and_l3(self):
response = self.client.get('/players/?search=jAck')
self.assertEqual(response.status_code, 200)
self.assertIn(b'jAckY0987', response.data)
def test_profile_keeps_all_primary_sections(self):
response = self.client.get(f'/players/{self.roster_ids[0]}')
self.assertEqual(response.status_code, 200)
for label in (
'近期表现走势',
'能力八维图',
'CORE (核心表现)',
'阶段表现',
'职业纪录',
'比赛记录',
'地图数据',
'留言板',
):
with self.subTest(label=label):
self.assertIn(label.encode('utf-8'), response.data)
def test_profile_period_api_and_trend_window(self):
steam_id = self.roster_ids[0]
response = self.client.get(
f'/players/{steam_id}/period_stats?period=last_20'
)
self.assertEqual(response.status_code, 200)
period = response.get_json()
self.assertEqual(period['period_key'], 'last_20')
self.assertEqual(period['matches'], 20)
self.assertEqual(period['sample_reliable'], 1)
response = self.client.get(
f'/players/{steam_id}/charts_data?period=last_10'
)
self.assertEqual(response.status_code, 200)
chart = response.get_json()
self.assertEqual(chart['period']['period_key'], 'last_10')
self.assertLessEqual(len(chart['trend']['labels']), 10)
def test_date_filter_uses_unix_timestamp_conversion(self):
from web.services.stats_service import StatsService
with self.app.app_context():
matches, total = StatsService.get_matches(
page=1,
per_page=20,
date_from='2025-01-01',
date_to='2026-12-31',
)
self.assertGreater(total, 0)
self.assertGreater(len(matches), 0)
def test_shared_matches_require_same_team(self):
from web.services.stats_service import StatsService
selected_ids = self.roster_ids[:2]
with self.app.app_context():
matches = StatsService.get_shared_matches(selected_ids)
self.assertGreater(len(matches), 0)
l2 = sqlite3.connect(Config.DB_L2_PATH)
try:
for match in matches:
placeholders = ','.join('?' for _ in selected_ids)
team_count = l2.execute(
f"""
SELECT COUNT(DISTINCT team_id)
FROM fact_match_players
WHERE match_id = ?
AND steam_id_64 IN ({placeholders})
""",
[match['match_id']] + selected_ids,
).fetchone()[0]
self.assertEqual(team_count, 1)
finally:
l2.close()
def test_opponent_list_only_contains_actual_opponents(self):
from web.services.opponent_service import OpponentService
with self.app.app_context():
opponents, total = OpponentService.get_opponent_list(
page=1,
per_page=20,
)
self.assertGreater(total, 0)
self.assertGreater(len(opponents), 0)
l2 = sqlite3.connect(Config.DB_L2_PATH)
try:
roster_ph = ','.join('?' for _ in self.roster_ids)
for opponent in opponents:
faced = l2.execute(
f"""
SELECT COUNT(*)
FROM fact_match_players opponent
JOIN fact_match_players roster
ON roster.match_id = opponent.match_id
AND roster.team_id != opponent.team_id
WHERE opponent.steam_id_64 = ?
AND roster.steam_id_64 IN ({roster_ph})
""",
[opponent['steam_id_64']] + self.roster_ids,
).fetchone()[0]
self.assertGreater(faced, 0)
finally:
l2.close()
class L3MartBuilderTests(unittest.TestCase):
def test_auxiliary_marts_build_on_database_copy(self):
from database.L3.L3_Builder import (
_get_team_players,
_rebuild_auxiliary_marts,
)
with tempfile.TemporaryDirectory() as temp_dir:
l3_path = os.path.join(temp_dir, 'L3.db')
shutil.copy2(Config.DB_L3_PATH, l3_path)
l2 = sqlite3.connect(Config.DB_L2_PATH)
l2.row_factory = sqlite3.Row
l3 = sqlite3.connect(l3_path)
try:
_rebuild_auxiliary_marts(
l2,
l3,
sorted(_get_team_players()),
)
l3.commit()
for table in (
'dm_player_match_history',
'dm_player_map_stats',
'dm_player_period_stats',
'dm_player_records',
'dm_player_weapon_stats',
):
count = l3.execute(
f'SELECT COUNT(*) FROM {table}'
).fetchone()[0]
self.assertGreater(count, 0)
self.assertEqual(
l3.execute('PRAGMA quick_check').fetchone()[0],
'ok',
)
finally:
l2.close()
l3.close()
def test_spatial_processor_does_not_emit_fake_geometry_metrics(self):
from database.L3.L3_Builder import _get_team_players
from database.L3.processors.intelligence_processor import IntelligenceProcessor
roster_ids = sorted(_get_team_players())
placeholders = ','.join('?' for _ in roster_ids)
l2 = sqlite3.connect(Config.DB_L2_PATH)
try:
row = l2.execute(
f"""
SELECT attacker_steam_id
FROM fact_round_events
WHERE attacker_steam_id IN ({placeholders})
AND attacker_pos_x IS NOT NULL
GROUP BY attacker_steam_id
ORDER BY COUNT(*) DESC
LIMIT 1
""",
roster_ids,
).fetchone()
self.assertIsNotNone(row)
features = IntelligenceProcessor._calculate_position_mastery(
str(row[0]),
l2,
)
finally:
l2.close()
for key in (
'int_pos_site_a_control_rate',
'int_pos_site_b_control_rate',
'int_pos_mid_control_rate',
'int_pos_rotation_speed',
'int_pos_lurk_tendency',
'int_pos_site_anchor_score',
'int_pos_retake_positioning',
'int_pos_postplant_positioning',
'int_pos_avg_distance_from_teammates',
):
with self.subTest(key=key):
self.assertIsNone(features[key])
self.assertIsNotNone(features['int_pos_position_diversity'])
def test_percentiles_are_calculated_from_peer_scores(self):
from database.L3.L3_Builder import _update_percentiles
with tempfile.TemporaryDirectory() as temp_dir:
l3_path = os.path.join(temp_dir, 'L3.db')
shutil.copy2(Config.DB_L3_PATH, l3_path)
l3 = sqlite3.connect(l3_path)
try:
player_ids = [
row[0] for row in l3.execute(
"""
SELECT steam_id_64
FROM dm_player_features
ORDER BY steam_id_64
LIMIT 3
"""
)
]
for steam_id, score in zip(player_ids, (10.0, 20.0, 30.0)):
l3.execute(
"""
UPDATE dm_player_features
SET score_overall = ?
WHERE steam_id_64 = ?
""",
[score, steam_id],
)
_update_percentiles(l3, player_ids)
values = [
row[0] for row in l3.execute(
f"""
SELECT tier_percentile
FROM dm_player_features
WHERE steam_id_64 IN ({','.join('?' for _ in player_ids)})
ORDER BY score_overall
""",
player_ids,
)
]
self.assertEqual(values, [33.33, 66.67, 100.0])
finally:
l3.close()
def test_l3_backup_is_a_valid_sqlite_database(self):
from database.L3.L3_Builder import _backup_l3_database
with tempfile.TemporaryDirectory() as temp_dir:
source = os.path.join(temp_dir, 'source.db')
backup = os.path.join(temp_dir, 'backup.db')
shutil.copy2(Config.DB_L3_PATH, source)
result_path = _backup_l3_database(source, backup)
self.assertEqual(result_path, backup)
with sqlite3.connect(backup) as db:
self.assertEqual(db.execute('PRAGMA quick_check').fetchone()[0], 'ok')
class FeatureFormulaTests(unittest.TestCase):
def test_empty_pace_inputs_do_not_receive_free_points(self):
from database.L3.processors.composite_processor import CompositeProcessor
self.assertEqual(CompositeProcessor._calculate_pace_score({}), 0.0)
def test_lower_map_and_elo_volatility_improves_stability_score(self):
from database.L3.processors.composite_processor import CompositeProcessor
common = {
'meta_rating_volatility': 0.2,
'meta_loss_rating': 1.0,
'meta_rating_consistency': 70,
'int_pressure_tilt_resistance': 0.8,
'meta_recent_form_rating': 1.15,
}
stable = dict(common, meta_map_stability=0.05, meta_elo_tier_stability=0.05)
volatile = dict(common, meta_map_stability=0.25, meta_elo_tier_stability=0.48)
self.assertGreater(
CompositeProcessor._calculate_stability_score(stable),
CompositeProcessor._calculate_stability_score(volatile),
)
def test_recent_form_uses_match_time_and_elo_stability_is_calculated(self):
from database.L3.L3_Builder import _get_team_players
from database.L3.processors.meta_processor import MetaProcessor
steam_id = sorted(_get_team_players())[0]
l2 = sqlite3.connect(Config.DB_L2_PATH)
try:
expected_rows = l2.execute(
"""
SELECT p.rating
FROM fact_match_players p
JOIN fact_matches m ON m.match_id = p.match_id
WHERE p.steam_id_64 = ?
ORDER BY m.start_time DESC, p.match_id DESC
LIMIT 10
""",
[steam_id],
).fetchall()
expected = sum(row[0] for row in expected_rows) / len(expected_rows)
features = MetaProcessor._calculate_stability(steam_id, l2)
finally:
l2.close()
self.assertAlmostEqual(
features['meta_recent_form_rating'],
round(expected, 3),
places=3,
)
self.assertGreaterEqual(features['meta_elo_tier_stability'], 0)
self.assertNotEqual(
features['meta_elo_tier_stability'],
features['meta_rating_volatility'],
)
class DatabaseGovernanceTests(unittest.TestCase):
def test_database_paths_are_absolute_and_exist(self):
from database.paths import L1_DB, L2_DB, L3_DB, WEB_DB
for path in (L1_DB, L2_DB, L3_DB, WEB_DB):
with self.subTest(path=path):
self.assertTrue(path.is_absolute())
self.assertTrue(path.exists())
def test_valid_capture_is_identified_from_network_urls(self):
from database.paths import L1_DB
from web.services.import_service import MatchImportService
with sqlite3.connect(str(L1_DB)) as db:
match_id, raw = db.execute(
"""
SELECT match_id, content
FROM raw_iframe_network
ORDER BY match_id
LIMIT 1
"""
).fetchone()
result = MatchImportService.validate_capture(raw.encode('utf-8'))
self.assertEqual(result['match_id'], match_id)
self.assertGreaterEqual(result['successful_responses'], 2)
def test_prepare_import_is_atomic_and_rejects_duplicate_queue(self):
import web.services.import_service as import_module
from database.paths import L1_DB
from web.services.import_service import (
DuplicateMatchError,
MatchImportService,
)
with sqlite3.connect(str(L1_DB)) as db:
original_id, raw = db.execute(
"""
SELECT match_id, content
FROM raw_iframe_network
ORDER BY match_id
LIMIT 1
"""
).fetchone()
new_id = 'g161-99999999999999999999999'
raw = raw.replace(original_id, new_id).encode('utf-8')
with tempfile.TemporaryDirectory() as temp_dir:
old_l1 = import_module.L1_DB
old_arena = import_module.OUTPUT_ARENA
old_web = Config.DB_WEB_PATH
fake_l1 = Path(temp_dir) / 'L1.db'
fake_web = Path(temp_dir) / 'Web_App.sqlite'
with sqlite3.connect(str(fake_l1)) as db:
db.execute(
"""
CREATE TABLE raw_iframe_network (
match_id TEXT PRIMARY KEY,
content TEXT
)
"""
)
with sqlite3.connect(str(fake_web)) as db:
with open(Config.DB_WEB_SCHEMA_PATH, 'r', encoding='utf-8') as schema:
db.executescript(schema.read())
import_module.L1_DB = fake_l1
import_module.OUTPUT_ARENA = Path(temp_dir) / 'output_arena'
Config.DB_WEB_PATH = str(fake_web)
try:
prepared = MatchImportService.prepare_import(
raw,
'iframe_network.json',
created_by='test',
)
self.assertEqual(prepared['match_id'], new_id)
self.assertTrue(Path(prepared['source_path']).exists())
with self.assertRaises(DuplicateMatchError):
MatchImportService.prepare_import(
raw,
'iframe_network.json',
created_by='test',
)
finally:
import_module.L1_DB = old_l1
import_module.OUTPUT_ARENA = old_arena
Config.DB_WEB_PATH = old_web
def test_pipeline_post_validation_accepts_current_databases(self):
from database.paths import L1_DB
from database.pipeline import _validate_pipeline_output
with sqlite3.connect(str(L1_DB)) as db:
match_id = db.execute(
'SELECT match_id FROM raw_iframe_network ORDER BY match_id LIMIT 1'
).fetchone()[0]
_validate_pipeline_output(match_id)
def test_job_store_tracks_progress_logs_and_completion(self):
from database.job_store import JobStore
with tempfile.TemporaryDirectory() as temp_dir:
web_path = Path(temp_dir) / 'Web.sqlite'
with sqlite3.connect(str(web_path)) as db:
with open(Config.DB_WEB_SCHEMA_PATH, 'r', encoding='utf-8') as schema:
db.executescript(schema.read())
store = JobStore(web_path)
job_id = store.create_job(
'test_pipeline',
match_id='g161-99999999999999999999999',
created_by='test',
)
store.start_job(job_id, 'backup', 'Starting')
store.update_progress(job_id, 'l2', 55, 'Building L2')
store.append_log(job_id, 'line one\n')
store.finish_job(job_id, True, 'Done', 1.25)
job = store.get_job(job_id)
self.assertEqual(job['status'], 'succeeded')
self.assertEqual(job['progress'], 100)
self.assertEqual(job['current_stage'], 'complete')
self.assertIn('line one', job['log_text'])
self.assertEqual(job['duration_seconds'], 1.25)
def test_database_backup_and_restore_round_trip(self):
from database.maintenance import backup_database, restore_database
with tempfile.TemporaryDirectory() as temp_dir:
source = Path(temp_dir) / 'source.db'
backup = Path(temp_dir) / 'backup.db'
with sqlite3.connect(str(source)) as db:
db.execute('CREATE TABLE values_table (value INTEGER)')
db.execute('INSERT INTO values_table VALUES (1)')
backup_database(source, backup)
with sqlite3.connect(str(source)) as db:
db.execute('UPDATE values_table SET value = 2')
restore_database(backup, source)
with sqlite3.connect(str(source)) as db:
value = db.execute(
'SELECT value FROM values_table'
).fetchone()[0]
self.assertEqual(value, 1)
def test_high_frequency_queries_use_operational_indexes(self):
l2 = sqlite3.connect(Config.DB_L2_PATH)
try:
plans = {
'player': l2.execute(
"""
EXPLAIN QUERY PLAN
SELECT * FROM fact_match_players
WHERE steam_id_64 = ?
""",
['76561198330488905'],
).fetchone()[3],
'party': l2.execute(
"""
EXPLAIN QUERY PLAN
SELECT * FROM fact_match_players
WHERE match_id = ? AND match_team_id = ?
""",
['match', 1],
).fetchone()[3],
'victim': l2.execute(
"""
EXPLAIN QUERY PLAN
SELECT * FROM fact_round_events
WHERE victim_steam_id = ?
""",
['player'],
).fetchone()[3],
}
finally:
l2.close()
self.assertIn('idx_match_players_player_match', plans['player'])
self.assertIn('idx_match_players_party', plans['party'])
self.assertIn('idx_round_events_victim', plans['victim'])
def test_player_records_reference_real_matches(self):
l3 = sqlite3.connect(Config.DB_L3_PATH)
l2 = sqlite3.connect(Config.DB_L2_PATH)
try:
records = l3.execute(
"""
SELECT match_id
FROM dm_player_records
WHERE match_id IS NOT NULL
"""
).fetchall()
self.assertGreater(len(records), 0)
for (match_id,) in records:
exists = l2.execute(
'SELECT 1 FROM fact_matches WHERE match_id = ?',
[match_id],
).fetchone()
self.assertIsNotNone(exists)
finally:
l3.close()
l2.close()
if __name__ == '__main__':
unittest.main()
web/app.py
+7 -10
View File
@@ -1,20 +1,20 @@
import sys
import os
# Add the project root directory to sys.path
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from flask import Flask, render_template
from flask import Flask
from web.config import Config
from web.database import close_dbs
from web.database import close_dbs, initialize_web_db
def create_app():
def create_app(config_object=Config):
app = Flask(__name__)
app.config.from_object(Config)
app.config.from_object(config_object)
initialize_web_db()
app.teardown_appcontext(close_dbs)
# Register Blueprints
from web.routes import main, matches, players, teams, tactics, admin, wiki, opponents
app.register_blueprint(main.bp)
app.register_blueprint(matches.bp)
@@ -25,12 +25,9 @@ def create_app():
app.register_blueprint(wiki.bp)
app.register_blueprint(opponents.bp)
@app.route('/')
def index():
return render_template('home/index.html')
return app
if __name__ == '__main__':
app = create_app()
app.run(debug=True, port=5000)
web/config.py
+16 -5
View File
@@ -1,14 +1,25 @@
import os
from database.paths import L2_DB, L3_DB, WEB_DB, WEB_SCHEMA
class Config:
SECRET_KEY = os.environ.get('SECRET_KEY') or 'yrtv-secret-key-dev'
BASE_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
DB_L2_PATH = os.path.join(BASE_DIR, 'database', 'L2', 'L2.db')
DB_L3_PATH = os.path.join(BASE_DIR, 'database', 'L3', 'L3.db')
DB_WEB_PATH = os.path.join(BASE_DIR, 'database', 'Web', 'Web_App.sqlite')
SECRET_KEY = os.environ.get('SECRET_KEY', 'yrtv-dev-only-change-me')
ADMIN_TOKEN = os.environ.get('ADMIN_TOKEN', 'yrtv-admin-dev')
ADMIN_TOKEN = 'jackyyang0929'
DB_L2_PATH = str(L2_DB)
DB_L3_PATH = str(L3_DB)
DB_WEB_PATH = str(WEB_DB)
DB_WEB_SCHEMA_PATH = str(WEB_SCHEMA)
WEB_SCHEMA_VERSION = 2
MAX_CONTENT_LENGTH = 5 * 1024 * 1024
SQLITE_TIMEOUT_SECONDS = 15
SLOW_QUERY_THRESHOLD_SECONDS = float(
os.environ.get('SLOW_QUERY_THRESHOLD_SECONDS', '0.25')
)
# Pagination
ITEMS_PER_PAGE = 20
web/database.py
+124 -13
View File
@@ -1,7 +1,96 @@
import os
import logging
import sqlite3
import time
from flask import g
from web.config import Config
logger = logging.getLogger(__name__)
def _database_path(db_name):
paths = {
'l2': Config.DB_L2_PATH,
'l3': Config.DB_L3_PATH,
'web': Config.DB_WEB_PATH,
}
try:
return paths[db_name]
except KeyError as exc:
raise ValueError(f"Unknown database: {db_name}") from exc
def initialize_web_db():
"""Create and migrate the small application-owned database."""
os.makedirs(os.path.dirname(Config.DB_WEB_PATH), exist_ok=True)
db = sqlite3.connect(
Config.DB_WEB_PATH,
timeout=Config.SQLITE_TIMEOUT_SECONDS,
)
try:
table_exists = db.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='team_lineups'"
).fetchone()
if table_exists:
columns = {
row[1] for row in db.execute("PRAGMA table_info(team_lineups)")
}
if 'is_active' not in columns:
db.execute(
"ALTER TABLE team_lineups "
"ADD COLUMN is_active INTEGER NOT NULL DEFAULT 0"
)
first_id = db.execute(
"SELECT id FROM team_lineups "
"ORDER BY created_at DESC, id DESC LIMIT 1"
).fetchone()
if first_id:
db.execute(
"UPDATE team_lineups SET is_active = 1 WHERE id = ?",
first_id,
)
else:
active_ids = [
row[0] for row in db.execute(
"SELECT id FROM team_lineups WHERE is_active = 1 "
"ORDER BY created_at DESC, id DESC"
)
]
if not active_ids:
latest_id = db.execute(
"SELECT id FROM team_lineups "
"ORDER BY created_at DESC, id DESC LIMIT 1"
).fetchone()
if latest_id:
db.execute(
"UPDATE team_lineups SET is_active = 1 WHERE id = ?",
latest_id,
)
elif len(active_ids) > 1:
db.execute("UPDATE team_lineups SET is_active = 0")
db.execute(
"UPDATE team_lineups SET is_active = 1 WHERE id = ?",
[active_ids[0]],
)
with open(Config.DB_WEB_SCHEMA_PATH, 'r', encoding='utf-8') as schema_file:
db.executescript(schema_file.read())
db.execute(
"""
INSERT OR IGNORE INTO schema_migrations (version, description)
VALUES (?, ?)
""",
[
Config.WEB_SCHEMA_VERSION,
'ETL jobs, match imports and active lineup governance',
],
)
db.commit()
finally:
db.close()
def get_db(db_name):
"""
db_name: 'l2', 'l3', or 'web'
@@ -10,18 +99,20 @@ def get_db(db_name):
db = getattr(g, db_attr, None)
if db is None:
if db_name == 'l2':
path = Config.DB_L2_PATH
elif db_name == 'l3':
path = Config.DB_L3_PATH
elif db_name == 'web':
path = Config.DB_WEB_PATH
else:
raise ValueError(f"Unknown database: {db_name}")
# Connect with check_same_thread=False if needed for dev, but default is safer per thread
db = sqlite3.connect(path)
path = _database_path(db_name)
if db_name != 'web' and not os.path.exists(path):
raise RuntimeError(
f"{db_name.upper()} database does not exist: {path}. "
"Run the corresponding data builder first."
)
db = sqlite3.connect(
path,
timeout=Config.SQLITE_TIMEOUT_SECONDS,
)
db.row_factory = sqlite3.Row
db.execute("PRAGMA busy_timeout = 15000")
if db_name != 'l3':
db.execute("PRAGMA foreign_keys = ON")
setattr(g, db_attr, db)
return db
@@ -34,14 +125,34 @@ def close_dbs(e=None):
db.close()
def query_db(db_name, query, args=(), one=False):
started = time.perf_counter()
cur = get_db(db_name).execute(query, args)
rv = cur.fetchall()
cur.close()
try:
rv = cur.fetchall()
finally:
cur.close()
duration = time.perf_counter() - started
if duration >= Config.SLOW_QUERY_THRESHOLD_SECONDS:
logger.warning(
"Slow query db=%s duration=%.3fs sql=%s",
db_name,
duration,
" ".join(query.split())[:500],
)
return (rv[0] if rv else None) if one else rv
def execute_db(db_name, query, args=()):
db = get_db(db_name)
started = time.perf_counter()
cur = db.execute(query, args)
db.commit()
duration = time.perf_counter() - started
if duration >= Config.SLOW_QUERY_THRESHOLD_SECONDS:
logger.warning(
"Slow write db=%s duration=%.3fs sql=%s",
db_name,
duration,
" ".join(query.split())[:500],
)
cur.close()
return cur.lastrowid
web/routes/admin.py
+109 -20
View File
@@ -1,16 +1,17 @@
from flask import Blueprint, render_template, request, redirect, url_for, session, flash
from flask import Blueprint, render_template, request, redirect, url_for, session, flash, jsonify
from web.config import Config
from web.auth import admin_required
from web.database import query_db
import os
from web.services.etl_service import EtlService
import hmac
bp = Blueprint('admin', __name__, url_prefix='/admin')
@bp.route('/login', methods=['GET', 'POST'])
def login():
if request.method == 'POST':
token = request.form.get('token')
if token == Config.ADMIN_TOKEN:
token = request.form.get('token') or ''
if hmac.compare_digest(token, Config.ADMIN_TOKEN):
session['is_admin'] = True
return redirect(url_for('admin.dashboard'))
else:
@@ -27,19 +28,104 @@ def logout():
def dashboard():
return render_template('admin/dashboard.html')
from web.services.etl_service import EtlService
@bp.route('/data-integrity')
@admin_required
def data_integrity():
from web.services.integrity_service import IntegrityService
report = IntegrityService.build_report()
if request.args.get('format') == 'json':
return jsonify(report)
return render_template('admin/data_integrity.html', report=report)
@bp.route('/trigger_etl', methods=['POST'])
@admin_required
def trigger_etl():
script_name = request.form.get('script')
allowed = ['L1A.py', 'L2_Builder.py', 'L3_Builder.py']
if script_name not in allowed:
return "Invalid script", 400
from database.job_store import JobStore
success, message = EtlService.run_script(script_name)
status_code = 200 if success else 500
return message, status_code
store = JobStore(Config.DB_WEB_PATH)
job_id = store.create_job(
'manual_pipeline',
created_by='admin',
)
try:
pid = EtlService.start_pipeline(job_id)
except Exception as exc:
store.finish_job(job_id, False, str(exc), 0)
return jsonify({'success': False, 'error': str(exc)}), 500
return jsonify({'success': True, 'job_id': job_id, 'pid': pid}), 202
@bp.route('/import-match', methods=['GET', 'POST'])
@admin_required
def import_match():
from database.job_store import JobStore
from web.services.import_service import (
DuplicateMatchError,
ImportValidationError,
MatchImportService,
)
store = JobStore(Config.DB_WEB_PATH)
if request.method == 'POST':
upload = request.files.get('capture')
if not upload or not upload.filename:
flash('请选择 iframe_network.json 文件。', 'error')
return redirect(url_for('admin.import_match'))
prepared = None
try:
prepared = MatchImportService.prepare_import(
upload.read(),
upload.filename,
created_by='admin',
replace=request.form.get('replace') == '1',
)
EtlService.start_pipeline(
prepared['job_id'],
match_id=prepared['match_id'],
replace=prepared['replace'],
)
flash(
f"比赛 {prepared['match_id']} 已进入导入队列。",
'success',
)
return redirect(url_for(
'admin.import_match',
job_id=prepared['job_id'],
))
except (DuplicateMatchError, ImportValidationError) as exc:
flash(str(exc), 'warning')
except Exception as exc:
if prepared:
store.finish_job(
prepared['job_id'],
False,
f'Failed to start pipeline: {exc}',
0,
)
flash(f'启动导入失败：{exc}', 'error')
selected_job = None
selected_job_id = request.args.get('job_id', type=int)
if selected_job_id:
selected_job = store.get_job(selected_job_id)
return render_template(
'admin/import_match.html',
jobs=store.list_jobs(30),
selected_job=selected_job,
)
@bp.route('/api/jobs/<int:job_id>')
@admin_required
def api_job(job_id):
from database.job_store import JobStore
job = JobStore(Config.DB_WEB_PATH).get_job(job_id)
if not job:
return jsonify({'error': 'Job not found'}), 404
return jsonify(job)
@bp.route('/sql', methods=['GET', 'POST'])
@admin_required
@@ -50,18 +136,21 @@ def sql_runner():
db_name = "l2"
if request.method == 'POST':
query = request.form.get('query')
query = (request.form.get('query') or '').strip()
db_name = request.form.get('db_name', 'l2')
# Safety check
forbidden = ['DELETE', 'DROP', 'UPDATE', 'INSERT', 'ALTER', 'GRANT', 'REVOKE']
if any(x in query.upper() for x in forbidden):
error = "Only SELECT queries allowed in Web Runner."
statement = query.rstrip(';').strip()
if db_name not in {'l2', 'l3', 'web'}:
error = "Unknown database."
elif not statement.upper().startswith('SELECT '):
error = "Only SELECT queries are allowed."
elif ';' in statement:
error = "Only one SQL statement is allowed."
else:
try:
# Enforce limit if not present
if 'LIMIT' not in query.upper():
query += " LIMIT 50"
query = statement
if 'LIMIT' not in statement.upper():
query = f"{statement} LIMIT 50"
rows = query_db(db_name, query)
if rows:
web/routes/main.py
+9 -11
View File
@@ -1,6 +1,5 @@
from flask import Blueprint, render_template, request, jsonify
from web.services.stats_service import StatsService
import time
bp = Blueprint('main', __name__)
@@ -18,18 +17,17 @@ def index():
return render_template('home/index.html', recent_matches=recent_matches, heatmap_data=heatmap_data, live_matches=live_matches)
from web.services.etl_service import EtlService
@bp.route('/parse_match', methods=['POST'])
def parse_match():
url = request.form.get('url')
if not url or '5eplay.com' not in url:
return jsonify({'success': False, 'message': 'Invalid 5EPlay URL'})
return jsonify({'success': False, 'message': 'Invalid 5EPlay URL'}), 400
# Trigger L1A.py with URL argument
success, msg = EtlService.run_script('L1A.py', args=[url])
if success:
return jsonify({'success': True, 'message': 'Match parsing completed successfully!'})
else:
return jsonify({'success': False, 'message': f'Error: {msg}'})
return jsonify({
'success': False,
'message': (
'URL downloader is not included in this repository. '
'Place iframe_network.json under output_arena/<match_id>/ '
'and run the L1/L2/L3 builders from Admin.'
),
}), 501
web/routes/matches.py
+2 -12
View File
@@ -33,19 +33,9 @@ def detail(match_id):
rounds = StatsService.get_match_rounds(match_id)
# --- Roster Identification ---
# Fetch active roster to identify "Our Team" players
from web.services.web_service import WebService
lineups = WebService.get_lineups()
# Assume we use the first/active lineup
active_roster_ids = []
if lineups:
try:
active_roster_ids = json.loads(lineups[0]['player_ids_json'])
except:
pass
from web.services.team_context_service import TeamContextService
# Mark roster players (Ensure strict string comparison)
active_roster_ids = TeamContextService.get_active_roster_ids()
roster_set = set(str(uid) for uid in active_roster_ids)
for p in players:
p['is_in_roster'] = str(p['steam_id_64']) in roster_set
web/routes/players.py
+35 -43
View File
@@ -1,15 +1,16 @@
from flask import Blueprint, render_template, request, jsonify, redirect, url_for, flash, current_app, session
from web.services.stats_service import StatsService
from web.services.feature_service import FeatureService
from web.services.player_profile_service import PlayerProfileService
from web.services.web_service import WebService
from web.database import execute_db, query_db
from web.config import Config
from datetime import datetime
import os
import json
from werkzeug.utils import secure_filename
bp = Blueprint('players', __name__, url_prefix='/players')
ALLOWED_AVATAR_EXTENSIONS = {'.jpg', '.jpeg', '.png', '.webp'}
@bp.route('/')
def index():
@@ -41,7 +42,12 @@ def detail(steam_id):
# Use steam_id as filename to ensure uniqueness per player
# Preserve extension
ext = os.path.splitext(file.filename)[1].lower()
if not ext: ext = '.jpg'
if (
ext not in ALLOWED_AVATAR_EXTENSIONS
or not (file.mimetype or '').startswith('image/')
):
flash('Avatar must be a JPG, PNG, or WebP image.', 'error')
return redirect(url_for('players.detail', steam_id=steam_id))
filename = f"{steam_id}{ext}"
upload_folder = os.path.join(current_app.root_path, 'static', 'avatars')
@@ -177,33 +183,9 @@ def detail(steam_id):
history_asc = StatsService.get_player_trend(steam_id, limit=1000)
history = history_asc[::-1] if history_asc else []
# Calculate Map Stats
map_stats = {}
for match in history:
m_name = match['map_name']
if m_name not in map_stats:
map_stats[m_name] = {'matches': 0, 'wins': 0, 'adr_sum': 0, 'rating_sum': 0}
map_stats[m_name]['matches'] += 1
if match['is_win']:
map_stats[m_name]['wins'] += 1
map_stats[m_name]['adr_sum'] += (match['adr'] or 0)
map_stats[m_name]['rating_sum'] += (match['rating'] or 0)
map_stats_list = []
for m_name, data in map_stats.items():
cnt = data['matches']
map_stats_list.append({
'map_name': m_name,
'matches': cnt,
'win_rate': data['wins'] / cnt,
'adr': data['adr_sum'] / cnt,
'rating': data['rating_sum'] / cnt
})
map_stats_list.sort(key=lambda x: x['matches'], reverse=True)
# --- New: Recent Performance Stats ---
# recent_stats = StatsService.get_recent_performance_stats(steam_id)
map_stats_list = PlayerProfileService.get_map_stats(steam_id)
period_stats = PlayerProfileService.get_period_stats(steam_id)
records = PlayerProfileService.get_records(steam_id)
return render_template('players/profile.html',
player=player,
@@ -213,6 +195,8 @@ def detail(steam_id):
history=history,
distribution=distribution,
map_stats=map_stats_list,
period_stats=period_stats,
records=records,
l2_stats=l2_stats,
side_stats=side_stats)
@@ -223,9 +207,13 @@ def like_comment(comment_id):
@bp.route('/<steam_id>/charts_data')
def charts_data(steam_id):
# ... (existing code) ...
# Trend Data
trends = StatsService.get_player_trend(steam_id, limit=1000)
period_key = request.args.get('period', 'last_20')
if period_key not in PlayerProfileService.PERIOD_KEYS:
period_key = 'last_20'
trends = PlayerProfileService.get_period_history(steam_id, period_key)
if not trends:
trends = StatsService.get_player_trend(steam_id, limit=20)
period_summary = PlayerProfileService.get_period(steam_id, period_key)
# Radar Data (Construct from features)
features = FeatureService.get_player_features(steam_id)
@@ -234,17 +222,11 @@ def charts_data(steam_id):
# Task 1: Strict Team Average Calculation
team_avg_radar = None
lineups = WebService.get_lineups()
if lineups:
target_lineup = None
try:
p_ids = [str(i) for i in json.loads(lineups[0].get("player_ids_json") or "[]")]
if str(steam_id) in p_ids:
target_lineup = p_ids
except:
target_lineup = None
from web.services.team_context_service import TeamContextService
if target_lineup:
active_roster_ids = TeamContextService.get_active_roster_ids()
target_lineup = active_roster_ids if str(steam_id) in active_roster_ids else None
if target_lineup:
# Calculate strict average for this lineup
team_sums = {
'score_aim': 0.0, 'score_defense': 0.0, 'score_utility': 0.0,
@@ -303,9 +285,19 @@ def charts_data(steam_id):
'trend': {'labels': trend_labels, 'values': trend_values},
'radar': radar_data,
'radar_dist': radar_dist,
'team_avg_radar': team_avg_radar
'team_avg_radar': team_avg_radar,
'period': period_summary,
})
@bp.route('/<steam_id>/period_stats')
def period_stats(steam_id):
period_key = request.args.get('period', 'last_20')
period = PlayerProfileService.get_period(steam_id, period_key)
if not period:
return jsonify({'error': 'Unknown period or player'}), 404
return jsonify(period)
# --- API for Comparison ---
@bp.route('/api/search')
def api_search():
web/routes/teams.py
+4 -6
View File
@@ -67,14 +67,12 @@ def api_search():
@bp.route('/api/roster', methods=['GET', 'POST'])
def api_roster():
# Assume single team mode, always operating on ID=1 or the first lineup
lineups = WebService.get_lineups()
if not lineups:
# Auto-create default team if none exists
target_team = WebService.get_active_lineup()
if not target_team:
WebService.save_lineup("My Team", "Default Roster", [])
lineups = WebService.get_lineups()
target_team = WebService.get_active_lineup()
target_team = dict(lineups[0]) # Get the latest one
target_team = dict(target_team)
if request.method == 'POST':
# Admin Check
web/services/etl_service.py
+41 -3
View File
@@ -4,13 +4,51 @@ import sys
from web.config import Config
class EtlService:
SCRIPT_PATHS = {
'L1A.py': os.path.join('database', 'L1', 'L1_Builder.py'),
'L2_Builder.py': os.path.join('database', 'L2', 'L2_Builder.py'),
'L3_Builder.py': os.path.join('database', 'L3', 'L3_Builder.py'),
}
@staticmethod
def start_pipeline(job_id, match_id=None, replace=False):
script_path = os.path.join(
Config.BASE_DIR,
'database',
'pipeline.py',
)
command = [
sys.executable,
script_path,
'--job-id',
str(int(job_id)),
]
if match_id:
command.extend(['--match-id', str(match_id)])
if replace:
command.append('--replace')
process = subprocess.Popen(
command,
cwd=Config.BASE_DIR,
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
start_new_session=True,
)
return process.pid
@staticmethod
def run_script(script_name, args=None):
"""
Executes an ETL script located in the ETL directory.
Executes an allow-listed data builder from its actual repository path.
Returns (success, message)
"""
script_path = os.path.join(Config.BASE_DIR, 'ETL', script_name)
relative_path = EtlService.SCRIPT_PATHS.get(script_name)
if not relative_path:
return False, f"Unsupported data script: {script_name}"
script_path = os.path.join(Config.BASE_DIR, relative_path)
if not os.path.exists(script_path):
return False, f"Script not found: {script_path}"
@@ -28,7 +66,7 @@ class EtlService:
cwd=Config.BASE_DIR,
capture_output=True,
text=True,
timeout=300 # 5 min timeout
timeout=900
)
if result.returncode == 0:
web/services/feature_service.py
+69 -34
View File
@@ -1,6 +1,6 @@
from __future__ import annotations
from typing import Any, Iterable
from typing import Any
from web.database import query_db
@@ -138,20 +138,47 @@ class FeatureService:
}
order_col = sort_map.get(sort_by, "core_avg_rating")
where = []
args: list[Any] = []
if search:
where.append("steam_id_64 IN (SELECT steam_id_64 FROM dim_players WHERE username LIKE ?)")
args.append(f"%{search}%")
where_sql = f"WHERE {' AND '.join(where)}" if where else ""
rows = query_db(
"l3",
f"SELECT * FROM dm_player_features {where_sql} ORDER BY {order_col} DESC LIMIT ? OFFSET ?",
args + [per_page, offset],
)
total_row = query_db("l3", f"SELECT COUNT(*) as cnt FROM dm_player_features {where_sql}", args, one=True)
total = int(total_row["cnt"]) if total_row else 0
dim_rows = query_db(
"l2",
"""
SELECT steam_id_64
FROM dim_players
WHERE LOWER(username) LIKE LOWER(?) OR steam_id_64 LIKE ?
""",
[f"%{search}%", f"%{search}%"],
)
matching_ids = [str(row["steam_id_64"]) for row in dim_rows]
rows = []
for start in range(0, len(matching_ids), 500):
chunk = matching_ids[start:start + 500]
placeholders = ",".join("?" for _ in chunk)
rows.extend(query_db(
"l3",
f"SELECT * FROM dm_player_features "
f"WHERE steam_id_64 IN ({placeholders})",
chunk,
))
rows = sorted(
rows,
key=lambda row: row[order_col] if row[order_col] is not None else float("-inf"),
reverse=True,
)
total = len(rows)
rows = rows[offset:offset + per_page]
else:
rows = query_db(
"l3",
f"SELECT * FROM dm_player_features "
f"ORDER BY {order_col} DESC LIMIT ? OFFSET ?",
[per_page, offset],
)
total_row = query_db(
"l3",
"SELECT COUNT(*) as cnt FROM dm_player_features",
one=True,
)
total = int(total_row["cnt"]) if total_row else 0
players = [FeatureService._normalize_features(dict(r)) for r in rows] if rows else []
players = [p for p in players if p]
@@ -160,19 +187,11 @@ class FeatureService:
@staticmethod
def get_roster_features_distribution(target_steam_id: str):
from web.services.web_service import WebService
import json
from web.services.team_context_service import TeamContextService
lineups = WebService.get_lineups()
roster_ids: list[str] = []
if lineups:
try:
p_ids = [str(i) for i in json.loads(lineups[0].get("player_ids_json") or "[]")]
if str(target_steam_id) in p_ids:
roster_ids = p_ids
except Exception:
roster_ids = []
roster_ids = TeamContextService.get_active_roster_ids()
if str(target_steam_id) not in roster_ids:
roster_ids = []
if not roster_ids:
return None
@@ -202,7 +221,17 @@ class FeatureService:
sample_keys = list(p.keys())
break
lower_is_better = {"int_timing_first_contact_time", "tac_avg_fd", "core_avg_match_duration"}
lower_is_better = {
"int_timing_first_contact_time",
"int_trade_response_time",
"tac_avg_fd",
"tac_fd_rate",
"core_avg_match_duration",
"core_dpr",
"meta_rating_volatility",
"meta_map_stability",
"meta_elo_tier_stability",
}
result: dict[str, Any] = {}
for m in sample_keys:
@@ -224,16 +253,22 @@ class FeatureService:
values = []
for p in stats_map.values():
v = (p or {}).get(m)
if v is None:
continue
try:
values.append(float(v) if v is not None else 0.0)
values.append(float(v))
except (ValueError, TypeError):
values.append(0.0)
continue
target_val_raw = (stats_map.get(target_steam_id) or {}).get(m)
if target_val_raw is None or not values:
result[m] = None
continue
try:
target_val = float(target_val_raw) if target_val_raw is not None else 0.0
target_val = float(target_val_raw)
except (ValueError, TypeError):
target_val = 0.0
result[m] = None
continue
is_reverse = m not in lower_is_better
# Sort values. For standard metrics, higher is better (reverse=True).
@@ -251,9 +286,9 @@ class FeatureService:
"val": target_val,
"rank": rank,
"total": len(values_sorted),
"min": min(values_sorted) if values_sorted else 0,
"max": max(values_sorted) if values_sorted else 0,
"avg": (sum(values_sorted) / len(values_sorted)) if values_sorted else 0,
"min": min(values_sorted),
"max": max(values_sorted),
"avg": sum(values_sorted) / len(values_sorted),
"inverted": not is_reverse,
}
return result
web/services/import_service.py
+186
View File
@@ -0,0 +1,186 @@
import hashlib
import json
import os
from pathlib import Path
import re
import sqlite3
from typing import Any, Dict
from database.job_store import JobStore
from database.paths import L1_DB, OUTPUT_ARENA
from web.config import Config
MATCH_ID_PATTERN = re.compile(r'\bg161-[0-9]{10,}\b')
class ImportValidationError(ValueError):
pass
class DuplicateMatchError(ImportValidationError):
pass
class MatchImportService:
@staticmethod
def validate_capture(raw_bytes: bytes) -> Dict[str, Any]:
if not raw_bytes:
raise ImportValidationError('Uploaded file is empty')
try:
text = raw_bytes.decode('utf-8-sig')
except UnicodeDecodeError as exc:
raise ImportValidationError('Capture must be UTF-8 JSON') from exc
try:
capture = json.loads(text)
except json.JSONDecodeError as exc:
raise ImportValidationError(
f'Invalid JSON at line {exc.lineno}, column {exc.colno}'
) from exc
if not isinstance(capture, list) or not capture:
raise ImportValidationError(
'Capture root must be a non-empty list of network responses'
)
urls = []
successful_responses = 0
for index, item in enumerate(capture):
if not isinstance(item, dict):
raise ImportValidationError(
f'Capture item {index} must be an object'
)
url = item.get('url')
if not isinstance(url, str) or not url:
raise ImportValidationError(
f'Capture item {index} has no URL'
)
urls.append(url)
if item.get('status') == 200 and item.get('body') is not None:
successful_responses += 1
match_ids = sorted({
match.group(0)
for url in urls
for match in MATCH_ID_PATTERN.finditer(url)
})
if len(match_ids) != 1:
raise ImportValidationError(
f'Capture must reference exactly one match ID; found {match_ids}'
)
if successful_responses < 2:
raise ImportValidationError(
'Capture does not contain enough successful API responses'
)
match_id = match_ids[0]
has_match_data = any(
f'/api/data/match/{match_id}' in url for url in urls
)
has_round_data = any(
f'/api/match/round/{match_id}' in url for url in urls
)
if not has_match_data or not has_round_data:
missing = []
if not has_match_data:
missing.append('match data')
if not has_round_data:
missing.append('round data')
raise ImportValidationError(
f"Capture is missing required endpoint(s): {', '.join(missing)}"
)
return {
'match_id': match_id,
'content_sha256': hashlib.sha256(raw_bytes).hexdigest(),
'response_count': len(capture),
'successful_responses': successful_responses,
'text': text,
}
@staticmethod
def _existing_l1_content(match_id: str):
if not L1_DB.exists():
return None
db = sqlite3.connect(str(L1_DB))
try:
row = db.execute(
"""
SELECT content
FROM raw_iframe_network
WHERE match_id = ?
""",
[match_id],
).fetchone()
return row[0] if row else None
finally:
db.close()
@staticmethod
def prepare_import(
raw_bytes: bytes,
original_filename: str,
created_by: str,
replace: bool = False,
):
validation = MatchImportService.validate_capture(raw_bytes)
match_id = validation['match_id']
content_hash = validation['content_sha256']
existing_content = MatchImportService._existing_l1_content(match_id)
if existing_content is not None:
existing_hash = hashlib.sha256(
existing_content.encode('utf-8')
).hexdigest()
if existing_hash == content_hash:
raise DuplicateMatchError(
f'Match {match_id} is already imported with identical data'
)
if not replace:
raise DuplicateMatchError(
f'Match {match_id} already exists with different data; '
'explicit replacement is required'
)
match_dir = OUTPUT_ARENA / match_id
match_dir.mkdir(parents=True, exist_ok=True)
destination = match_dir / 'iframe_network.json'
if destination.exists() and not replace:
current_hash = hashlib.sha256(destination.read_bytes()).hexdigest()
if current_hash == content_hash:
raise DuplicateMatchError(
f'Match {match_id} is already queued with identical data'
)
raise DuplicateMatchError(
f'Pending capture already exists for {match_id}'
)
temporary = destination.with_suffix('.json.tmp')
temporary.write_bytes(raw_bytes)
os.replace(str(temporary), str(destination))
store = JobStore(Config.DB_WEB_PATH)
job_id = store.create_job(
'match_import',
match_id=match_id,
input_path=str(destination),
created_by=created_by,
)
store.upsert_match_import(
match_id,
content_hash,
str(destination),
'queued',
job_id,
)
return {
'job_id': job_id,
'match_id': match_id,
'content_sha256': content_hash,
'response_count': validation['response_count'],
'source_path': str(destination),
'original_filename': Path(original_filename or '').name,
'replace': bool(replace),
}
web/services/integrity_service.py
+511
View File
@@ -0,0 +1,511 @@
from datetime import datetime, timezone
import json
import os
import sqlite3
from database.maintenance import backup_storage_status
from web.config import Config
from web.services.team_context_service import TeamContextService
class IntegrityService:
DATABASES = {
'L2': Config.DB_L2_PATH,
'L3': Config.DB_L3_PATH,
'Web': Config.DB_WEB_PATH,
}
@staticmethod
def _check(checks, name, status, detail, value=None):
checks.append({
'name': name,
'status': status,
'detail': detail,
'value': value,
})
@staticmethod
def _connect(path):
db = sqlite3.connect(path, timeout=Config.SQLITE_TIMEOUT_SECONDS)
db.row_factory = sqlite3.Row
return db
@staticmethod
def build_report():
checks = []
counts = {}
connections = {}
try:
for name, path in IntegrityService.DATABASES.items():
if not os.path.exists(path):
IntegrityService._check(
checks,
f'{name} database',
'fail',
f'Missing file: {path}',
)
continue
try:
db = IntegrityService._connect(path)
connections[name] = db
result = db.execute('PRAGMA quick_check').fetchone()[0]
IntegrityService._check(
checks,
f'{name} database',
'pass' if result == 'ok' else 'fail',
f'quick_check: {result}',
os.path.getsize(path),
)
except sqlite3.Error as exc:
IntegrityService._check(
checks,
f'{name} database',
'fail',
str(exc),
)
l2 = connections.get('L2')
if l2:
IntegrityService._check_l2(l2, checks, counts)
l3 = connections.get('L3')
roster_ids = TeamContextService.get_active_roster_ids()
counts['active_roster'] = len(roster_ids)
if l3:
IntegrityService._check_l3(l3, roster_ids, checks, counts)
web = connections.get('Web')
if web:
IntegrityService._check_web(web, checks, counts)
backup_status = backup_storage_status()
counts['backup_sets'] = backup_status['sets']
counts['backup_bytes'] = backup_status['total_bytes']
IntegrityService._check(
checks,
'Backup retention',
'warn' if backup_status['sets'] > 3 else 'pass',
(
f"{backup_status['sets']} backup sets, "
f"{backup_status['total_bytes']:,} bytes"
),
backup_status['sets'],
)
finally:
for db in connections.values():
db.close()
status_order = {'pass': 0, 'warn': 1, 'fail': 2}
overall_status = max(
(check['status'] for check in checks),
key=lambda status: status_order[status],
default='fail',
)
return {
'generated_at': datetime.now(timezone.utc).isoformat(),
'overall_status': overall_status,
'counts': counts,
'checks': checks,
'totals': {
status: sum(1 for check in checks if check['status'] == status)
for status in ('pass', 'warn', 'fail')
},
}
@staticmethod
def _table_names(db):
return {
row[0]
for row in db.execute(
"SELECT name FROM sqlite_master WHERE type='table'"
)
}
@staticmethod
def _check_l2(db, checks, counts):
required_tables = {
'dim_players',
'fact_matches',
'fact_match_teams',
'fact_match_players',
'fact_rounds',
'fact_round_events',
'fact_round_player_economy',
}
missing = sorted(required_tables - IntegrityService._table_names(db))
IntegrityService._check(
checks,
'L2 required tables',
'fail' if missing else 'pass',
f"Missing: {', '.join(missing)}" if missing else 'All required tables exist',
)
if missing:
return
table_count_map = {
'matches': 'fact_matches',
'players': 'dim_players',
'player_match_rows': 'fact_match_players',
'rounds': 'fact_rounds',
'events': 'fact_round_events',
'economy_rows': 'fact_round_player_economy',
}
for key, table in table_count_map.items():
counts[key] = db.execute(f'SELECT COUNT(*) FROM {table}').fetchone()[0]
orphan_players = db.execute(
"""
SELECT COUNT(*)
FROM fact_match_players mp
LEFT JOIN fact_matches m ON m.match_id = mp.match_id
WHERE m.match_id IS NULL
"""
).fetchone()[0]
IntegrityService._check(
checks,
'Player-match referential integrity',
'fail' if orphan_players else 'pass',
f'{orphan_players} player rows reference missing matches',
orphan_players,
)
orphan_events = db.execute(
"""
SELECT COUNT(*)
FROM fact_round_events e
LEFT JOIN fact_rounds r
ON r.match_id = e.match_id AND r.round_num = e.round_num
WHERE r.match_id IS NULL
"""
).fetchone()[0]
IntegrityService._check(
checks,
'Round-event referential integrity',
'fail' if orphan_events else 'pass',
f'{orphan_events} events reference missing rounds',
orphan_events,
)
unusual_rosters = db.execute(
"""
SELECT COUNT(*)
FROM (
SELECT match_id, COUNT(*) AS player_count
FROM fact_match_players
GROUP BY match_id
HAVING player_count != 10
)
"""
).fetchone()[0]
IntegrityService._check(
checks,
'Match player cardinality',
'warn' if unusual_rosters else 'pass',
f'{unusual_rosters} matches do not contain exactly 10 players',
unusual_rosters,
)
missing_names = db.execute(
"SELECT COUNT(*) FROM dim_players WHERE username IS NULL OR TRIM(username) = ''"
).fetchone()[0]
IntegrityService._check(
checks,
'Player identity coverage',
'warn' if missing_names else 'pass',
f'{missing_names} players have no username',
missing_names,
)
required_indexes = {
'idx_match_players_player_match',
'idx_match_players_match_team',
'idx_match_players_party',
'idx_round_events_victim',
'idx_economy_player_match',
'idx_matches_map_time',
}
existing_indexes = {
row[0] for row in db.execute(
"SELECT name FROM sqlite_master WHERE type = 'index'"
)
}
missing_indexes = sorted(required_indexes - existing_indexes)
IntegrityService._check(
checks,
'L2 operational indexes',
'fail' if missing_indexes else 'pass',
(
f"Missing: {', '.join(missing_indexes)}"
if missing_indexes else
'All high-frequency query indexes exist'
),
len(required_indexes) - len(missing_indexes),
)
@staticmethod
def _check_l3(db, roster_ids, checks, counts):
required_tables = {
'dm_player_features',
'dm_player_match_history',
'dm_player_map_stats',
'dm_player_period_stats',
'dm_player_records',
'dm_player_weapon_stats',
}
missing = sorted(required_tables - IntegrityService._table_names(db))
IntegrityService._check(
checks,
'L3 required tables',
'fail' if missing else 'pass',
f"Missing: {', '.join(missing)}" if missing else 'All required tables exist',
)
if missing:
return
counts['l3_features'] = db.execute(
'SELECT COUNT(*) FROM dm_player_features'
).fetchone()[0]
counts['l3_history'] = db.execute(
'SELECT COUNT(*) FROM dm_player_match_history'
).fetchone()[0]
counts['l3_maps'] = db.execute(
'SELECT COUNT(*) FROM dm_player_map_stats'
).fetchone()[0]
counts['l3_weapons'] = db.execute(
'SELECT COUNT(*) FROM dm_player_weapon_stats'
).fetchone()[0]
counts['l3_periods'] = db.execute(
'SELECT COUNT(*) FROM dm_player_period_stats'
).fetchone()[0]
counts['l3_records'] = db.execute(
'SELECT COUNT(*) FROM dm_player_records'
).fetchone()[0]
if roster_ids:
placeholders = ','.join('?' for _ in roster_ids)
covered = db.execute(
f"""
SELECT COUNT(DISTINCT steam_id_64)
FROM dm_player_features
WHERE steam_id_64 IN ({placeholders})
""",
roster_ids,
).fetchone()[0]
IntegrityService._check(
checks,
'Active roster feature coverage',
'pass' if covered == len(roster_ids) else 'fail',
f'{covered}/{len(roster_ids)} roster players have L3 features',
covered,
)
expected_history = db.execute(
f"""
SELECT COALESCE(SUM(total_matches), 0)
FROM dm_player_features
WHERE steam_id_64 IN ({placeholders})
""",
roster_ids,
).fetchone()[0]
actual_history = db.execute(
f"""
SELECT COUNT(*)
FROM dm_player_match_history
WHERE steam_id_64 IN ({placeholders})
""",
roster_ids,
).fetchone()[0]
IntegrityService._check(
checks,
'Roster history completeness',
'pass' if actual_history == expected_history else 'fail',
f'{actual_history}/{expected_history} player-match rows materialized',
actual_history,
)
score_rows = db.execute(
f"""
SELECT steam_id_64, score_overall, tier_percentile
FROM dm_player_features
WHERE steam_id_64 IN ({placeholders})
AND score_overall > 0
""",
roster_ids,
).fetchall()
scores = [float(row['score_overall']) for row in score_rows]
invalid_percentiles = 0
for row in score_rows:
expected = (
sum(value <= float(row['score_overall']) for value in scores)
/ len(scores)
* 100
)
actual = row['tier_percentile']
if actual is None or abs(float(actual) - expected) > 0.011:
invalid_percentiles += 1
IntegrityService._check(
checks,
'Roster percentile correctness',
'pass' if invalid_percentiles == 0 else 'warn',
f'{invalid_percentiles} eligible players have stale percentiles',
invalid_percentiles,
)
for table, label in (
('dm_player_period_stats', 'Roster period-stat coverage'),
('dm_player_records', 'Roster record coverage'),
):
covered = db.execute(
f"""
SELECT COUNT(DISTINCT steam_id_64)
FROM {table}
WHERE steam_id_64 IN ({placeholders})
""",
roster_ids,
).fetchone()[0]
IntegrityService._check(
checks,
label,
'pass' if covered == len(roster_ids) else 'fail',
f'{covered}/{len(roster_ids)} roster players covered',
covered,
)
for key, label in (
('l3_history', 'Player match history mart'),
('l3_maps', 'Player map stats mart'),
('l3_weapons', 'Player weapon stats mart'),
('l3_periods', 'Player period stats mart'),
('l3_records', 'Player records mart'),
):
value = counts[key]
IntegrityService._check(
checks,
label,
'pass' if value else 'warn',
f'{value} rows',
value,
)
if roster_ids:
placeholders = ','.join('?' for _ in roster_ids)
scope_sql = f"AND steam_id_64 IN ({placeholders})"
scope_args = roster_ids
else:
scope_sql = ''
scope_args = []
placeholder_rows = db.execute(
f"""
SELECT COUNT(*)
FROM dm_player_features
WHERE int_pos_site_a_control_rate = 0.33
AND int_pos_site_b_control_rate = 0.33
AND int_pos_mid_control_rate = 0.34
{scope_sql}
""",
scope_args,
).fetchone()[0]
IntegrityService._check(
checks,
'Experimental spatial metrics',
'warn' if placeholder_rows else 'pass',
f'{placeholder_rows} active-roster rows contain placeholder site-control values',
placeholder_rows,
)
@staticmethod
def _check_web(db, checks, counts):
required_tables = {
'comments',
'etl_jobs',
'match_imports',
'player_metadata',
'schema_migrations',
'strategy_boards',
'team_lineups',
'wiki_pages',
}
missing = sorted(required_tables - IntegrityService._table_names(db))
IntegrityService._check(
checks,
'Web required tables',
'fail' if missing else 'pass',
f"Missing: {', '.join(missing)}" if missing else 'All required tables exist',
)
if missing:
return
counts['etl_jobs'] = db.execute(
'SELECT COUNT(*) FROM etl_jobs'
).fetchone()[0]
counts['match_imports'] = db.execute(
'SELECT COUNT(*) FROM match_imports'
).fetchone()[0]
schema_version = db.execute(
'SELECT COALESCE(MAX(version), 0) FROM schema_migrations'
).fetchone()[0]
counts['web_schema_version'] = schema_version
IntegrityService._check(
checks,
'Web schema version',
'pass' if schema_version == Config.WEB_SCHEMA_VERSION else 'fail',
f'{schema_version}/{Config.WEB_SCHEMA_VERSION}',
schema_version,
)
foreign_key_errors = db.execute(
'PRAGMA foreign_key_check'
).fetchall()
IntegrityService._check(
checks,
'Web foreign key integrity',
'fail' if foreign_key_errors else 'pass',
f'{len(foreign_key_errors)} foreign key violations',
len(foreign_key_errors),
)
running_jobs = db.execute(
"""
SELECT COUNT(*)
FROM etl_jobs
WHERE status = 'running'
"""
).fetchone()[0]
IntegrityService._check(
checks,
'Pipeline concurrency',
'warn' if running_jobs > 1 else 'pass',
f'{running_jobs} running pipeline jobs',
running_jobs,
)
lineups = db.execute(
'SELECT id, player_ids_json, is_active FROM team_lineups'
).fetchall()
counts['lineups'] = len(lineups)
invalid_lineups = 0
for lineup in lineups:
try:
player_ids = json.loads(lineup['player_ids_json'] or '[]')
if not isinstance(player_ids, list):
invalid_lineups += 1
except (TypeError, json.JSONDecodeError):
invalid_lineups += 1
IntegrityService._check(
checks,
'Lineup JSON validity',
'fail' if invalid_lineups else 'pass',
f'{invalid_lineups} lineups contain invalid player ID JSON',
invalid_lineups,
)
active_count = sum(1 for lineup in lineups if lineup['is_active'] == 1)
IntegrityService._check(
checks,
'Active lineup',
'pass' if active_count == 1 else 'warn',
f'{active_count} active lineups configured',
active_count,
)
web/services/opponent_service.py
+44 -52
View File
@@ -1,19 +1,10 @@
from web.database import query_db
from web.services.web_service import WebService
import json
from web.services.team_context_service import TeamContextService
class OpponentService:
@staticmethod
def _get_active_roster_ids():
lineups = WebService.get_lineups()
active_roster_ids = []
if lineups:
try:
raw_ids = json.loads(lineups[0]['player_ids_json'])
active_roster_ids = [str(uid) for uid in raw_ids]
except:
pass
return active_roster_ids
return TeamContextService.get_active_roster_ids()
@staticmethod
def get_opponent_list(page=1, per_page=20, sort_by='matches', search=None):
@@ -21,30 +12,21 @@ class OpponentService:
if not roster_ids:
return [], 0
# Placeholders
roster_ph = ','.join('?' for _ in roster_ids)
# 1. Identify Matches involving our roster (at least 1 member? usually 2 for 'team' match)
# Let's say at least 1 for broader coverage as requested ("1 match sample")
# But "Our Team" usually implies the entity. Let's stick to matches where we can identify "Us".
# If we use >=1, we catch solo Q matches of roster members. The user said "Non-team members or 1 match sample",
# but implied "facing different our team lineups".
# Let's use the standard "candidate matches" logic (>=2 roster members) to represent "The Team".
# OR, if user wants "Opponent Analysis" for even 1 match, maybe they mean ANY match in DB?
# "Left Top add Opponent Analysis... (non-team member or 1 sample)"
# This implies we analyze PLAYERS who are NOT us.
# Let's stick to matches where >= 1 roster member played, to define "Us" vs "Them".
# Actually, let's look at ALL matches in DB, and any player NOT in active roster is an "Opponent".
# This covers "1 sample".
# Query:
# Select all players who are NOT in active roster.
# Group by steam_id.
# Aggregate stats.
where_clauses = [f"CAST(mp.steam_id_64 AS TEXT) NOT IN ({roster_ph})"]
args = list(roster_ids)
where_clauses = [
f"CAST(mp.steam_id_64 AS TEXT) NOT IN ({roster_ph})",
f"""
EXISTS (
SELECT 1
FROM fact_match_players roster_mp
WHERE roster_mp.match_id = mp.match_id
AND CAST(roster_mp.steam_id_64 AS TEXT) IN ({roster_ph})
AND roster_mp.team_id != mp.team_id
)
""",
]
args = list(roster_ids) + list(roster_ids)
if search:
where_clauses.append("(LOWER(p.username) LIKE LOWER(?) OR mp.steam_id_64 LIKE ?)")
@@ -61,16 +43,6 @@ class OpponentService:
elif sort_by == 'win_rate':
sort_sql = "win_rate DESC"
# Main Aggregation Query
# We need to join fact_matches to get match info (win/loss, elo) if needed,
# but fact_match_players has is_win (boolean) usually? No, it has team_id.
# We need to determine if THEY won.
# fact_match_players doesn't store is_win directly in schema (I should check schema, but stats_service calculates it).
# Wait, stats_service.get_player_trend uses `mp.is_win`?
# Let's check schema. `fact_match_players` usually has `match_id`, `team_id`.
# `fact_matches` has `winner_team`.
# So we join.
offset = (page - 1) * per_page
sql = f"""
@@ -151,10 +123,17 @@ class OpponentService:
LEFT JOIN fact_match_teams fmt_gid ON mp.match_id = fmt_gid.match_id AND fmt_gid.group_id = mp.team_id
LEFT JOIN fact_match_teams fmt_tid ON mp.match_id = fmt_tid.match_id AND fmt_tid.group_tid = mp.match_team_id
WHERE CAST(mp.steam_id_64 AS TEXT) NOT IN ({roster_ph})
AND EXISTS (
SELECT 1
FROM fact_match_players roster_mp
WHERE roster_mp.match_id = mp.match_id
AND CAST(roster_mp.steam_id_64 AS TEXT) IN ({roster_ph})
AND roster_mp.team_id != mp.team_id
)
GROUP BY mp.steam_id_64
"""
rows = query_db('l2', sql, roster_ids)
rows = query_db('l2', sql, roster_ids + roster_ids)
# Initialize Buckets
elo_buckets = {'<1000': 0, '1000-1200': 0, '1200-1400': 0, '1400-1600': 0, '1600-1800': 0, '1800-2000': 0, '>2000': 0}
@@ -216,13 +195,12 @@ class OpponentService:
player = dict(info)
player['avatar_url'] = StatsService.resolve_avatar_url(steam_id, player.get('avatar_url'))
# 2. Match History vs Us (All matches this player played)
# We define "Us" as matches where this player is an opponent.
# But actually, we just show ALL their matches in our DB, assuming our DB only contains matches relevant to us?
# Usually yes, but if we have a huge DB, we might want to filter by "Contains Roster Member".
# For now, show all matches in DB for this player.
roster_ids = OpponentService._get_active_roster_ids()
if not roster_ids:
return None
roster_ph = ','.join('?' for _ in roster_ids)
sql_history = """
sql_history = f"""
SELECT
m.match_id, m.start_time, m.map_name, m.score_team1, m.score_team2, m.winner_team,
mp.team_id, mp.match_team_id, mp.rating, mp.kd_ratio, mp.adr, mp.kills, mp.deaths,
@@ -236,9 +214,16 @@ class OpponentService:
LEFT JOIN fact_match_teams fmt_gid ON mp.match_id = fmt_gid.match_id AND fmt_gid.group_id = mp.team_id
LEFT JOIN fact_match_teams fmt_tid ON mp.match_id = fmt_tid.match_id AND fmt_tid.group_tid = mp.match_team_id
WHERE mp.steam_id_64 = ?
AND EXISTS (
SELECT 1
FROM fact_match_players roster_mp
WHERE roster_mp.match_id = mp.match_id
AND CAST(roster_mp.steam_id_64 AS TEXT) IN ({roster_ph})
AND roster_mp.team_id != mp.team_id
)
ORDER BY m.start_time DESC
"""
history = query_db('l2', sql_history, [steam_id])
history = query_db('l2', sql_history, [steam_id] + roster_ids)
# 3. Aggregation by ELO
elo_buckets = {
@@ -389,11 +374,18 @@ class OpponentService:
LEFT JOIN fact_match_teams fmt_gid ON mp.match_id = fmt_gid.match_id AND fmt_gid.group_id = mp.team_id
LEFT JOIN fact_match_teams fmt_tid ON mp.match_id = fmt_tid.match_id AND fmt_tid.group_tid = mp.match_team_id
WHERE CAST(mp.steam_id_64 AS TEXT) NOT IN ({roster_ph})
AND EXISTS (
SELECT 1
FROM fact_match_players roster_mp
WHERE roster_mp.match_id = mp.match_id
AND CAST(roster_mp.steam_id_64 AS TEXT) IN ({roster_ph})
AND roster_mp.team_id != mp.team_id
)
AND m.map_name IS NOT NULL AND m.map_name <> ''
GROUP BY m.map_name
ORDER BY matches DESC
"""
rows = query_db('l2', sql, roster_ids)
rows = query_db('l2', sql, roster_ids + roster_ids)
results = []
for r in rows:
d = dict(r)
web/services/player_profile_service.py
+123
View File
@@ -0,0 +1,123 @@
from web.database import query_db
class PlayerProfileService:
PERIOD_KEYS = (
'career',
'last_10',
'last_20',
'last_30',
'days_30',
'days_90',
)
@staticmethod
def get_period_stats(steam_id):
rows = query_db(
'l3',
"""
SELECT *
FROM dm_player_period_stats
WHERE steam_id_64 = ?
ORDER BY CASE period_key
WHEN 'career' THEN 1
WHEN 'last_10' THEN 2
WHEN 'last_20' THEN 3
WHEN 'last_30' THEN 4
WHEN 'days_30' THEN 5
WHEN 'days_90' THEN 6
ELSE 99
END
""",
[steam_id],
)
return [dict(row) for row in rows]
@staticmethod
def get_period(steam_id, period_key):
if period_key not in PlayerProfileService.PERIOD_KEYS:
return None
row = query_db(
'l3',
"""
SELECT *
FROM dm_player_period_stats
WHERE steam_id_64 = ? AND period_key = ?
""",
[steam_id, period_key],
one=True,
)
return dict(row) if row else None
@staticmethod
def get_records(steam_id):
rows = query_db(
'l3',
"""
SELECT *
FROM dm_player_records
WHERE steam_id_64 = ?
ORDER BY CASE record_key
WHEN 'highest_rating' THEN 1
WHEN 'most_kills' THEN 2
WHEN 'highest_adr' THEN 3
WHEN 'highest_kd' THEN 4
WHEN 'most_headshots' THEN 5
WHEN 'longest_win_streak' THEN 6
ELSE 99
END
""",
[steam_id],
)
return [dict(row) for row in rows]
@staticmethod
def get_period_history(steam_id, period_key):
period = PlayerProfileService.get_period(steam_id, period_key)
if not period:
return []
rows = query_db(
'l3',
"""
SELECT
match_date AS start_time,
rating,
kd_ratio,
adr,
kast,
match_id,
map_name,
is_win,
match_sequence AS match_index
FROM dm_player_match_history
WHERE steam_id_64 = ?
AND match_date BETWEEN ? AND ?
ORDER BY match_date, match_id
""",
[steam_id, period['period_start'], period['period_end']],
)
return [dict(row) for row in rows]
@staticmethod
def get_map_stats(steam_id):
rows = query_db(
'l3',
"""
SELECT
map_name,
matches,
wins,
win_rate,
avg_rating AS rating,
avg_kd AS kd,
avg_adr AS adr,
avg_kast AS kast,
best_rating,
worst_rating
FROM dm_player_map_stats
WHERE steam_id_64 = ?
ORDER BY matches DESC, map_name
""",
[steam_id],
)
return [dict(row) for row in rows]
web/services/stats_service.py
+96 -157
View File
@@ -1,4 +1,4 @@
from web.database import query_db, execute_db
from web.database import query_db
from flask import current_app, url_for
import os
@@ -13,7 +13,7 @@ class StatsService:
try:
# Check local file first (User Request: "directly associate if exists")
base = os.path.join(current_app.root_path, 'static', 'avatars')
for ext in ('.jpg', '.png', '.jpeg'):
for ext in ('.jpg', '.png', '.jpeg', '.webp'):
fname = f"{steam_id}{ext}"
fpath = os.path.join(base, fname)
if os.path.exists(fpath):
@@ -38,18 +38,9 @@ class StatsService:
'round_stats': [{'type', 'count', 'wins', 'win_rate'}]
}
"""
# 1. Get Active Roster
from web.services.web_service import WebService
import json
from web.services.team_context_service import TeamContextService
lineups = WebService.get_lineups()
active_roster_ids = []
if lineups:
try:
raw_ids = json.loads(lineups[0]['player_ids_json'])
active_roster_ids = [str(uid) for uid in raw_ids]
except:
pass
active_roster_ids = TeamContextService.get_active_roster_ids()
if not active_roster_ids:
return {}
@@ -60,21 +51,23 @@ class StatsService:
placeholders = ','.join('?' for _ in active_roster_ids)
# Step A: Get Candidate Match IDs (matches with >= 2 roster players)
# Also get the team_id of our players in that match to determine win
candidate_sql = f"""
SELECT mp.match_id, MAX(mp.team_id) as our_team_id
SELECT mp.match_id, mp.team_id as our_team_id,
COUNT(DISTINCT mp.steam_id_64) as roster_count
FROM fact_match_players mp
WHERE CAST(mp.steam_id_64 AS TEXT) IN ({placeholders})
GROUP BY mp.match_id
GROUP BY mp.match_id, mp.team_id
HAVING COUNT(DISTINCT mp.steam_id_64) >= 2
ORDER BY mp.match_id, roster_count DESC, mp.team_id
"""
candidate_rows = query_db('l2', candidate_sql, active_roster_ids)
if not candidate_rows:
return {}
candidate_map = {row['match_id']: row['our_team_id'] for row in candidate_rows}
candidate_map = {}
for row in candidate_rows:
candidate_map.setdefault(row['match_id'], row['our_team_id'])
match_ids = list(candidate_map.keys())
match_placeholders = ','.join('?' for _ in match_ids)
@@ -221,11 +214,15 @@ class StatsService:
args.append(map_name)
if date_from:
where_clauses.append("start_time >= ?")
where_clauses.append(
"start_time >= CAST(strftime('%s', ?) AS INTEGER)"
)
args.append(date_from)
if date_to:
where_clauses.append("start_time <= ?")
where_clauses.append(
"start_time < CAST(strftime('%s', date(?, '+1 day')) AS INTEGER)"
)
args.append(date_to)
where_str = " AND ".join(where_clauses)
@@ -270,109 +267,51 @@ class StatsService:
party_rows = query_db('l2', party_sql, match_ids)
party_map = {row['match_id']: row['max_party'] for row in party_rows}
# --- New: Determine "Our Team" Result ---
# Logic: Check if any player from `active_roster` played in these matches.
# Use WebService to get the active roster
from web.services.web_service import WebService
import json
from web.services.team_context_service import TeamContextService
lineups = WebService.get_lineups()
active_roster_ids = []
if lineups:
try:
# Load IDs and ensure they are all strings for DB comparison consistency
raw_ids = json.loads(lineups[0]['player_ids_json'])
active_roster_ids = [str(uid) for uid in raw_ids]
except:
pass
active_roster_ids = TeamContextService.get_active_roster_ids()
# If no roster, we can't determine "Our Result"
if not active_roster_ids:
result_map = {}
else:
# 1. Get UIDs for Roster Members involved in these matches
# We query fact_match_players to ensure we get the UIDs actually used in these matches
roster_placeholders = ','.join('?' for _ in active_roster_ids)
uid_sql = f"""
SELECT DISTINCT steam_id_64, uid
roster_team_sql = f"""
SELECT match_id, team_id,
COUNT(DISTINCT steam_id_64) as roster_count
FROM fact_match_players
WHERE match_id IN ({placeholders})
AND CAST(steam_id_64 AS TEXT) IN ({roster_placeholders})
GROUP BY match_id, team_id
"""
combined_args_uid = match_ids + active_roster_ids
uid_rows = query_db('l2', uid_sql, combined_args_uid)
roster_team_rows = query_db(
'l2',
roster_team_sql,
match_ids + active_roster_ids,
)
winner_by_match = {
str(match['match_id']): match['winner_team']
for match in matches
}
teams_by_match = {}
for row in roster_team_rows:
teams_by_match.setdefault(str(row['match_id']), []).append(
row['team_id']
)
# Set of "Our UIDs" (as strings)
our_uids = set()
for r in uid_rows:
if r['uid']:
our_uids.add(str(r['uid']))
# 2. Get Group UIDs and Winner info from fact_match_teams
# We need to know which group contains our UIDs
teams_sql = f"""
SELECT fmt.match_id, fmt.group_id, fmt.group_uids, m.winner_team
FROM fact_match_teams fmt
JOIN fact_matches m ON fmt.match_id = m.match_id
WHERE fmt.match_id IN ({placeholders})
"""
teams_rows = query_db('l2', teams_sql, match_ids)
# 3. Determine Result per Match
result_map = {}
for match_id, team_ids in teams_by_match.items():
unique_team_ids = set(team_ids)
if len(unique_team_ids) > 1:
result_map[match_id] = 'mixed'
continue
our_team_id = next(iter(unique_team_ids))
result_map[match_id] = (
'win'
if str(our_team_id) == str(winner_by_match.get(match_id))
else 'loss'
)
# Group data by match
match_groups = {} # match_id -> {group_id: [uids...], winner: int}
for r in teams_rows:
mid = r['match_id']
gid = r['group_id']
uids_str = r['group_uids'] or ""
# Split and clean UIDs
uids = set(str(u).strip() for u in uids_str.split(',') if u.strip())
if mid not in match_groups:
match_groups[mid] = {'groups': {}, 'winner': r['winner_team']}
match_groups[mid]['groups'][gid] = uids
# Analyze
for mid, data in match_groups.items():
winner_gid = data['winner']
groups = data['groups']
our_in_winner = False
our_in_loser = False
# Check each group
for gid, uids in groups.items():
# Intersection of Our UIDs and Group UIDs
common = our_uids.intersection(uids)
if common:
if gid == winner_gid:
our_in_winner = True
else:
our_in_loser = True
if our_in_winner and not our_in_loser:
result_map[mid] = 'win'
elif our_in_loser and not our_in_winner:
result_map[mid] = 'loss'
elif our_in_winner and our_in_loser:
result_map[mid] = 'mixed'
else:
# Fallback: If UID matching failed (maybe missing UIDs), try old team_id method?
# Or just leave it as None (safe)
pass
# Convert to dict to modify
matches = [dict(m) for m in matches]
for m in matches:
m['avg_elo'] = elo_map.get(m['match_id'], 0)
m['max_party'] = party_map.get(m['match_id'], 1)
m['our_result'] = result_map.get(m['match_id'])
# Convert to dict to modify
matches = [dict(m) for m in matches]
for m in matches:
m['avg_elo'] = elo_map.get(m['match_id'], 0)
@@ -542,33 +481,20 @@ class StatsService:
@staticmethod
def get_shared_matches(steam_ids):
# Find matches where ALL steam_ids were present
if not steam_ids or len(steam_ids) < 1:
return []
steam_ids = list(dict.fromkeys(str(steam_id) for steam_id in steam_ids))
placeholders = ','.join('?' for _ in steam_ids)
count = len(steam_ids)
# We need to know which team the players were on to determine win/loss
# Assuming they were on the SAME team for "shared experience"
# If count=1, it's just match history
# Query: Get matches where all steam_ids are present
# Also join to get team_id to check if they were on the same team (optional but better)
# For simplicity in v1: Just check presence in the match.
# AND check if the player won.
# We need to return: match_id, map_name, score, result (Win/Loss)
# "Result" is relative to the lineup.
# If they were on the winning team, it's a Win.
sql = f"""
SELECT m.match_id, m.start_time, m.map_name, m.score_team1, m.score_team2, m.winner_team,
MAX(mp.team_id) as player_team_id -- Just take one team_id (assuming same)
mp.team_id as player_team_id
FROM fact_matches m
JOIN fact_match_players mp ON m.match_id = mp.match_id
WHERE mp.steam_id_64 IN ({placeholders})
GROUP BY m.match_id
GROUP BY m.match_id, mp.team_id
HAVING COUNT(DISTINCT mp.steam_id_64) = ?
ORDER BY m.start_time DESC
"""
@@ -580,14 +506,7 @@ class StatsService:
results = []
for r in rows:
# Determine if Win
# winner_team in DB is 'Team 1' or 'Team 2' usually, or the team name.
# fact_matches.winner_team stores the NAME of the winner? Or 'team1'/'team2'?
# Let's check how L2_Builder stores it. Usually it stores the name.
# But fact_match_players.team_id stores the name too.
# Logic: If m.winner_team == mp.team_id, then Win.
is_win = (r['winner_team'] == r['player_team_id'])
is_win = str(r['winner_team']) == str(r['player_team_id'])
# If winner_team is NULL or empty, it's a draw?
if not r['winner_team']:
@@ -628,7 +547,31 @@ class StatsService:
"""
l3_rows = query_db("l3", l3_sql, [steam_id, limit])
if l3_rows:
return l3_rows
history = [dict(row) for row in l3_rows]
match_ids = [row['match_id'] for row in history]
placeholders = ','.join('?' for _ in match_ids)
party_rows = query_db(
"l2",
f"""
SELECT me.match_id, COUNT(p.steam_id_64) AS party_size
FROM fact_match_players me
LEFT JOIN fact_match_players p
ON p.match_id = me.match_id
AND p.match_team_id = me.match_team_id
AND me.match_team_id > 0
WHERE me.steam_id_64 = ?
AND me.match_id IN ({placeholders})
GROUP BY me.match_id
""",
[steam_id] + match_ids,
)
party_map = {
row['match_id']: max(int(row['party_size'] or 0), 1)
for row in party_rows
}
for row in history:
row['party_size'] = party_map.get(row['match_id'], 1)
return history
sql = """
SELECT * FROM (
@@ -729,19 +672,10 @@ class StatsService:
Calculates rank and distribution of the target player within the active roster.
Now covers all L3 Basic Features for Detailed Panel.
"""
from web.services.web_service import WebService
from web.services.feature_service import FeatureService
import json
from web.services.team_context_service import TeamContextService
# 1. Get Active Roster IDs
lineups = WebService.get_lineups()
active_roster_ids = []
if lineups:
try:
raw_ids = json.loads(lineups[0]['player_ids_json'])
active_roster_ids = [str(uid) for uid in raw_ids]
except:
pass
active_roster_ids = TeamContextService.get_active_roster_ids()
if not active_roster_ids:
return None
@@ -851,33 +785,38 @@ class StatsService:
"basic_avg_rating", "basic_avg_kd", "basic_avg_adr", "basic_avg_kast", "basic_avg_rws",
]
lower_is_better = []
lower_is_better = {
"int_timing_first_contact_time",
"int_trade_response_time",
"tac_avg_fd",
"tac_fd_rate",
"core_avg_match_duration",
"core_dpr",
"meta_rating_volatility",
"meta_map_stability",
"meta_elo_tier_stability",
}
result = {}
for m in metrics:
values = []
non_numeric = False
for p in stats_map.values():
raw = (p or {}).get(m)
if raw is None:
raw = 0
continue
try:
values.append(float(raw))
except Exception:
non_numeric = True
break
except (TypeError, ValueError):
continue
raw_target = (stats_map.get(target_steam_id) or {}).get(m)
if raw_target is None:
raw_target = 0
result[m] = None
continue
try:
target_val = float(raw_target)
except Exception:
non_numeric = True
target_val = 0
if non_numeric:
except (TypeError, ValueError):
result[m] = None
continue
web/services/team_context_service.py
+36
View File
@@ -0,0 +1,36 @@
import json
from web.services.web_service import WebService
class TeamContextService:
"""Single source of truth for the private team's active roster."""
@staticmethod
def get_active_lineup():
lineup = WebService.get_active_lineup()
return dict(lineup) if lineup else None
@staticmethod
def get_active_roster_ids():
lineup = TeamContextService.get_active_lineup()
if not lineup:
return []
try:
raw_ids = json.loads(lineup.get('player_ids_json') or '[]')
except (TypeError, json.JSONDecodeError):
return []
if not isinstance(raw_ids, list):
return []
seen = set()
roster_ids = []
for raw_id in raw_ids:
steam_id = str(raw_id).strip()
if steam_id and steam_id not in seen:
seen.add(steam_id)
roster_ids.append(steam_id)
return roster_ids
web/services/web_service.py
+25 -3
View File
@@ -53,17 +53,39 @@ class WebService:
sql = "UPDATE team_lineups SET name=?, description=?, player_ids_json=? WHERE id=?"
return execute_db('web', sql, [name, description, ids_json, lineup_id])
else:
sql = "INSERT INTO team_lineups (name, description, player_ids_json) VALUES (?, ?, ?)"
return execute_db('web', sql, [name, description, ids_json])
active = 0 if WebService.get_active_lineup() else 1
sql = """
INSERT INTO team_lineups
(name, description, player_ids_json, is_active)
VALUES (?, ?, ?, ?)
"""
return execute_db('web', sql, [name, description, ids_json, active])
@staticmethod
def get_lineups():
return query_db('web', "SELECT * FROM team_lineups ORDER BY created_at DESC")
return query_db(
'web',
"SELECT * FROM team_lineups ORDER BY is_active DESC, created_at DESC, id DESC",
)
@staticmethod
def get_lineup(lineup_id):
return query_db('web', "SELECT * FROM team_lineups WHERE id = ?", [lineup_id], one=True)
@staticmethod
def get_active_lineup():
lineup = query_db(
'web',
"SELECT * FROM team_lineups WHERE is_active = 1 ORDER BY id LIMIT 1",
one=True,
)
if lineup:
return lineup
return query_db(
'web',
"SELECT * FROM team_lineups ORDER BY created_at DESC, id DESC LIMIT 1",
one=True,
)
# --- Users / Auth ---
@staticmethod
web/templates/admin/dashboard.html
+12 -9
View File
@@ -12,9 +12,8 @@
<div class="border border-gray-200 dark:border-gray-700 rounded-lg p-4">
<h3 class="text-lg font-bold text-gray-900 dark:text-white mb-4">数据管线 (ETL)</h3>
<div class="space-y-2">
<button onclick="triggerEtl('L1A.py')" class="w-full bg-blue-600 text-white py-2 px-4 rounded hover:bg-blue-700">Trigger L1A (Ingest)</button>
<button onclick="triggerEtl('L2_Builder.py')" class="w-full bg-blue-600 text-white py-2 px-4 rounded hover:bg-blue-700">Trigger L2 Builder</button>
<button onclick="triggerEtl('L3_Builder.py')" class="w-full bg-blue-600 text-white py-2 px-4 rounded hover:bg-blue-700">Trigger L3 Builder</button>
<a href="{{ url_for('admin.import_match') }}" class="block w-full text-center bg-yrtv-600 text-white py-2 px-4 rounded hover:bg-yrtv-500">上传并导入比赛</a>
<button onclick="triggerEtl()" class="w-full bg-blue-600 text-white py-2 px-4 rounded hover:bg-blue-700">运行完整 L1 → L2 → L3</button>
</div>
<div id="etlResult" class="mt-4 text-sm text-gray-600 dark:text-gray-400"></div>
</div>
@@ -23,6 +22,7 @@
<div class="border border-gray-200 dark:border-gray-700 rounded-lg p-4">
<h3 class="text-lg font-bold text-gray-900 dark:text-white mb-4">工具箱</h3>
<div class="space-y-2">
<a href="{{ url_for('admin.data_integrity') }}" class="block w-full text-center bg-emerald-600 text-white py-2 px-4 rounded hover:bg-emerald-700">数据完整性中心</a>
<a href="{{ url_for('admin.sql_runner') }}" class="block w-full text-center bg-gray-600 text-white py-2 px-4 rounded hover:bg-gray-700">SQL Runner</a>
<a href="{{ url_for('wiki.index') }}" class="block w-full text-center bg-gray-600 text-white py-2 px-4 rounded hover:bg-gray-700">Manage Wiki</a>
</div>
@@ -31,20 +31,23 @@
</div>
<script>
function triggerEtl(scriptName) {
function triggerEtl() {
const resultDiv = document.getElementById('etlResult');
resultDiv.innerText = "Triggering " + scriptName + "...";
resultDiv.innerText = "正在创建后台流水线...";
fetch("{{ url_for('admin.trigger_etl') }}", {
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
},
body: 'script=' + scriptName
})
.then(response => response.text())
.then(text => {
resultDiv.innerText = text;
.then(response => response.json())
.then(data => {
if (data.success) {
window.location.href = "{{ url_for('admin.import_match') }}?job_id=" + data.job_id;
} else {
resultDiv.innerText = data.error || "启动失败";
}
})
.catch(err => {
resultDiv.innerText = "Error: " + err;
web/templates/admin/data_integrity.html
+84
View File
@@ -0,0 +1,84 @@
{% extends "base.html" %}
{% block title %}数据完整性 - YRTV{% endblock %}
{% block content %}
{% set status_styles = {
'pass': 'bg-emerald-100 text-emerald-800 dark:bg-emerald-900/40 dark:text-emerald-300',
'warn': 'bg-amber-100 text-amber-800 dark:bg-amber-900/40 dark:text-amber-300',
'fail': 'bg-red-100 text-red-800 dark:bg-red-900/40 dark:text-red-300'
} %}
<div class="space-y-6 px-4 sm:px-0">
<div class="flex flex-col gap-4 sm:flex-row sm:items-center sm:justify-between">
<div>
<div class="flex items-center gap-3">
<h1 class="text-2xl font-bold text-slate-900 dark:text-white">数据完整性中心</h1>
<span class="rounded-full px-3 py-1 text-xs font-semibold uppercase {{ status_styles[report.overall_status] }}">
{{ report.overall_status }}
</span>
</div>
<p class="mt-1 text-sm text-slate-500 dark:text-slate-400">
校验时间：{{ report.generated_at }}
</p>
</div>
<div class="flex gap-2">
<a href="{{ url_for('admin.data_integrity', format='json') }}"
class="rounded-lg border border-slate-300 px-4 py-2 text-sm font-medium text-slate-700 hover:bg-slate-50 dark:border-slate-600 dark:text-slate-200 dark:hover:bg-slate-800">
JSON
</a>
<a href="{{ url_for('admin.data_integrity') }}"
class="rounded-lg bg-yrtv-600 px-4 py-2 text-sm font-medium text-white hover:bg-yrtv-500">
重新校验
</a>
</div>
</div>
<div class="grid grid-cols-3 gap-3">
{% for status, label in [('pass', '通过'), ('warn', '警告'), ('fail', '失败')] %}
<div class="rounded-xl bg-white p-4 shadow dark:bg-slate-800">
<div class="text-sm text-slate-500 dark:text-slate-400">{{ label }}</div>
<div class="mt-1 text-3xl font-bold {% if status == 'pass' %}text-emerald-600{% elif status == 'warn' %}text-amber-600{% else %}text-red-600{% endif %}">
{{ report.totals[status] }}
</div>
</div>
{% endfor %}
</div>
<div class="rounded-xl bg-white p-5 shadow dark:bg-slate-800">
<h2 class="mb-4 text-lg font-semibold text-slate-900 dark:text-white">数据规模</h2>
<div class="grid grid-cols-2 gap-3 sm:grid-cols-3 lg:grid-cols-5">
{% for key, value in report.counts.items() %}
<div class="rounded-lg bg-slate-50 p-3 dark:bg-slate-900/60">
<div class="truncate text-xs uppercase tracking-wide text-slate-500">{{ key|replace('_', ' ') }}</div>
<div class="mt-1 text-xl font-semibold text-slate-900 dark:text-white">{{ "{:,}".format(value) }}</div>
</div>
{% endfor %}
</div>
</div>
<div class="overflow-hidden rounded-xl bg-white shadow dark:bg-slate-800">
<div class="border-b border-slate-200 px-5 py-4 dark:border-slate-700">
<h2 class="text-lg font-semibold text-slate-900 dark:text-white">校验项目</h2>
</div>
<div class="divide-y divide-slate-100 dark:divide-slate-700">
{% for check in report.checks %}
<div class="flex flex-col gap-2 px-5 py-4 sm:flex-row sm:items-center sm:justify-between">
<div>
<div class="font-medium text-slate-900 dark:text-white">{{ check.name }}</div>
<div class="mt-1 text-sm text-slate-500 dark:text-slate-400">{{ check.detail }}</div>
</div>
<span class="self-start rounded-full px-3 py-1 text-xs font-semibold uppercase sm:self-center {{ status_styles[check.status] }}">
{{ check.status }}
</span>
</div>
{% endfor %}
</div>
</div>
<div>
<a href="{{ url_for('admin.dashboard') }}" class="text-sm font-medium text-yrtv-600 hover:text-yrtv-500">
返回管理后台
</a>
</div>
</div>
{% endblock %}
web/templates/admin/import_match.html
+153
View File
@@ -0,0 +1,153 @@
{% extends "base.html" %}
{% block title %}比赛导入 - YRTV{% endblock %}
{% block content %}
<div class="space-y-6 px-4 sm:px-0">
{% with messages = get_flashed_messages(with_categories=true) %}
{% if messages %}
<div class="space-y-2">
{% for category, message in messages %}
<div class="rounded-lg px-4 py-3 text-sm {% if category == 'success' %}bg-emerald-100 text-emerald-800{% elif category == 'warning' %}bg-amber-100 text-amber-800{% else %}bg-red-100 text-red-800{% endif %}">
{{ message }}
</div>
{% endfor %}
</div>
{% endif %}
{% endwith %}
<div class="flex items-center justify-between">
<div>
<h1 class="text-2xl font-bold text-slate-900 dark:text-white">比赛数据导入</h1>
<p class="mt-1 text-sm text-slate-500 dark:text-slate-400">
上传完整的 iframe_network.json，系统会自动识别比赛 ID，并执行 L1 → L2 → L3。
</p>
</div>
<a href="{{ url_for('admin.dashboard') }}" class="text-sm font-medium text-yrtv-600 hover:text-yrtv-500">
返回管理后台
</a>
</div>
<div class="grid grid-cols-1 gap-6 lg:grid-cols-3">
<div class="rounded-xl bg-white p-6 shadow dark:bg-slate-800">
<h2 class="text-lg font-semibold text-slate-900 dark:text-white">上传抓包</h2>
<form method="POST" enctype="multipart/form-data" class="mt-5 space-y-4">
<div>
<label class="block text-sm font-medium text-slate-700 dark:text-slate-300">
iframe_network.json
</label>
<input type="file" name="capture" accept=".json,application/json" required
class="mt-2 block w-full text-sm text-slate-600 file:mr-4 file:rounded-lg file:border-0 file:bg-yrtv-50 file:px-4 file:py-2 file:font-medium file:text-yrtv-700 hover:file:bg-yrtv-100 dark:text-slate-300">
</div>
<label class="flex items-start gap-2 text-sm text-slate-600 dark:text-slate-300">
<input type="checkbox" name="replace" value="1" class="mt-1 rounded border-slate-300 text-yrtv-600">
<span>允许替换已存在但内容不同的比赛。流水线失败时会自动恢复数据库。</span>
</label>
<button type="submit"
class="w-full rounded-lg bg-yrtv-600 px-4 py-2.5 font-medium text-white hover:bg-yrtv-500">
校验并开始导入
</button>
</form>
<div class="mt-5 rounded-lg bg-amber-50 p-3 text-xs text-amber-800 dark:bg-amber-900/30 dark:text-amber-200">
直接重复上传会被拒绝。每次流水线运行前都会备份 L1、L2、L3，并执行后置完整性检查。
</div>
</div>
<div class="rounded-xl bg-white p-6 shadow dark:bg-slate-800 lg:col-span-2">
<div class="flex items-center justify-between">
<h2 class="text-lg font-semibold text-slate-900 dark:text-white">作业状态</h2>
{% if selected_job %}
<span id="job-status" class="rounded-full bg-slate-100 px-3 py-1 text-xs font-semibold uppercase text-slate-700 dark:bg-slate-700 dark:text-slate-200">
{{ selected_job.status }}
</span>
{% endif %}
</div>
{% if selected_job %}
<div class="mt-5">
<div class="flex justify-between text-sm text-slate-600 dark:text-slate-300">
<span id="job-stage">{{ selected_job.current_stage or 'queued' }}</span>
<span id="job-progress-label">{{ selected_job.progress }}%</span>
</div>
<div class="mt-2 h-2 overflow-hidden rounded-full bg-slate-200 dark:bg-slate-700">
<div id="job-progress" class="h-full bg-yrtv-500 transition-all" style="width: {{ selected_job.progress }}%"></div>
</div>
<p id="job-message" class="mt-3 text-sm text-slate-600 dark:text-slate-300">
{{ selected_job.message or '等待执行' }}
</p>
<pre id="job-log" class="mt-4 max-h-96 overflow-auto whitespace-pre-wrap rounded-lg bg-slate-950 p-4 text-xs text-slate-200">{{ selected_job.log_text }}</pre>
</div>
{% else %}
<div class="mt-8 rounded-lg border border-dashed border-slate-300 p-8 text-center text-sm text-slate-500 dark:border-slate-600">
上传比赛或从下方选择历史作业。
</div>
{% endif %}
</div>
</div>
<div class="overflow-hidden rounded-xl bg-white shadow dark:bg-slate-800">
<div class="border-b border-slate-200 px-5 py-4 dark:border-slate-700">
<h2 class="font-semibold text-slate-900 dark:text-white">最近作业</h2>
</div>
<div class="overflow-x-auto">
<table class="min-w-full divide-y divide-slate-200 dark:divide-slate-700">
<thead class="bg-slate-50 dark:bg-slate-900/50">
<tr>
<th class="px-5 py-3 text-left text-xs font-semibold uppercase text-slate-500">ID</th>
<th class="px-5 py-3 text-left text-xs font-semibold uppercase text-slate-500">类型</th>
<th class="px-5 py-3 text-left text-xs font-semibold uppercase text-slate-500">比赛</th>
<th class="px-5 py-3 text-left text-xs font-semibold uppercase text-slate-500">状态</th>
<th class="px-5 py-3 text-left text-xs font-semibold uppercase text-slate-500">阶段</th>
<th class="px-5 py-3 text-left text-xs font-semibold uppercase text-slate-500">耗时</th>
</tr>
</thead>
<tbody class="divide-y divide-slate-100 dark:divide-slate-700">
{% for job in jobs %}
<tr class="hover:bg-slate-50 dark:hover:bg-slate-700/40">
<td class="px-5 py-3 text-sm">
<a href="{{ url_for('admin.import_match', job_id=job.id) }}" class="font-medium text-yrtv-600">#{{ job.id }}</a>
</td>
<td class="px-5 py-3 text-sm text-slate-600 dark:text-slate-300">{{ job.job_type }}</td>
<td class="px-5 py-3 text-sm font-mono text-slate-600 dark:text-slate-300">{{ job.match_id or '-' }}</td>
<td class="px-5 py-3 text-sm text-slate-600 dark:text-slate-300">{{ job.status }}</td>
<td class="px-5 py-3 text-sm text-slate-600 dark:text-slate-300">{{ job.current_stage or '-' }}</td>
<td class="px-5 py-3 text-sm text-slate-600 dark:text-slate-300">{{ '%.2fs'|format(job.duration_seconds) if job.duration_seconds is not none else '-' }}</td>
</tr>
{% else %}
<tr><td colspan="6" class="px-5 py-8 text-center text-sm text-slate-500">暂无作业</td></tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
</div>
{% endblock %}
{% block scripts %}
{% if selected_job %}
<script>
const jobId = {{ selected_job.id }};
let pollTimer = null;
async function refreshJob() {
const response = await fetch(`/admin/api/jobs/${jobId}`);
if (!response.ok) return;
const job = await response.json();
document.getElementById('job-status').textContent = job.status;
document.getElementById('job-stage').textContent = job.current_stage || job.status;
document.getElementById('job-progress-label').textContent = `${job.progress}%`;
document.getElementById('job-progress').style.width = `${job.progress}%`;
document.getElementById('job-message').textContent = job.message || '';
const log = document.getElementById('job-log');
log.textContent = job.log_text || '';
log.scrollTop = log.scrollHeight;
if (job.status === 'succeeded' || job.status === 'failed') {
clearInterval(pollTimer);
}
}
pollTimer = setInterval(refreshJob, 1500);
refreshJob();
</script>
{% endif %}
{% endblock %}
web/templates/players/_career_dashboard.html
+88
View File
@@ -0,0 +1,88 @@
<div class="grid grid-cols-1 gap-8 xl:grid-cols-2">
<section class="rounded-2xl border border-gray-100 bg-white p-6 shadow-lg dark:border-slate-700 dark:bg-slate-800"
x-data="{ selected: 'last_20', periods: {{ period_stats|tojson }} }">
<div class="flex flex-col gap-4 sm:flex-row sm:items-center sm:justify-between">
<div>
<h3 class="text-lg font-bold text-gray-900 dark:text-white">阶段表现</h3>
<p class="text-xs text-gray-500">窗口按该玩家最新一场比赛向前计算</p>
</div>
<select x-model="selected"
class="rounded-lg border-gray-200 bg-gray-50 text-sm dark:border-slate-600 dark:bg-slate-700 dark:text-white">
{% for period in period_stats %}
<option value="{{ period.period_key }}">{{ period.period_label }}</option>
{% endfor %}
</select>
</div>
{% for period in period_stats %}
<div x-show="selected === '{{ period.period_key }}'"
{% if period.period_key != 'last_20' %}style="display:none"{% endif %}
class="mt-6">
<div class="grid grid-cols-2 gap-3 sm:grid-cols-3">
<div class="rounded-xl bg-gray-50 p-3 dark:bg-slate-700/40">
<div class="text-xs font-bold uppercase text-gray-400">Matches</div>
<div class="mt-1 text-2xl font-black text-gray-900 dark:text-white">{{ period.matches }}</div>
</div>
<div class="rounded-xl bg-gray-50 p-3 dark:bg-slate-700/40">
<div class="text-xs font-bold uppercase text-gray-400">Rating</div>
<div class="mt-1 text-2xl font-black text-yrtv-600">{{ '%.2f'|format(period.avg_rating or 0) }}</div>
</div>
<div class="rounded-xl bg-gray-50 p-3 dark:bg-slate-700/40">
<div class="text-xs font-bold uppercase text-gray-400">K/D</div>
<div class="mt-1 text-2xl font-black text-gray-900 dark:text-white">{{ '%.2f'|format(period.avg_kd or 0) }}</div>
</div>
<div class="rounded-xl bg-gray-50 p-3 dark:bg-slate-700/40">
<div class="text-xs font-bold uppercase text-gray-400">ADR</div>
<div class="mt-1 text-2xl font-black text-gray-900 dark:text-white">{{ '%.1f'|format(period.avg_adr or 0) }}</div>
</div>
<div class="rounded-xl bg-gray-50 p-3 dark:bg-slate-700/40">
<div class="text-xs font-bold uppercase text-gray-400">KAST</div>
<div class="mt-1 text-2xl font-black text-gray-900 dark:text-white">{{ '%.1f%%'|format((period.avg_kast or 0) * 100) }}</div>
</div>
<div class="rounded-xl bg-gray-50 p-3 dark:bg-slate-700/40">
<div class="text-xs font-bold uppercase text-gray-400">Win Rate</div>
<div class="mt-1 text-2xl font-black {% if period.win_rate >= 0.5 %}text-green-600{% else %}text-red-500{% endif %}">
{{ '%.0f%%'|format((period.win_rate or 0) * 100) }}
</div>
</div>
</div>
<div class="mt-3 text-xs text-gray-400">
{% if period.sample_reliable %}
样本充足
{% else %}
样本不足，仅供参考
{% endif %}
</div>
</div>
{% else %}
<div class="mt-8 text-center text-sm text-gray-400">暂无阶段统计</div>
{% endfor %}
</section>
<section class="rounded-2xl border border-gray-100 bg-white p-6 shadow-lg dark:border-slate-700 dark:bg-slate-800">
<div>
<h3 class="text-lg font-bold text-gray-900 dark:text-white">职业纪录</h3>
<p class="text-xs text-gray-500">每项纪录都可追溯到具体比赛</p>
</div>
<div class="mt-6 grid grid-cols-2 gap-3 sm:grid-cols-3">
{% for record in records %}
<a href="{{ url_for('matches.detail', match_id=record.match_id) }}"
class="rounded-xl border border-gray-100 bg-gray-50 p-3 transition hover:border-yrtv-300 hover:bg-yrtv-50 dark:border-slate-600 dark:bg-slate-700/40 dark:hover:bg-slate-700">
<div class="truncate text-xs font-bold uppercase text-gray-400">{{ record.record_label }}</div>
<div class="mt-1 text-2xl font-black text-gray-900 dark:text-white">
{% if record.record_key in ['most_kills', 'most_headshots', 'longest_win_streak'] %}
{{ record.record_value|int }}
{% elif record.record_key == 'highest_adr' %}
{{ '%.1f'|format(record.record_value) }}
{% else %}
{{ '%.2f'|format(record.record_value) }}
{% endif %}
</div>
<div class="mt-2 truncate text-[10px] font-mono text-gray-400">{{ record.map_name }}</div>
</a>
{% else %}
<div class="col-span-full py-8 text-center text-sm text-gray-400">暂无职业纪录</div>
{% endfor %}
</div>
</section>
</div>
web/templates/players/profile.html
+36 -5
View File
@@ -213,6 +213,8 @@
</div>
</div>
{% include "players/_career_dashboard.html" %}
<!-- 2. Charts Section (Middle) -->
<div class="grid grid-cols-1 lg:grid-cols-3 gap-8">
<!-- Trend Chart -->
@@ -221,8 +223,11 @@
<h3 class="text-lg font-bold text-gray-900 dark:text-white flex items-center gap-2">
<span>📈</span> 近期表现走势 (Performance Trend)
</h3>
<div class="flex bg-gray-100 dark:bg-slate-700 rounded-lg p-1">
<button class="px-3 py-1 text-xs font-bold rounded-md bg-white dark:bg-slate-600 shadow-sm text-gray-800 dark:text-white">Recent 20</button>
<div id="trend-period-buttons" class="flex flex-wrap bg-gray-100 dark:bg-slate-700 rounded-lg p-1">
<button onclick="loadTrendPeriod('last_10', this)" class="px-3 py-1 text-xs font-bold rounded-md text-gray-500">10</button>
<button onclick="loadTrendPeriod('last_20', this)" class="trend-active px-3 py-1 text-xs font-bold rounded-md bg-white dark:bg-slate-600 shadow-sm text-gray-800 dark:text-white">20</button>
<button onclick="loadTrendPeriod('last_30', this)" class="px-3 py-1 text-xs font-bold rounded-md text-gray-500">30</button>
<button onclick="loadTrendPeriod('career', this)" class="px-3 py-1 text-xs font-bold rounded-md text-gray-500">Career</button>
</div>
</div>
<div class="relative h-80 w-full">
@@ -256,7 +261,7 @@
</div>
{% macro detail_item(label, value, key, format_str='{:.2f}', sublabel=None, count_label=None) %}
{% set dist = distribution[key] if distribution else None %}
{% set dist = distribution[key] if distribution and distribution[key] else None %}
<div class="flex flex-col group relative h-full p-2 rounded hover:bg-gray-50 dark:hover:bg-slate-700/30 transition-colors">
<div class="flex justify-between items-center mb-1">
<span class="text-xs font-bold text-gray-400 uppercase tracking-wider truncate max-w-[150px]" title="{{ label }}">{{ label }}</span>
@@ -273,7 +278,11 @@
<div class="flex justify-between items-end mb-1">
<div class="flex items-baseline gap-1">
<span class="text-lg font-black text-gray-900 dark:text-white font-mono">
{{ format_str.format(value if value is not none else 0) }}
{% if value is none %}
<span class="text-sm text-gray-400">N/A</span>
{% else %}
{{ format_str.format(value) }}
{% endif %}
</span>
{% if sublabel %}
<span class="text-[10px] text-gray-400">{{ sublabel }}</span>
@@ -834,6 +843,7 @@
{% block scripts %}
<script>
let trendChartInstance = null;
const profileSteamId = "{{ player.steam_id_64 }}";
function resetZoom() {
if (trendChartInstance) {
@@ -853,8 +863,29 @@ function likeComment(commentId, btn) {
});
}
function loadTrendPeriod(periodKey, button) {
fetch(`/players/${profileSteamId}/charts_data?period=${periodKey}`)
.then(response => response.json())
.then(data => {
if (!trendChartInstance) return;
trendChartInstance.data.labels = data.trend.labels;
trendChartInstance.data.datasets[0].data = data.trend.values;
trendChartInstance.data.datasets[1].data = Array(data.trend.labels.length).fill(1.5);
trendChartInstance.data.datasets[2].data = Array(data.trend.labels.length).fill(1.0);
trendChartInstance.data.datasets[3].data = Array(data.trend.labels.length).fill(0.6);
trendChartInstance.update();
document.querySelectorAll('#trend-period-buttons button').forEach(item => {
item.classList.remove('bg-white', 'dark:bg-slate-600', 'shadow-sm', 'text-gray-800', 'dark:text-white');
item.classList.add('text-gray-500');
});
button.classList.remove('text-gray-500');
button.classList.add('bg-white', 'dark:bg-slate-600', 'shadow-sm', 'text-gray-800', 'dark:text-white');
});
}
document.addEventListener('DOMContentLoaded', function() {
const steamId = "{{ player.steam_id_64 }}";
const steamId = profileSteamId;
fetch(`/players/${steamId}/charts_data`)
.then(response => response.json())