diff --git a/.gitignore b/.gitignore index af70359..a655ac1 100644 --- a/.gitignore +++ b/.gitignore @@ -47,6 +47,7 @@ coverage.xml local_settings.py db.sqlite3 +*.db.bak instance/ @@ -66,7 +67,9 @@ venv.bak/ output/ output_arena/ +database/backups/ +database/.pipeline.lock arena/ scripts/ experiment -yrtv.zip \ No newline at end of file +yrtv.zip diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..47dd256 --- /dev/null +++ b/Makefile @@ -0,0 +1,32 @@ +PYTHON := .venv/bin/python + +.PHONY: install run test check l1 l2 l3 l3-all pipeline + +install: + python3 -m venv .venv + $(PYTHON) -m pip install -r requirements.txt + +run: + $(PYTHON) -m web.app + +test: + $(PYTHON) -m unittest discover -v + +check: + $(PYTHON) -m compileall -q web database tests wsgi.py + $(PYTHON) -m unittest discover -v + +l1: + $(PYTHON) database/L1/L1_Builder.py + +l2: + $(PYTHON) database/L2/L2_Builder.py + +l3: + $(PYTHON) database/L3/L3_Builder.py + +l3-all: + $(PYTHON) database/L3/L3_Builder.py --force + +pipeline: + $(PYTHON) -c "from web.app import create_app; create_app(); from database.job_store import JobStore; from database.pipeline import run_pipeline; job_id = JobStore().create_job('manual_pipeline', created_by='cli'); print('job_id=', job_id); raise SystemExit(0 if run_pipeline(job_id) else 1)" diff --git a/README.md b/README.md index a473c85..c598e80 100644 --- a/README.md +++ b/README.md @@ -1,154 +1,269 @@ -# YRTV 项目说明 till 1.0.2hotfix +# YRTV 2.0.0 Alpha -## 项目概览 -YRTV 是一个基于 CS2 比赛数据的综合分析与战队管理平台。它集成了数据采集、ETL 清洗建模、特征挖掘以及现代化的 Web 交互界面。 -核心目标是为战队提供数据驱动的决策支持，包括战术分析、队员表现评估、阵容管理（Clubhouse）以及实时战术板功能。 +YRTV 是面向固定 CS2 战队的私人数据站。它不是公共玩家排行榜，而是让队员拥有类似职业选手的个人主页，并为战队提供比赛档案、队内比较、阵容分析、对手情报和战术工具。 ---- +当前 Alpha 版本已经建立可重复运行的数据流水线、数据库治理和职业主页数据集市，重点服务 active roster。 -您可以使用以下命令快速配置环境： -pip install -r requirements.txt +## 当前基线 -数据来源与处理核心包括： -- 比赛页面的 iframe JSON 数据（`iframe_network.json`） -- 可选的 demo 文件（`.zip/.dem`） -- L1A/L2/L3 分层数据库建模与校验 +- 208 场比赛 +- 1,181 名采集到的玩家 +- 2,080 条玩家比赛记录 +- 4,315 个回合 +- 33,560 条回合事件 +- 38,423 条经济记录 +- 9 名 active roster 队员 +- 885 条 roster 逐场历史 +- 76 条地图统计 +- 236 条武器统计 +- 54 条时间窗口统计 +- 54 条个人职业纪录 +- 24 项自动化测试通过 +- 28 项数据完整性检查通过 -## v3.0.0 Release 更新要点 -- **核心算法升级**: 严格确立 Active Roster (Lineup 1) 为战队平均数据计算基准，修复了雷达图与平均数据的计算偏差。 -- **Clubhouse 增强**: - - 布局优化为 3 列网格。 - - 新增 **OVR (Overall Score)** 显示，优先展示真实评分 (Real Rating)，直观反映选手综合实力。 -- **Tactics 系统**: - - 统一评分逻辑：全站优先采用 L3 `core_avg_rating2` (真实评分)，智能回退至 `basic_avg_rating`。 - - Data Center 数据中心现在完整映射了 Utility、Trading 等高阶战术数据。 -- **稳定性修复**: 修正了特征服务中的语法错误，增强了对缺失数据的鲁棒性处理。 +数据规模会随导入变化，Admin 数据完整性中心显示的结果是运行时事实。 -## Web 交互系统 (Core) -基于 Flask + TailwindCSS + Alpine.js 构建的现代化 Web 应用。 +## 核心功能 -### 核心功能模块 -1. **Clubhouse (战队管理)** - - **Roster Management**: 拖拽式管理当前激活阵容 (Active Roster)。 - - **Scout System**: 全库模糊搜索玩家，支持按 Rating/Matches/KD 排序筛选。 - - **Contract System**: 模拟签约/解约流程 (Sign/Release)，管理战队资产。 - - **Identity**: 统一的头像与 ID 显示逻辑 (SteamID/Name)，支持自动生成首字母头像。 +### 玩家职业主页 -2. **Tactics Board (战术终端)** - - **SPA 架构**: 基于 Alpine.js 的单页应用，无刷新切换四大功能区。 - - **Board (战术板)**: 集成 Leaflet.js 的交互式地图，支持战术点位标记。 - - **Data (数据中心)**: 实时查看全队近期数据表现，集成 Utility/Trading 等高阶战术指标。 - - **Analysis (深度分析)**: - - **Chemistry**: 任意组合 (2-5人) 的共同比赛胜率与数据分析。 - - **Depth**: 阵容深度与位置分析。 - - **Economy (经济计算)**: 简单的经济局/长枪局计算器。 +- Rating、K/D、ADR、KAST 等生涯数据 +- Aim、Clutch、Pistol、Defense、Utility、Stability、Economy、Pace 八维能力 +- 生涯、最近 10/20/30 场、最近 30/90 天阶段统计 +- 可切换时间窗口的 Rating 趋势 +- 最高 Rating、最多击杀、最高 ADR、最高 K/D、最多爆头和最长连胜 +- 每项个人纪录可追溯到具体比赛 +- 地图表现、比赛历史、Party 信息、队内排名和留言板 +- 缺失或尚未实现的指标显示为 `N/A`，不使用伪造分数 -3. **Match Center (比赛中心)** - - **List View**: - - 显示比赛平均 ELO。 - - **Party Identification**: 自动识别组排车队 (👥 2-5)，并用颜色区分规模 (Indigo/Blue/Purple/Orange)。 - - **Result Tracking**: 基于 "Our Team" (Active Roster) 的胜负判定 (VICTORY/DEFEAT/CIVIL WAR)。 - - **Detail View**: - - 按 Rating 降序排列双方队员。 - - 高亮显示组排关系。 - - 集成 Round-by-Round 经济与事件详情。 +### 比赛中心 -4. **Player Profile (玩家档案)** - - 综合能力雷达图 (八维数据: Aim, Clutch, Pistol, Defense, Util, Stability, Economy, Pace)。 - - 近期 Rating/KD/ADR 趋势折线图。 - - 详细的历史比赛记录（含 Party info 与 Result）。 - - 头像上传与管理。 +- 比赛列表、地图、比分、平均 ELO 和己方结果 +- Active roster 与 Party 识别 +- 双方玩家表现和 Rating 排序 +- Head-to-head 击杀矩阵 +- 回合事件、经济和装备信息 +- 原始比赛数据查看 -## 自动化与运维 -新增 `ETL/refresh.py` 自动化脚本，用于一键执行全量数据刷新： -- 自动清理旧数据库。 -- 顺序执行 L1A -> L2 -> L3 构建。 -- 自动处理 schema 迁移。 +### 战队与战术 -## 数据流程 -1. **下载与落盘** - 通过 `downloader/downloader.py` 抓取比赛页面数据，生成 `output_arena//iframe_network.json`，并可同时下载 demo 文件。 -2. **L1A 入库（原始 JSON）** - `ETL/L1A.py` 将 `output_arena/*/iframe_network.json` 批量写入 `database/L1A/L1A.sqlite`。 -3. **L2 入库（结构化事实表/维度表）** - `ETL/L2_Builder.py` 读取 L1A 数据，按 `database/L2/schema.sql` 构建维度表与事实表，生成 `database/L2/L2_Main.sqlite`。 -4. **L3 入库（特征集市）** - `ETL/L3_Builder.py` 读取 L2 数据，计算 Basic 及 6 大挖掘能力维度特征，生成 `database/L3/L3_Features.sqlite`。 -5. **质量校验与覆盖分析** - `ETL/verify/verify_L2.py` 与 `ETL/verify/verify_deep.py` 用于 L2 字段覆盖与逻辑检查。 +- Active roster 管理 +- 玩家搜索、签入和移出 +- 2-5 人同队比赛与 Chemistry 分析 +- 对手档案和真实交手记录 +- 地图战术板、阵容数据中心和经济工具 +- Wiki、玩家标签、备注和评论 + +### 数据运营 + +- Admin 上传 `iframe_network.json` +- 自动识别唯一 `g161-*` 比赛 ID +- JSON 结构、必要接口、哈希和重复比赛校验 +- 后台执行 L1 → L2 → L3 +- 实时查看作业阶段、进度、日志和耗时 +- 数据完整性中心与 JSON 报告 + +## 快速开始 + +环境要求： + +- macOS/Linux +- Python 3.9+ +- SQLite 3 + +安装并启动： + +```bash +make install + +export SECRET_KEY='replace-with-a-random-secret' +export ADMIN_TOKEN='replace-with-an-admin-token' + +make run +``` + +默认地址： + +- 应用：`http://127.0.0.1:5000` +- Admin：`/admin/` +- 比赛导入：`/admin/import-match` +- 数据完整性：`/admin/data-integrity` + +生产进程入口： + +```bash +.venv/bin/gunicorn wsgi:app +``` + +## 常用命令 + +```bash +make run # 启动 Flask +make check # 编译检查 + 自动化测试 +make pipeline # 备份后执行完整 L1 -> L2 -> L3 + +make l1 # 仅构建 L1 +make l2 # 仅构建 L2 +make l3 # 仅构建 active roster L3 +make l3-all # 为全部采集玩家构建 L3 +``` + +正常维护优先使用 `make pipeline`。单层命令主要用于开发和排错。 + +## 比赛导入 + +推荐从 Admin 页面上传完整的 `iframe_network.json`。 + +导入流程： + +1. 验证 UTF-8 和 JSON 结构。 +2. 从网络 URL 中提取唯一比赛 ID。 +3. 检查 match 和 round 必要接口。 +4. 计算 SHA256，拒绝相同数据重复导入。 +5. 保存到 `output_arena//iframe_network.json`。 +6. 创建 `etl_jobs` 作业。 +7. 备份 L1/L2/L3。 +8. 串行执行三个 Builder。 +9. 验证目标比赛具有 10 名玩家和回合事实。 +10. 成功提交；失败自动恢复备份。 + +仓库当前不包含自动访问 5E 网页的下载器，因此首页 URL 输入不会抓取数据。 + +## 数据架构 + +```text +iframe_network.json + | + v +L1 raw capture + | + v +L2 normalized facts + | + v +L3 roster features and profile marts + | + v +Flask services and player profiles +``` + +### L1：原始层 + +- 数据库：`database/L1/L1.db` +- Builder：`database/L1/L1_Builder.py` +- Grain：每场比赛一份完整网络抓包 +- 核心表：`raw_iframe_network` + +### L2：事实层 + +- 数据库：`database/L2/L2.db` +- Schema：`database/L2/schema.sql` +- Builder：`database/L2/L2_Builder.py` +- 核心表： + - `dim_players` + - `dim_maps` + - `fact_matches` + - `fact_match_teams` + - `fact_match_players` + - `fact_match_players_t` + - `fact_match_players_ct` + - `fact_rounds` + - `fact_round_events` + - `fact_round_player_economy` + +### L3：特征与主页集市 + +- 数据库：`database/L3/L3.db` +- Schema：`database/L3/schema.sql` +- Builder：`database/L3/L3_Builder.py` +- 核心表： + - `dm_player_features` + - `dm_player_match_history` + - `dm_player_map_stats` + - `dm_player_weapon_stats` + - `dm_player_period_stats` + - `dm_player_records` + +### Web：应用状态 + +- 数据库：`database/Web/Web_App.sqlite` +- Schema：`database/Web/schema.sql` +- 当前 schema version：2 +- 保存 lineup、玩家备注、评论、Wiki、战术板、导入登记和 ETL 作业 + +## 数据库治理 + +- 所有运行路径集中定义在 `database/paths.py` +- 完整编排入口为 `database/pipeline.py` +- 同一时间只允许一个 pipeline +- Pipeline 运行前备份 L1/L2/L3 +- 失败时恢复三层数据库，Web 作业日志继续保留 +- 备份位于 `database/backups/` +- 自动保留最近 3 组备份 +- Web schema 使用 `schema_migrations` 记录版本 +- 高频玩家历史、Party、事件和经济查询具有专用索引 +- 数据库和目录规则详见 `database/README.md` + +## 数据质量 + +Admin 数据完整性中心检查： + +- 四个 SQLite 数据库的 `quick_check` +- 必要表和 Web schema version +- 玩家比赛、回合事件的引用完整性 +- 每场比赛玩家数量 +- 玩家身份覆盖 +- 高频查询索引 +- Active roster 的 L3 特征覆盖 +- 逐场历史与总场次一致性 +- 真实队内 percentile +- 地图、武器、时间窗口和职业纪录集市 +- 占位空间指标 +- Web 外键、active lineup 和 pipeline 并发 +- 备份数量与存储规模 + +运行测试： + +```bash +make check +``` ## 目录结构 -``` + +```text yrtv/ -├── downloader/ # 下载器（抓取 iframe JSON 与 demo） -├── ETL/ # ETL 脚本 -│ ├── L1A.py -│ ├── L2_Builder.py -│ ├── L3_Builder.py -│ ├── refresh.py # [NEW] 一键刷新脚本 -│ └── verify/ -├── database/ # SQLite 数据库存储 -│ ├── L1A/ -│ ├── L2/ -│ ├── L3/ -│ └── original_json_schema/ -├── web/ # [NEW] Web 应用程序 -│ ├── app.py # 应用入口 -│ ├── routes/ # 路由 (matches, players, teams, tactics) -│ ├── services/ # 业务逻辑 (stats, web) -│ ├── templates/ # Jinja2 模板 (TailwindCSS + Alpine.js) -│ └── static/ # 静态资源 (CSS, JS, Uploads) -└── utils/ - └── json_extractor/ # JSON Schema 抽取工具 +├── database/ +│ ├── L1/ # 原始抓包与 Builder +│ ├── L2/ # 事实层、Schema、Processor +│ ├── L3/ # 特征层、Schema、Processor +│ ├── Web/ # 应用数据库 Schema +│ ├── paths.py # 统一路径 +│ ├── maintenance.py # 备份、恢复、健康检查 +│ ├── job_store.py # ETL 作业状态 +│ └── pipeline.py # 完整流水线 +├── tests/ # 自动化测试 +├── utils/ # JSON 结构分析工具 +├── web/ +│ ├── routes/ +│ ├── services/ +│ ├── templates/ +│ └── static/ +├── Makefile +├── requirements.txt +└── wsgi.py ``` -## 环境要求 -- Python 3.11.4+ -- Flask, Jinja2 -- Playwright（下载器依赖） -- pandas, numpy（数据处理依赖） +## Alpha 限制 -## 数据库层级说明 -### L1A -- **用途**：保存原始 iframe JSON -- **输入**：`output_arena/*/iframe_network.json` -- **输出**：`database/L1A/L1A.sqlite` -- **脚本**：`ETL/L1A.py` +- 当前主要数据源为 5E iframe 网络响应。 +- 不包含自动网页下载器和 Demo parser。 +- 认证仍是单一 Admin Token，适合私人部署，不适合开放注册。 +- SQLite 适合当前单战队规模，不面向高并发多租户。 +- 部分高级空间能力需要地图边界、路径和 Demo 数据，当前显示 `N/A`。 +- `StatsService` 仍保留部分兼容逻辑，后续会继续按领域拆分。 -### L1B -- **用途**：保存 demo 解析后的原始数据（由 demoparser2 产出） -- **输出**：`database/L1B/L1B.sqlite` -- 当前仓库提供目录与说明，解析流程需结合外部工具执行 +## 版本 -### L2 -结构化事实表/维度表数据库，覆盖比赛、玩家、回合与经济等数据： -- **Schema**：`database/L2/schema.sql` -- **输出**：`database/L2/L2_Main.sqlite` -- **核心表**： - - `dim_players`、`dim_maps` - - `fact_matches`、`fact_match_teams` - - `fact_match_players`、`fact_match_players_t`、`fact_match_players_ct` - - `fact_rounds`、`fact_round_events`、`fact_round_player_economy` +当前版本：`2.0.0 Alpha` -### L3 -玩家特征集市 (Player Features Data Mart)，聚合 Basic 及 6 大挖掘能力维度 (STA, BAT, HPS, PTL, T/CT, UTIL)。 -- **Schema**：`database/L3/schema.sql` -- **输出**：`database/L3/L3_Features.sqlite` -- **脚本**：`ETL/L3_Builder.py` -- **核心表**：`dm_player_features` (玩家聚合画像) - -## JSON Schema 抽取工具 -用于分析大量 `iframe_network.json` 的字段结构与覆盖情况，支持动态 Key 归并与多格式输出。 - -输出内容通常位于 `output_reports/` 或 `database/original_json_schema/`，包括： -- `schema_summary.md`：结构概览 -- `schema_flat.csv`：扁平字段列表 -- `uncovered_features.csv`：未覆盖字段清单 - -## 数据源互斥说明 -L2 中 `fact_matches.data_source_type` 用于区分数据来源与字段覆盖范围： -- `classic`：含 round_list 详细回合与坐标信息 -- `leetify`：含 leetify 评分与经济信息 -- `unknown`：无法识别来源 - -入库逻辑保持互斥：同一场比赛只会按其来源覆盖相应字段，避免重复或冲突。 +这一版本的目标是建立可信、可恢复、可持续导入的私人战队 HLTV 基线，而不是冻结产品功能。 diff --git a/database/L1/L1A.db b/database/L1/L1A.db deleted file mode 100644 index e69de29..0000000 diff --git a/database/L1/L1_Builder.py b/database/L1/L1_Builder.py index 350f2bd..ab7b516 100644 --- a/database/L1/L1_Builder.py +++ b/database/L1/L1_Builder.py @@ -14,13 +14,18 @@ import os import json import sqlite3 import glob -import argparse # Added +import argparse +import sys -# Paths BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -OUTPUT_ARENA_DIR = os.path.join(BASE_DIR, 'output_arena') -DB_DIR = os.path.join(BASE_DIR, 'database', 'L1') -DB_PATH = os.path.join(DB_DIR, 'L1.db') +if BASE_DIR not in sys.path: + sys.path.insert(0, BASE_DIR) + +from database.paths import L1_DB, L1_DIR, OUTPUT_ARENA + +OUTPUT_ARENA_DIR = str(OUTPUT_ARENA) +DB_DIR = str(L1_DIR) +DB_PATH = str(L1_DB) def init_db(): if not os.path.exists(DB_DIR): @@ -65,6 +70,7 @@ def process_files(): count = 0 skipped = 0 + errors = 0 for file_path in files: try: @@ -92,11 +98,14 @@ def process_files(): conn.commit() except Exception as e: + errors += 1 print(f"Error processing {file_path}: {e}") conn.commit() conn.close() - print(f"Finished. Processed: {count}, Skipped: {skipped}.") + print(f"Finished. Processed: {count}, Skipped: {skipped}, Errors: {errors}.") + if errors: + raise RuntimeError(f"L1 ingestion failed for {errors} file(s)") if __name__ == '__main__': - process_files() \ No newline at end of file + process_files() diff --git a/database/L1/README.md b/database/L1/README.md index 137bf6b..69a64bb 100644 --- a/database/L1/README.md +++ b/database/L1/README.md @@ -1,16 +1,25 @@ -L1A 5eplay平台网页爬虫原始数据。 +# L1 Raw Match Store -## ETL Step 1: -从原始json数据库提取到L1A级数据库中。 -`output_arena/*/iframe_network.json` -> `database/L1A/L1A.sqlite` +L1 stores one complete 5E network capture per match without transforming its +payload. -### 脚本说明 -- **脚本位置**: `ETL/L1A.py` -- **功能**: 自动遍历 `output_arena` 目录下所有的 `iframe_network.json` 文件，提取原始内容并以 `match_id` (文件夹名) 为主键存入 `L1A.sqlite` 数据库的 `raw_iframe_network` 表中。 +## Runtime Files -### 运行方式 -使用项目指定的 Python 环境运行脚本： +- Database: `database/L1/L1.db` +- Builder: `database/L1/L1_Builder.py` +- Input: `output_arena//iframe_network.json` +- Primary key: `raw_iframe_network.match_id` + +## Commands ```bash -C:/ProgramData/anaconda3/python.exe ETL/L1A.py +make l1 +make pipeline ``` + +Normal ingestion is incremental. `--force` re-reads every capture currently +present in `output_arena`. + +`L1A.db` and the historical `database/L1A/L1A.sqlite` path are retired. L1B is +reserved for a future demo-parser source and is not part of the runtime +pipeline. diff --git a/database/L2/L2.db b/database/L2/L2.db index 1a3984c..1eb5817 100644 Binary files a/database/L2/L2.db and b/database/L2/L2.db differ diff --git a/database/L2/L2_Builder.py b/database/L2/L2_Builder.py index 753bc60..68dcfc8 100644 --- a/database/L2/L2_Builder.py +++ b/database/L2/L2_Builder.py @@ -7,14 +7,20 @@ from dataclasses import dataclass, field from typing import List, Dict, Optional, Any, Tuple from datetime import datetime +PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +if PROJECT_ROOT not in sys.path: + sys.path.insert(0, PROJECT_ROOT) + +from database.paths import L1_DB, L2_DB, L2_SCHEMA + # Setup logging logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') logger = logging.getLogger(__name__) # Constants -L1A_DB_PATH = 'database/L1/L1.db' -L2_DB_PATH = 'database/L2/L2.db' -SCHEMA_PATH = 'database/L2/schema.sql' +L1A_DB_PATH = str(L1_DB) +L2_DB_PATH = str(L2_DB) +SCHEMA_PATH = str(L2_SCHEMA) # --- Data Structures for Unification --- @@ -1238,6 +1244,10 @@ def process_matches(): l1_conn.close() l2_conn.close() logger.info(f"\nDone. Processed {count} matches ({success_count} success, {error_count} errors).") + if error_count: + raise RuntimeError( + f"L2 build failed for {error_count}/{count} matches" + ) if __name__ == "__main__": process_matches() diff --git a/database/L2/schema.sql b/database/L2/schema.sql index 77ecec6..4226cef 100644 --- a/database/L2/schema.sql +++ b/database/L2/schema.sql @@ -636,3 +636,24 @@ SELECT FROM fact_match_players fmp JOIN fact_matches fm ON fmp.match_id = fm.match_id GROUP BY fmp.steam_id_64, fm.map_name; + +-- ========================================== +-- Operational query indexes +-- ========================================== +CREATE INDEX IF NOT EXISTS idx_match_players_player_match +ON fact_match_players(steam_id_64, match_id); + +CREATE INDEX IF NOT EXISTS idx_match_players_match_team +ON fact_match_players(match_id, team_id, steam_id_64); + +CREATE INDEX IF NOT EXISTS idx_match_players_party +ON fact_match_players(match_id, match_team_id, steam_id_64); + +CREATE INDEX IF NOT EXISTS idx_round_events_victim +ON fact_round_events(victim_steam_id, match_id); + +CREATE INDEX IF NOT EXISTS idx_economy_player_match +ON fact_round_player_economy(steam_id_64, match_id, round_num); + +CREATE INDEX IF NOT EXISTS idx_matches_map_time +ON fact_matches(map_name, start_time DESC); diff --git a/database/L3/L3.db b/database/L3/L3.db index 5fc8a70..430987a 100644 Binary files a/database/L3/L3.db and b/database/L3/L3.db differ diff --git a/database/L3/L3_Builder.py b/database/L3/L3_Builder.py index 40743c4..e0ae7c6 100644 --- a/database/L3/L3_Builder.py +++ b/database/L3/L3_Builder.py @@ -6,6 +6,8 @@ import sqlite3 import json import argparse import concurrent.futures +from collections import defaultdict, deque +from typing import Optional # Setup logging logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') @@ -15,10 +17,14 @@ logger = logging.getLogger(__name__) BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) # Points to database/ directory PROJECT_ROOT = os.path.dirname(BASE_DIR) # Points to project root sys.path.insert(0, PROJECT_ROOT) # Add project root to Python path -L2_DB_PATH = os.path.join(BASE_DIR, 'L2', 'L2.db') -L3_DB_PATH = os.path.join(BASE_DIR, 'L3', 'L3.db') -WEB_DB_PATH = os.path.join(BASE_DIR, 'Web', 'Web_App.sqlite') -SCHEMA_PATH = os.path.join(BASE_DIR, 'L3', 'schema.sql') + +from database.paths import L2_DB, L3_DB, L3_SCHEMA, WEB_DB + +L2_DB_PATH = str(L2_DB) +L3_DB_PATH = str(L3_DB) +L3_BACKUP_PATH = f"{L3_DB_PATH}.bak" +WEB_DB_PATH = str(WEB_DB) +SCHEMA_PATH = str(L3_SCHEMA) def _get_existing_columns(conn, table_name): cur = conn.execute(f"PRAGMA table_info({table_name})") @@ -76,7 +82,28 @@ def _get_team_players(): try: conn = sqlite3.connect(WEB_DB_PATH) cursor = conn.cursor() - cursor.execute("SELECT player_ids_json FROM team_lineups") + columns = { + row[1] for row in cursor.execute("PRAGMA table_info(team_lineups)") + } + if 'is_active' in columns: + cursor.execute( + """ + SELECT player_ids_json + FROM team_lineups + WHERE is_active = 1 + ORDER BY created_at DESC, id DESC + LIMIT 1 + """ + ) + else: + cursor.execute( + """ + SELECT player_ids_json + FROM team_lineups + ORDER BY created_at DESC, id DESC + LIMIT 1 + """ + ) rows = cursor.fetchall() steam_ids = set() @@ -150,7 +177,25 @@ def _build_player_record(steam_id: str): "error": str(e), } -def main(force_all: bool = False, workers: int = 1): +def _backup_l3_database(source_path=L3_DB_PATH, backup_path=L3_BACKUP_PATH): + if not os.path.exists(source_path): + return None + + source = sqlite3.connect(source_path) + backup = sqlite3.connect(backup_path) + try: + source.backup(backup) + result = backup.execute("PRAGMA quick_check").fetchone()[0] + if result != 'ok': + raise RuntimeError(f"L3 backup quick_check failed: {result}") + finally: + source.close() + backup.close() + logger.info("L3 backup created at %s", backup_path) + return backup_path + + +def main(force_all: bool = False, workers: int = 1, create_backup: bool = True): """ Main L3 feature building pipeline using modular processors """ @@ -158,6 +203,9 @@ def main(force_all: bool = False, workers: int = 1): logger.info("Starting L3 Builder with 5-Tier Architecture") logger.info("========================================") + if create_backup: + _backup_l3_database() + # 1. Ensure Schema is up to date init_db() @@ -181,6 +229,7 @@ def main(force_all: bool = False, workers: int = 1): conn_l3 = sqlite3.connect(L3_DB_PATH) try: + conn_l3.execute("BEGIN IMMEDIATE") cursor_l2 = conn_l2.cursor() if force_all: logger.info("Force mode enabled: building L3 for all players in L2.") @@ -240,7 +289,6 @@ def main(force_all: bool = False, workers: int = 1): ) success_count += 1 if processed_count % 2 == 0: - conn_l3.commit() logger.info(f"Progress: {processed_count}/{total_players} ({success_count} success, {error_count} errors)") else: for idx, row in enumerate(players, 1): @@ -268,10 +316,20 @@ def main(force_all: bool = False, workers: int = 1): processed_count = idx if processed_count % 2 == 0: - conn_l3.commit() logger.info(f"Progress: {processed_count}/{total_players} ({success_count} success, {error_count} errors)") - - # Final commit + + if error_count: + raise RuntimeError( + f"L3 feature build failed for {error_count}/{total_players} players" + ) + + processed_ids = [str(row[0]) for row in players] + _update_percentiles(conn_l3, processed_ids) + _rebuild_auxiliary_marts(conn_l2, conn_l3, processed_ids) + + quick_check = conn_l3.execute("PRAGMA quick_check").fetchone()[0] + if quick_check != 'ok': + raise RuntimeError(f"L3 quick_check failed before commit: {quick_check}") conn_l3.commit() logger.info("========================================") @@ -283,9 +341,11 @@ def main(force_all: bool = False, workers: int = 1): logger.info("========================================") except Exception as e: + conn_l3.rollback() logger.error(f"Fatal error during L3 build: {e}") import traceback traceback.print_exc() + raise finally: conn_l2.close() @@ -313,7 +373,7 @@ def _get_round_count(steam_id: str, conn_l2: sqlite3.Connection) -> int: def _upsert_features(conn_l3: sqlite3.Connection, steam_id: str, features: dict, - match_count: int, round_count: int, conn_l2: sqlite3.Connection | None, + match_count: int, round_count: int, conn_l2: Optional[sqlite3.Connection], first_match_date=None, last_match_date=None): """ Insert or update player features in dm_player_features @@ -353,12 +413,430 @@ def _upsert_features(conn_l3: sqlite3.Connection, steam_id: str, features: dict, cursor_l3.execute(sql, values) + +def _rebuild_auxiliary_marts(conn_l2, conn_l3, steam_ids): + """Rebuild player-grain marts used by profiles and trend APIs.""" + if not steam_ids: + return + + logger.info("Rebuilding L3 match, map and weapon marts") + total_history = 0 + total_maps = 0 + total_weapons = 0 + total_periods = 0 + total_records = 0 + + for start in range(0, len(steam_ids), 400): + chunk = steam_ids[start:start + 400] + placeholders = ','.join('?' for _ in chunk) + + for table in ( + 'dm_player_match_history', + 'dm_player_map_stats', + 'dm_player_weapon_stats', + 'dm_player_period_stats', + 'dm_player_records', + ): + conn_l3.execute( + f"DELETE FROM {table} WHERE steam_id_64 IN ({placeholders})", + chunk, + ) + + history_rows = conn_l2.execute( + f""" + SELECT + mp.steam_id_64, + mp.match_id, + m.start_time, + mp.rating, + mp.kd_ratio, + mp.adr, + mp.kast, + mp.is_win, + m.map_name, + mp.kills, + mp.deaths, + mp.headshot_count, + ( + SELECT AVG(teammate.rating) + FROM fact_match_players teammate + WHERE teammate.match_id = mp.match_id + AND teammate.team_id = mp.team_id + AND teammate.steam_id_64 != mp.steam_id_64 + ) AS teammate_avg_rating + FROM fact_match_players mp + JOIN fact_matches m ON m.match_id = mp.match_id + WHERE mp.steam_id_64 IN ({placeholders}) + ORDER BY mp.steam_id_64, m.start_time, mp.match_id + """, + chunk, + ).fetchall() + + history_values = [] + player_state = defaultdict(lambda: { + 'sequence': 0, + 'rating_sum': 0.0, + 'recent': deque(maxlen=10), + }) + for row in history_rows: + steam_id = str(row[0]) + state = player_state[steam_id] + rating = float(row[3] or 0.0) + state['sequence'] += 1 + state['rating_sum'] += rating + state['recent'].append(rating) + history_values.append(( + steam_id, + row[1], + row[2], + state['sequence'], + row[3], + row[4], + row[5], + row[6], + row[7], + row[8], + None, + row[12], + state['rating_sum'] / state['sequence'], + sum(state['recent']) / len(state['recent']), + )) + + conn_l3.executemany( + """ + INSERT INTO dm_player_match_history ( + steam_id_64, match_id, match_date, match_sequence, + rating, kd_ratio, adr, kast, is_win, map_name, + opponent_avg_elo, teammate_avg_rating, + cumulative_rating, rolling_10_rating + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + history_values, + ) + total_history += len(history_values) + + map_rows = conn_l2.execute( + f""" + SELECT + mp.steam_id_64, + m.map_name, + COUNT(*) AS matches, + SUM(CASE WHEN mp.is_win = 1 THEN 1 ELSE 0 END) AS wins, + AVG(mp.rating) AS avg_rating, + AVG(mp.kd_ratio) AS avg_kd, + AVG(mp.adr) AS avg_adr, + AVG(mp.kast) AS avg_kast, + MAX(mp.rating) AS best_rating, + MIN(mp.rating) AS worst_rating + FROM fact_match_players mp + JOIN fact_matches m ON m.match_id = mp.match_id + WHERE mp.steam_id_64 IN ({placeholders}) + AND m.map_name IS NOT NULL + AND m.map_name != '' + GROUP BY mp.steam_id_64, m.map_name + """, + chunk, + ).fetchall() + map_values = [ + tuple(row[:4]) + ( + (row[3] or 0) / row[2] if row[2] else 0.0, + ) + tuple(row[4:]) + for row in map_rows + ] + conn_l3.executemany( + """ + INSERT INTO dm_player_map_stats ( + steam_id_64, map_name, matches, wins, win_rate, + avg_rating, avg_kd, avg_adr, avg_kast, + best_rating, worst_rating + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + map_values, + ) + total_maps += len(map_values) + + round_counts = { + str(row[0]): int(row[1] or 0) + for row in conn_l2.execute( + f""" + SELECT steam_id_64, SUM(round_total) + FROM fact_match_players + WHERE steam_id_64 IN ({placeholders}) + GROUP BY steam_id_64 + """, + chunk, + ) + } + weapon_rows = conn_l2.execute( + f""" + SELECT + attacker_steam_id, + weapon, + COUNT(*) AS total_kills, + SUM(CASE WHEN is_headshot = 1 THEN 1 ELSE 0 END) AS total_headshots, + COUNT(DISTINCT match_id || ':' || round_num) AS usage_rounds + FROM fact_round_events + WHERE event_type = 'kill' + AND attacker_steam_id IN ({placeholders}) + AND weapon IS NOT NULL + AND weapon != '' + GROUP BY attacker_steam_id, weapon + """, + chunk, + ).fetchall() + weapon_values = [] + for row in weapon_rows: + rounds = round_counts.get(str(row[0]), 0) + kills = int(row[2] or 0) + headshots = int(row[3] or 0) + usage_rounds = int(row[4] or 0) + hs_rate = headshots / kills if kills else 0.0 + usage_rate = usage_rounds / rounds if rounds else 0.0 + kills_per_round = kills / rounds if rounds else 0.0 + effectiveness = kills / usage_rounds if usage_rounds else 0.0 + weapon_values.append(( + str(row[0]), + row[1], + kills, + headshots, + hs_rate, + usage_rounds, + usage_rate, + kills_per_round, + effectiveness, + )) + conn_l3.executemany( + """ + INSERT INTO dm_player_weapon_stats ( + steam_id_64, weapon_name, total_kills, total_headshots, + hs_rate, usage_rounds, usage_rate, + avg_kills_per_round, effectiveness_score + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + weapon_values, + ) + total_weapons += len(weapon_values) + + period_values = _calculate_period_rows(history_rows) + conn_l3.executemany( + """ + INSERT INTO dm_player_period_stats ( + steam_id_64, period_key, period_label, + period_start, period_end, matches, wins, win_rate, + avg_rating, avg_kd, avg_adr, avg_kast, + total_kills, total_deaths, sample_reliable + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + period_values, + ) + total_periods += len(period_values) + + record_values = _calculate_record_rows(history_rows) + conn_l3.executemany( + """ + INSERT INTO dm_player_records ( + steam_id_64, record_key, record_label, record_value, + match_id, map_name, match_date + ) VALUES (?, ?, ?, ?, ?, ?, ?) + """, + record_values, + ) + total_records += len(record_values) + + logger.info( + "Auxiliary marts rebuilt: %s history, %s map, %s weapon, " + "%s period, %s record rows", + total_history, + total_maps, + total_weapons, + total_periods, + total_records, + ) + + +def _group_player_match_rows(history_rows): + grouped = defaultdict(list) + for row in history_rows: + grouped[str(row['steam_id_64'])].append(row) + for rows in grouped.values(): + rows.sort(key=lambda row: (row['start_time'] or 0, row['match_id'])) + return grouped + + +def _safe_average(rows, key): + values = [float(row[key]) for row in rows if row[key] is not None] + return sum(values) / len(values) if values else None + + +def _calculate_period_rows(history_rows): + result = [] + for steam_id, all_rows in _group_player_match_rows(history_rows).items(): + latest_time = max(int(row['start_time'] or 0) for row in all_rows) + period_groups = [ + ('career', '生涯', all_rows), + ('last_10', '最近 10 场', all_rows[-10:]), + ('last_20', '最近 20 场', all_rows[-20:]), + ('last_30', '最近 30 场', all_rows[-30:]), + ( + 'days_30', + '最近 30 天', + [ + row for row in all_rows + if int(row['start_time'] or 0) >= latest_time - 30 * 86400 + ], + ), + ( + 'days_90', + '最近 90 天', + [ + row for row in all_rows + if int(row['start_time'] or 0) >= latest_time - 90 * 86400 + ], + ), + ] + + for period_key, period_label, rows in period_groups: + if not rows: + continue + matches = len(rows) + wins = sum(1 for row in rows if row['is_win']) + kills = sum(int(row['kills'] or 0) for row in rows) + deaths = sum(int(row['deaths'] or 0) for row in rows) + result.append(( + steam_id, + period_key, + period_label, + min(int(row['start_time'] or 0) for row in rows), + max(int(row['start_time'] or 0) for row in rows), + matches, + wins, + wins / matches, + _safe_average(rows, 'rating'), + kills / deaths if deaths else float(kills), + _safe_average(rows, 'adr'), + _safe_average(rows, 'kast'), + kills, + deaths, + 1 if matches >= 10 else 0, + )) + return result + + +def _calculate_record_rows(history_rows): + result = [] + metric_definitions = ( + ('highest_rating', '最高 Rating', 'rating'), + ('most_kills', '单场最多击杀', 'kills'), + ('highest_adr', '单场最高 ADR', 'adr'), + ('highest_kd', '单场最高 K/D', 'kd_ratio'), + ('most_headshots', '单场最多爆头', 'headshot_count'), + ) + + for steam_id, rows in _group_player_match_rows(history_rows).items(): + for record_key, record_label, field in metric_definitions: + candidates = [row for row in rows if row[field] is not None] + if not candidates: + continue + best = max( + candidates, + key=lambda row: ( + float(row[field]), + int(row['start_time'] or 0), + ), + ) + result.append(( + steam_id, + record_key, + record_label, + float(best[field]), + best['match_id'], + best['map_name'], + best['start_time'], + )) + + longest_streak = 0 + current_streak = 0 + streak_end = None + for row in rows: + if row['is_win']: + current_streak += 1 + if current_streak >= longest_streak: + longest_streak = current_streak + streak_end = row + else: + current_streak = 0 + if streak_end is not None: + result.append(( + steam_id, + 'longest_win_streak', + '最长连胜', + float(longest_streak), + streak_end['match_id'], + streak_end['map_name'], + streak_end['start_time'], + )) + return result + + +def _update_percentiles(conn_l3, steam_ids): + """Calculate a real percentile among eligible players in this build.""" + if not steam_ids: + return + + score_rows = [] + for start in range(0, len(steam_ids), 400): + chunk = steam_ids[start:start + 400] + placeholders = ','.join('?' for _ in chunk) + conn_l3.execute( + f""" + UPDATE dm_player_features + SET tier_percentile = NULL + WHERE steam_id_64 IN ({placeholders}) + """, + chunk, + ) + score_rows.extend(conn_l3.execute( + f""" + SELECT steam_id_64, score_overall + FROM dm_player_features + WHERE steam_id_64 IN ({placeholders}) + AND score_overall > 0 + """, + chunk, + ).fetchall()) + + if not score_rows: + return + + scores = [float(row[1]) for row in score_rows] + percentile_values = [] + for row in score_rows: + score = float(row[1]) + percentile = sum(value <= score for value in scores) / len(scores) * 100 + percentile_values.append((round(percentile, 2), str(row[0]))) + + conn_l3.executemany( + """ + UPDATE dm_player_features + SET tier_percentile = ? + WHERE steam_id_64 = ? + """, + percentile_values, + ) + logger.info("Updated percentiles for %s eligible players", len(score_rows)) + + def _parse_args(): parser = argparse.ArgumentParser() parser.add_argument("--force", action="store_true") parser.add_argument("--workers", type=int, default=1) + parser.add_argument("--no-backup", action="store_true") return parser.parse_args() if __name__ == "__main__": args = _parse_args() - main(force_all=args.force, workers=args.workers) + main( + force_all=args.force, + workers=args.workers, + create_backup=not args.no_backup, + ) diff --git a/database/L3/processors/composite_processor.py b/database/L3/processors/composite_processor.py index 1c30f8b..001ad4d 100644 --- a/database/L3/processors/composite_processor.py +++ b/database/L3/processors/composite_processor.py @@ -65,8 +65,8 @@ class CompositeProcessor(BaseFeatureProcessor): # Classify tier based on overall score features['tier_classification'] = CompositeProcessor._classify_tier(features['score_overall']) - # Percentile rank (placeholder - requires all players) - features['tier_percentile'] = min(features['score_overall'], 100.0) + # Filled by L3_Builder after every eligible player has been calculated. + features['tier_percentile'] = None return features @@ -266,13 +266,13 @@ class CompositeProcessor(BaseFeatureProcessor): STABILITY Score (0-100) | 8% """ # Extract features - volatility = features.get('meta_rating_volatility', 0.0) - loss_rating = features.get('meta_loss_rating', 0.0) - consistency = features.get('meta_rating_consistency', 0.0) - tilt_resilience = features.get('int_pressure_tilt_resistance', 0.0) - map_stable = features.get('meta_map_stability', 0.0) - elo_stable = features.get('meta_elo_tier_stability', 0.0) - recent_form = features.get('meta_recent_form_rating', 0.0) + volatility = features.get('meta_rating_volatility') or 0.0 + loss_rating = features.get('meta_loss_rating') or 0.0 + consistency = features.get('meta_rating_consistency') or 0.0 + tilt_resilience = features.get('int_pressure_tilt_resistance') or 0.0 + map_stable = features.get('meta_map_stability') or 0.0 + elo_stable = features.get('meta_elo_tier_stability') or 0.0 + recent_form = features.get('meta_recent_form_rating') or 0.0 # Normalize # Volatility: Reverse score. 100 - (Vol * 220) @@ -281,8 +281,8 @@ class CompositeProcessor(BaseFeatureProcessor): loss_score = min((loss_rating / 1.00) * 100, 100) cons_score = min((consistency / 70) * 100, 100) tilt_score = min((tilt_resilience / 0.80) * 100, 100) - map_score = min((map_stable / 0.25) * 100, 100) - elo_score = min((elo_stable / 0.48) * 100, 100) + map_score = max(0, min(100, 100 - (map_stable / 0.25) * 100)) + elo_score = max(0, min(100, 100 - (elo_stable / 0.48) * 100)) recent_score = min((recent_form / 1.15) * 100, 100) # Weighted Sum @@ -337,12 +337,12 @@ class CompositeProcessor(BaseFeatureProcessor): PACE Score (0-100) | 5% """ # Extract features - early_kill_pct = features.get('int_timing_early_kill_share', 0.0) - aggression = features.get('int_timing_aggression_index', 0.0) - trade_speed = features.get('int_trade_response_time', 0.0) - trade_kill = features.get('int_trade_kill_count', 0) - teamwork = features.get('int_teamwork_score', 0.0) - first_contact = features.get('int_timing_first_contact_time', 0.0) + early_kill_pct = features.get('int_timing_early_kill_share') or 0.0 + aggression = features.get('int_timing_aggression_index') or 0.0 + trade_speed = features.get('int_trade_response_time') or 0.0 + trade_kill = features.get('int_trade_kill_count') or 0 + teamwork = features.get('int_teamwork_score') or 0.0 + first_contact = features.get('int_timing_first_contact_time') or 0.0 # Normalize early_score = min((early_kill_pct / 0.44) * 100, 100) @@ -353,7 +353,7 @@ class CompositeProcessor(BaseFeatureProcessor): if trade_speed > 0.01: trade_speed_score = min((2.0 / trade_speed) * 100, 100) else: - trade_speed_score = 100 # Instant trade + trade_speed_score = 0 trade_kill_score = min((trade_kill / 650) * 100, 100) teamwork_score = min((teamwork / 29) * 100, 100) @@ -362,13 +362,7 @@ class CompositeProcessor(BaseFeatureProcessor): if first_contact > 0.01: first_contact_score = min((30 / first_contact) * 100, 100) else: - first_contact_score = 0 # If 0, probably no data, safe to say 0? Or 100? - # 0 first contact time means instant damage. - # But "30 / Contact" means smaller contact time gives higher score. - # If contact time is 0, score explodes. - # Realistically first contact time is > 0. - # I will clamp it. - first_contact_score = 100 # Assume very fast + first_contact_score = 0 # Weighted Sum pace_score = ( @@ -416,5 +410,5 @@ def _get_default_composite_features() -> Dict[str, Any]: 'score_pace': 0.0, 'score_overall': 0.0, 'tier_classification': 'Beginner', - 'tier_percentile': 0.0, + 'tier_percentile': None, } diff --git a/database/L3/processors/intelligence_processor.py b/database/L3/processors/intelligence_processor.py index e00bd15..28396d6 100644 --- a/database/L3/processors/intelligence_processor.py +++ b/database/L3/processors/intelligence_processor.py @@ -466,7 +466,8 @@ class IntelligenceProcessor(BaseFeatureProcessor): - int_pos_spatial_iq_score - int_pos_avg_distance_from_teammates - Note: Simplified implementation - full version requires DBSCAN clustering + Only geometry-independent values are calculated here. Metrics that + require map boundaries, paths or teammate positions remain NULL. """ cursor = conn_l2.cursor() @@ -481,26 +482,23 @@ class IntelligenceProcessor(BaseFeatureProcessor): has_position_data = cursor.fetchone()[0] > 0 if not has_position_data: - # Return placeholder values if no position data return { - 'int_pos_site_a_control_rate': 0.0, - 'int_pos_site_b_control_rate': 0.0, - 'int_pos_mid_control_rate': 0.0, - 'int_pos_favorite_position': 'unknown', - 'int_pos_position_diversity': 0.0, - 'int_pos_rotation_speed': 0.0, - 'int_pos_map_coverage': 0.0, - 'int_pos_lurk_tendency': 0.0, - 'int_pos_site_anchor_score': 0.0, - 'int_pos_entry_route_diversity': 0.0, - 'int_pos_retake_positioning': 0.0, - 'int_pos_postplant_positioning': 0.0, - 'int_pos_spatial_iq_score': 0.0, - 'int_pos_avg_distance_from_teammates': 0.0, + 'int_pos_site_a_control_rate': None, + 'int_pos_site_b_control_rate': None, + 'int_pos_mid_control_rate': None, + 'int_pos_favorite_position': None, + 'int_pos_position_diversity': None, + 'int_pos_rotation_speed': None, + 'int_pos_map_coverage': None, + 'int_pos_lurk_tendency': None, + 'int_pos_site_anchor_score': None, + 'int_pos_entry_route_diversity': None, + 'int_pos_retake_positioning': None, + 'int_pos_postplant_positioning': None, + 'int_pos_spatial_iq_score': None, + 'int_pos_avg_distance_from_teammates': None, } - # Simplified position analysis (proper implementation needs clustering) - # Calculate basic position variance as proxy for mobility cursor.execute(""" SELECT AVG(attacker_pos_x) as avg_x, @@ -515,34 +513,24 @@ class IntelligenceProcessor(BaseFeatureProcessor): pos_row = cursor.fetchone() position_count = pos_row[3] if pos_row[3] else 1 - # Position diversity based on unique grid cells visited position_diversity = min(position_count / 50.0, 1.0) # Normalize to 0-1 - - # Map coverage (simplified) map_coverage = position_diversity - - # Site control rates CANNOT be calculated without map-specific geometry data - # Each map (Dust2, Mirage, Nuke, etc.) has different site boundaries - # Would require: CREATE TABLE map_boundaries (map_name, site_name, min_x, max_x, min_y, max_y) - # Commenting out these 3 features: - # - int_pos_site_a_control_rate - # - int_pos_site_b_control_rate - # - int_pos_mid_control_rate + return { - 'int_pos_site_a_control_rate': 0.33, # Placeholder - 'int_pos_site_b_control_rate': 0.33, # Placeholder - 'int_pos_mid_control_rate': 0.34, # Placeholder - 'int_pos_favorite_position': 'mid', + 'int_pos_site_a_control_rate': None, + 'int_pos_site_b_control_rate': None, + 'int_pos_mid_control_rate': None, + 'int_pos_favorite_position': None, 'int_pos_position_diversity': round(position_diversity, 3), - 'int_pos_rotation_speed': 50.0, + 'int_pos_rotation_speed': None, 'int_pos_map_coverage': round(map_coverage, 3), - 'int_pos_lurk_tendency': 0.25, - 'int_pos_site_anchor_score': 50.0, + 'int_pos_lurk_tendency': None, + 'int_pos_site_anchor_score': None, 'int_pos_entry_route_diversity': round(position_diversity, 3), - 'int_pos_retake_positioning': 50.0, - 'int_pos_postplant_positioning': 50.0, + 'int_pos_retake_positioning': None, + 'int_pos_postplant_positioning': None, 'int_pos_spatial_iq_score': round(position_diversity * 100, 2), - 'int_pos_avg_distance_from_teammates': 500.0, + 'int_pos_avg_distance_from_teammates': None, } @staticmethod @@ -706,20 +694,20 @@ def _get_default_intelligence_features() -> Dict[str, Any]: 'int_pressure_big_moment_score': 0.0, 'int_pressure_tilt_resistance': 0.0, # Position Mastery (14) - 'int_pos_site_a_control_rate': 0.0, - 'int_pos_site_b_control_rate': 0.0, - 'int_pos_mid_control_rate': 0.0, - 'int_pos_favorite_position': 'unknown', - 'int_pos_position_diversity': 0.0, - 'int_pos_rotation_speed': 0.0, - 'int_pos_map_coverage': 0.0, - 'int_pos_lurk_tendency': 0.0, - 'int_pos_site_anchor_score': 0.0, - 'int_pos_entry_route_diversity': 0.0, - 'int_pos_retake_positioning': 0.0, - 'int_pos_postplant_positioning': 0.0, - 'int_pos_spatial_iq_score': 0.0, - 'int_pos_avg_distance_from_teammates': 0.0, + 'int_pos_site_a_control_rate': None, + 'int_pos_site_b_control_rate': None, + 'int_pos_mid_control_rate': None, + 'int_pos_favorite_position': None, + 'int_pos_position_diversity': None, + 'int_pos_rotation_speed': None, + 'int_pos_map_coverage': None, + 'int_pos_lurk_tendency': None, + 'int_pos_site_anchor_score': None, + 'int_pos_entry_route_diversity': None, + 'int_pos_retake_positioning': None, + 'int_pos_postplant_positioning': None, + 'int_pos_spatial_iq_score': None, + 'int_pos_avg_distance_from_teammates': None, # Trade Network (8) 'int_trade_kill_count': 0, 'int_trade_kill_rate': 0.0, diff --git a/database/L3/processors/meta_processor.py b/database/L3/processors/meta_processor.py index a219409..99ee116 100644 --- a/database/L3/processors/meta_processor.py +++ b/database/L3/processors/meta_processor.py @@ -60,10 +60,11 @@ class MetaProcessor(BaseFeatureProcessor): # Get recent matches for volatility cursor.execute(""" - SELECT rating - FROM fact_match_players - WHERE steam_id_64 = ? - ORDER BY match_id DESC + SELECT p.rating + FROM fact_match_players p + JOIN fact_matches m ON m.match_id = p.match_id + WHERE p.steam_id_64 = ? + ORDER BY m.start_time DESC, p.match_id DESC LIMIT 20 """, (steam_id,)) @@ -141,8 +142,35 @@ class MetaProcessor(BaseFeatureProcessor): map_ratings = [row[1] for row in cursor.fetchall() if row[1] is not None] map_stability = SafeAggregator.safe_stddev(map_ratings, 0.0) - # ELO tier stability (placeholder) - elo_tier_stability = rating_volatility # Simplified + cursor.execute(""" + SELECT + CASE + WHEN p.origin_elo - opponent.avg_elo > 200 THEN 'lower' + WHEN p.origin_elo - opponent.avg_elo < -200 THEN 'higher' + ELSE 'similar' + END AS opponent_tier, + AVG(p.rating) AS avg_rating + FROM fact_match_players p + JOIN ( + SELECT match_id, team_id, AVG(origin_elo) AS avg_elo + FROM fact_match_players + WHERE origin_elo IS NOT NULL + GROUP BY match_id, team_id + ) opponent + ON opponent.match_id = p.match_id + AND opponent.team_id != p.team_id + WHERE p.steam_id_64 = ? + AND p.origin_elo IS NOT NULL + AND p.rating IS NOT NULL + GROUP BY opponent_tier + """, (steam_id,)) + elo_tier_ratings = [ + row[1] for row in cursor.fetchall() if row[1] is not None + ] + elo_tier_stability = SafeAggregator.safe_stddev( + elo_tier_ratings, + 0.0, + ) return { 'meta_rating_volatility': round(rating_volatility, 3), diff --git a/database/L3/schema.sql b/database/L3/schema.sql index 97d8d58..6c3c667 100644 --- a/database/L3/schema.sql +++ b/database/L3/schema.sql @@ -378,6 +378,56 @@ CREATE TABLE IF NOT EXISTS dm_player_weapon_stats ( CREATE INDEX IF NOT EXISTS idx_player_weapon_stats_player ON dm_player_weapon_stats(steam_id_64); CREATE INDEX IF NOT EXISTS idx_player_weapon_stats_weapon ON dm_player_weapon_stats(weapon_name); +-- ============================================================================ +-- Profile Mart: Time-window statistics +-- ============================================================================ +CREATE TABLE IF NOT EXISTS dm_player_period_stats ( + steam_id_64 TEXT NOT NULL, + period_key TEXT NOT NULL, + period_label TEXT NOT NULL, + period_start INTEGER, + period_end INTEGER, + matches INTEGER NOT NULL DEFAULT 0, + wins INTEGER NOT NULL DEFAULT 0, + win_rate REAL, + avg_rating REAL, + avg_kd REAL, + avg_adr REAL, + avg_kast REAL, + total_kills INTEGER NOT NULL DEFAULT 0, + total_deaths INTEGER NOT NULL DEFAULT 0, + sample_reliable BOOLEAN NOT NULL DEFAULT 0, + last_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + + PRIMARY KEY (steam_id_64, period_key), + FOREIGN KEY (steam_id_64) + REFERENCES dm_player_features(steam_id_64) ON DELETE CASCADE +); + +CREATE INDEX IF NOT EXISTS idx_player_period_player +ON dm_player_period_stats(steam_id_64, period_key); + +-- ============================================================================ +-- Profile Mart: Career records linked to the source match +-- ============================================================================ +CREATE TABLE IF NOT EXISTS dm_player_records ( + steam_id_64 TEXT NOT NULL, + record_key TEXT NOT NULL, + record_label TEXT NOT NULL, + record_value REAL, + match_id TEXT, + map_name TEXT, + match_date INTEGER, + last_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + + PRIMARY KEY (steam_id_64, record_key), + FOREIGN KEY (steam_id_64) + REFERENCES dm_player_features(steam_id_64) ON DELETE CASCADE +); + +CREATE INDEX IF NOT EXISTS idx_player_records_player +ON dm_player_records(steam_id_64, record_key); + -- ============================================================================ -- Schema Summary -- ============================================================================ @@ -391,4 +441,6 @@ CREATE INDEX IF NOT EXISTS idx_player_weapon_stats_weapon ON dm_player_weapon_st -- dm_player_match_history: Per-match snapshots for trend analysis -- dm_player_map_stats: Map-level aggregations -- dm_player_weapon_stats: Weapon usage statistics +-- dm_player_period_stats: Career/recent time-window aggregations +-- dm_player_records: Career record values and source matches -- ============================================================================ diff --git a/database/README.md b/database/README.md new file mode 100644 index 0000000..1a7024d --- /dev/null +++ b/database/README.md @@ -0,0 +1,45 @@ +# Database Governance + +The repository intentionally keeps SQLite for the current private-team scale. +This directory separates four different responsibilities: + +| Layer | Database | Grain | Owner | +|---|---|---|---| +| L1 | `L1/L1.db` | One raw network capture per match | Import pipeline | +| L2 | `L2/L2.db` | Normalized match, player, round and event facts | L2 Builder | +| L3 | `L3/L3.db` | Roster features and profile marts | L3 Builder | +| Web | `Web/Web_App.sqlite` | Lineups, comments, jobs and editorial data | Flask app | + +## Rules + +1. Paths are defined only in `database/paths.py`. +2. Schemas live next to their owning database. +3. Builders may read the previous layer and write only their own layer. +4. User-generated Web data is never restored as part of an ETL rollback. +5. A full import must run through `database/pipeline.py`. +6. Pipeline runs are serialized by `database/.pipeline.lock`. +7. L1/L2/L3 are backed up before a full pipeline run. +8. Missing metrics are stored as `NULL`, not fabricated zero values. +9. `Admin -> Data Integrity` is the operational source of truth. +10. Web schema changes increment `Config.WEB_SCHEMA_VERSION`. + +## Entry Points + +```bash +make l1 # Import output_arena JSON into L1 +make l2 # Rebuild normalized facts +make l3 # Rebuild active-roster features +make pipeline # Run L1 -> L2 -> L3 with backup and validation +make check # Compile and run tests +``` + +## Directory Policy + +- `L1/`, `L2/`, `L3/`, `Web/`: active code, schema and database. +- `backups/`: generated rollback snapshots; ignored by Git. +- `schema_bkp/`: historical schema research only; not used at runtime. +- `L1B/`: reserved demo-parser integration; not used at runtime. +- `L3/Roadmap/`: historical design notes; not used at runtime. + +Large-scale directory moves are deliberately deferred until the legacy +builders no longer depend on their current module layout. diff --git a/database/Web/Web_App.sqlite b/database/Web/Web_App.sqlite index 5695a31..da55218 100644 Binary files a/database/Web/Web_App.sqlite and b/database/Web/Web_App.sqlite differ diff --git a/database/Web/schema.sql b/database/Web/schema.sql new file mode 100644 index 0000000..e2c1761 --- /dev/null +++ b/database/Web/schema.sql @@ -0,0 +1,95 @@ +CREATE TABLE IF NOT EXISTS schema_migrations ( + version INTEGER PRIMARY KEY, + description TEXT NOT NULL, + applied_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE IF NOT EXISTS comments ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + user_id TEXT, + username TEXT, + target_type TEXT NOT NULL, + target_id TEXT NOT NULL, + content TEXT NOT NULL, + likes INTEGER NOT NULL DEFAULT 0, + is_hidden INTEGER NOT NULL DEFAULT 0, + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX IF NOT EXISTS idx_comments_target +ON comments(target_type, target_id, is_hidden, created_at DESC); + +CREATE TABLE IF NOT EXISTS player_metadata ( + steam_id_64 TEXT PRIMARY KEY, + notes TEXT, + tags TEXT NOT NULL DEFAULT '[]', + updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE IF NOT EXISTS strategy_boards ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + title TEXT NOT NULL, + map_name TEXT NOT NULL, + data_json TEXT NOT NULL, + created_by TEXT, + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE IF NOT EXISTS team_lineups ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL, + description TEXT, + player_ids_json TEXT NOT NULL DEFAULT '[]', + is_active INTEGER NOT NULL DEFAULT 0, + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE UNIQUE INDEX IF NOT EXISTS idx_team_lineups_single_active +ON team_lineups(is_active) +WHERE is_active = 1; + +CREATE TABLE IF NOT EXISTS wiki_pages ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + path TEXT NOT NULL UNIQUE, + title TEXT NOT NULL, + content TEXT NOT NULL DEFAULT '', + updated_by TEXT, + updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE IF NOT EXISTS etl_jobs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + job_type TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'queued' + CHECK (status IN ('queued', 'running', 'succeeded', 'failed')), + match_id TEXT, + input_path TEXT, + current_stage TEXT, + progress INTEGER NOT NULL DEFAULT 0 + CHECK (progress >= 0 AND progress <= 100), + message TEXT, + log_text TEXT NOT NULL DEFAULT '', + created_by TEXT, + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + started_at TIMESTAMP, + finished_at TIMESTAMP, + duration_seconds REAL +); + +CREATE INDEX IF NOT EXISTS idx_etl_jobs_created +ON etl_jobs(created_at DESC, id DESC); + +CREATE INDEX IF NOT EXISTS idx_etl_jobs_status +ON etl_jobs(status, created_at); + +CREATE TABLE IF NOT EXISTS match_imports ( + match_id TEXT PRIMARY KEY, + content_sha256 TEXT NOT NULL, + source_path TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'queued', + job_id INTEGER, + imported_at TIMESTAMP, + updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + FOREIGN KEY (job_id) REFERENCES etl_jobs(id) ON DELETE SET NULL +); diff --git a/database/__init__.py b/database/__init__.py new file mode 100644 index 0000000..40fb874 --- /dev/null +++ b/database/__init__.py @@ -0,0 +1,2 @@ +"""Database builders, schemas, maintenance tools and local data stores.""" + diff --git a/database/job_store.py b/database/job_store.py new file mode 100644 index 0000000..4203da4 --- /dev/null +++ b/database/job_store.py @@ -0,0 +1,198 @@ +import sqlite3 +from typing import Any, Dict, List, Optional + +from database.paths import WEB_DB + + +class JobStore: + def __init__(self, database_path=WEB_DB): + self.database_path = str(database_path) + + def _connect(self): + db = sqlite3.connect(self.database_path, timeout=30) + db.row_factory = sqlite3.Row + db.execute('PRAGMA busy_timeout = 30000') + db.execute('PRAGMA foreign_keys = ON') + return db + + def create_job( + self, + job_type: str, + match_id: Optional[str] = None, + input_path: Optional[str] = None, + created_by: Optional[str] = None, + ) -> int: + db = self._connect() + try: + cursor = db.execute( + """ + INSERT INTO etl_jobs ( + job_type, match_id, input_path, created_by + ) VALUES (?, ?, ?, ?) + """, + [job_type, match_id, input_path, created_by], + ) + db.commit() + return int(cursor.lastrowid) + finally: + db.close() + + def get_job(self, job_id: int) -> Optional[Dict[str, Any]]: + db = self._connect() + try: + row = db.execute( + 'SELECT * FROM etl_jobs WHERE id = ?', + [job_id], + ).fetchone() + return dict(row) if row else None + finally: + db.close() + + def list_jobs(self, limit: int = 20) -> List[Dict[str, Any]]: + db = self._connect() + try: + rows = db.execute( + """ + SELECT * + FROM etl_jobs + ORDER BY created_at DESC, id DESC + LIMIT ? + """, + [max(1, min(int(limit), 100))], + ).fetchall() + return [dict(row) for row in rows] + finally: + db.close() + + def start_job(self, job_id: int, stage: str, message: str): + db = self._connect() + try: + db.execute( + """ + UPDATE etl_jobs + SET status = 'running', + current_stage = ?, + progress = 1, + message = ?, + started_at = CURRENT_TIMESTAMP + WHERE id = ? AND status = 'queued' + """, + [stage, message, job_id], + ) + db.commit() + finally: + db.close() + + def update_progress( + self, + job_id: int, + stage: str, + progress: int, + message: str, + ): + db = self._connect() + try: + db.execute( + """ + UPDATE etl_jobs + SET current_stage = ?, progress = ?, message = ? + WHERE id = ? + """, + [stage, max(0, min(int(progress), 100)), message, job_id], + ) + db.commit() + finally: + db.close() + + def append_log(self, job_id: int, text: str): + if not text: + return + db = self._connect() + try: + db.execute( + """ + UPDATE etl_jobs + SET log_text = substr(log_text || ?, -100000) + WHERE id = ? + """, + [text, job_id], + ) + db.commit() + finally: + db.close() + + def finish_job( + self, + job_id: int, + succeeded: bool, + message: str, + duration_seconds: float, + ): + status = 'succeeded' if succeeded else 'failed' + db = self._connect() + try: + db.execute( + """ + UPDATE etl_jobs + SET status = ?, + current_stage = ?, + progress = ?, + message = ?, + finished_at = CURRENT_TIMESTAMP, + duration_seconds = ? + WHERE id = ? + """, + [ + status, + 'complete' if succeeded else 'failed', + 100 if succeeded else 0, + message, + round(float(duration_seconds), 3), + job_id, + ], + ) + db.execute( + """ + UPDATE match_imports + SET status = ?, + imported_at = CASE + WHEN ? = 'succeeded' THEN CURRENT_TIMESTAMP + ELSE imported_at + END, + updated_at = CURRENT_TIMESTAMP + WHERE job_id = ? + """, + [status, status, job_id], + ) + db.commit() + finally: + db.close() + + def upsert_match_import( + self, + match_id: str, + content_sha256: str, + source_path: str, + status: str, + job_id: int, + ): + db = self._connect() + try: + db.execute( + """ + INSERT INTO match_imports ( + match_id, content_sha256, source_path, status, job_id + ) VALUES (?, ?, ?, ?, ?) + ON CONFLICT(match_id) DO UPDATE SET + content_sha256 = excluded.content_sha256, + source_path = excluded.source_path, + status = excluded.status, + job_id = excluded.job_id, + updated_at = CURRENT_TIMESTAMP + """, + [match_id, content_sha256, source_path, status, job_id], + ) + db.commit() + finally: + db.close() + diff --git a/database/maintenance.py b/database/maintenance.py new file mode 100644 index 0000000..787f7a2 --- /dev/null +++ b/database/maintenance.py @@ -0,0 +1,140 @@ +from datetime import datetime, timezone +import json +from pathlib import Path +import shutil +import sqlite3 +from typing import Dict + +from database.paths import BACKUP_ROOT, L1_DB, L2_DB, L3_DB + + +MANAGED_DATABASES = { + 'l1': L1_DB, + 'l2': L2_DB, + 'l3': L3_DB, +} + + +def quick_check(path: Path) -> str: + if not path.exists(): + return 'missing' + db = sqlite3.connect(str(path)) + try: + return str(db.execute('PRAGMA quick_check').fetchone()[0]) + finally: + db.close() + + +def backup_database(source_path: Path, backup_path: Path): + if not source_path.exists(): + raise FileNotFoundError(f'Database does not exist: {source_path}') + + backup_path.parent.mkdir(parents=True, exist_ok=True) + source = sqlite3.connect(str(source_path)) + destination = sqlite3.connect(str(backup_path)) + try: + source.backup(destination) + result = destination.execute('PRAGMA quick_check').fetchone()[0] + if result != 'ok': + raise RuntimeError( + f'Backup quick_check failed for {source_path.name}: {result}' + ) + finally: + source.close() + destination.close() + + +def restore_database(backup_path: Path, target_path: Path): + if not backup_path.exists(): + raise FileNotFoundError(f'Backup does not exist: {backup_path}') + + source = sqlite3.connect(str(backup_path)) + target = sqlite3.connect(str(target_path), timeout=30) + try: + source.backup(target) + result = target.execute('PRAGMA quick_check').fetchone()[0] + if result != 'ok': + raise RuntimeError( + f'Restored quick_check failed for {target_path.name}: {result}' + ) + finally: + source.close() + target.close() + + +def create_backup_set(label: str) -> Path: + safe_label = ''.join( + character for character in str(label) + if character.isalnum() or character in {'-', '_'} + ) + if not safe_label: + raise ValueError('Backup label is empty after sanitization') + + backup_dir = BACKUP_ROOT / safe_label + backup_dir.mkdir(parents=True, exist_ok=True) + manifest: Dict[str, object] = { + 'label': safe_label, + 'created_at': datetime.now(timezone.utc).isoformat(), + 'databases': {}, + } + + for name, source_path in MANAGED_DATABASES.items(): + backup_path = backup_dir / source_path.name + backup_database(source_path, backup_path) + manifest['databases'][name] = { + 'source': str(source_path), + 'backup': str(backup_path), + 'size_bytes': backup_path.stat().st_size, + 'quick_check': quick_check(backup_path), + } + + with (backup_dir / 'manifest.json').open('w', encoding='utf-8') as file: + json.dump(manifest, file, ensure_ascii=True, indent=2) + return backup_dir + + +def restore_backup_set(backup_dir: Path): + for name, target_path in MANAGED_DATABASES.items(): + backup_path = backup_dir / target_path.name + restore_database(backup_path, target_path) + + +def prune_backup_sets(keep: int = 3): + BACKUP_ROOT.mkdir(parents=True, exist_ok=True) + backup_dirs = sorted( + [path for path in BACKUP_ROOT.iterdir() if path.is_dir()], + key=lambda path: path.stat().st_mtime, + reverse=True, + ) + removed = [] + for path in backup_dirs[max(int(keep), 0):]: + shutil.rmtree(path) + removed.append(str(path)) + return removed + + +def backup_storage_status(): + BACKUP_ROOT.mkdir(parents=True, exist_ok=True) + backup_dirs = [path for path in BACKUP_ROOT.iterdir() if path.is_dir()] + total_bytes = sum( + file.stat().st_size + for directory in backup_dirs + for file in directory.rglob('*') + if file.is_file() + ) + return { + 'sets': len(backup_dirs), + 'total_bytes': total_bytes, + } + + +def check_managed_databases(): + return { + name: { + 'path': str(path), + 'exists': path.exists(), + 'size_bytes': path.stat().st_size if path.exists() else 0, + 'quick_check': quick_check(path), + } + for name, path in MANAGED_DATABASES.items() + } diff --git a/database/paths.py b/database/paths.py new file mode 100644 index 0000000..45188d6 --- /dev/null +++ b/database/paths.py @@ -0,0 +1,36 @@ +from pathlib import Path + + +PROJECT_ROOT = Path(__file__).resolve().parent.parent +DATABASE_ROOT = PROJECT_ROOT / 'database' + +L1_DIR = DATABASE_ROOT / 'L1' +L2_DIR = DATABASE_ROOT / 'L2' +L3_DIR = DATABASE_ROOT / 'L3' +WEB_DIR = DATABASE_ROOT / 'Web' + +L1_DB = L1_DIR / 'L1.db' +L2_DB = L2_DIR / 'L2.db' +L3_DB = L3_DIR / 'L3.db' +WEB_DB = WEB_DIR / 'Web_App.sqlite' + +L2_SCHEMA = L2_DIR / 'schema.sql' +L3_SCHEMA = L3_DIR / 'schema.sql' +WEB_SCHEMA = WEB_DIR / 'schema.sql' + +OUTPUT_ARENA = PROJECT_ROOT / 'output_arena' +BACKUP_ROOT = DATABASE_ROOT / 'backups' +PIPELINE_LOCK = DATABASE_ROOT / '.pipeline.lock' + + +def ensure_runtime_directories(): + for path in ( + L1_DIR, + L2_DIR, + L3_DIR, + WEB_DIR, + OUTPUT_ARENA, + BACKUP_ROOT, + ): + path.mkdir(parents=True, exist_ok=True) + diff --git a/database/pipeline.py b/database/pipeline.py new file mode 100644 index 0000000..01f69e0 --- /dev/null +++ b/database/pipeline.py @@ -0,0 +1,212 @@ +import argparse +import fcntl +import os +import sqlite3 +import subprocess +import sys +import time +from pathlib import Path + +PROJECT_ROOT = Path(__file__).resolve().parent.parent +if str(PROJECT_ROOT) not in sys.path: + sys.path.insert(0, str(PROJECT_ROOT)) + +from database.job_store import JobStore +from database.maintenance import ( + check_managed_databases, + create_backup_set, + prune_backup_sets, + restore_backup_set, +) +from database.paths import L1_DB, L2_DB, PIPELINE_LOCK + + +STAGES = ( + ('l1', 15, Path('database/L1/L1_Builder.py')), + ('l2', 55, Path('database/L2/L2_Builder.py')), + ('l3', 85, Path('database/L3/L3_Builder.py')), +) + + +class PipelineError(RuntimeError): + pass + + +def _run_stage(store, job_id, stage, progress, script_path, replace=False): + store.update_progress( + job_id, + stage, + progress, + f'Running {stage.upper()} builder', + ) + command = [sys.executable, str(PROJECT_ROOT / script_path)] + if stage == 'l1' and replace: + command.append('--force') + if stage == 'l3': + command.append('--no-backup') + + started = time.monotonic() + result = subprocess.run( + command, + cwd=str(PROJECT_ROOT), + capture_output=True, + text=True, + timeout=1200, + ) + duration = time.monotonic() - started + store.append_log( + job_id, + ( + f'\n===== {stage.upper()} ({duration:.2f}s) =====\n' + f'{result.stdout}\n{result.stderr}' + ), + ) + if result.returncode != 0: + raise PipelineError( + f'{stage.upper()} builder exited with code {result.returncode}' + ) + + +def _validate_pipeline_output(match_id=None): + database_status = check_managed_databases() + failures = [ + f"{name}: {status['quick_check']}" + for name, status in database_status.items() + if status['quick_check'] != 'ok' + ] + if failures: + raise PipelineError( + 'Database quick_check failed: ' + ', '.join(failures) + ) + + if not match_id: + return + + l1 = sqlite3.connect(str(L1_DB)) + l2 = sqlite3.connect(str(L2_DB)) + try: + raw_count = l1.execute( + 'SELECT COUNT(*) FROM raw_iframe_network WHERE match_id = ?', + [match_id], + ).fetchone()[0] + match_count = l2.execute( + 'SELECT COUNT(*) FROM fact_matches WHERE match_id = ?', + [match_id], + ).fetchone()[0] + player_count = l2.execute( + 'SELECT COUNT(*) FROM fact_match_players WHERE match_id = ?', + [match_id], + ).fetchone()[0] + round_count = l2.execute( + 'SELECT COUNT(*) FROM fact_rounds WHERE match_id = ?', + [match_id], + ).fetchone()[0] + finally: + l1.close() + l2.close() + + if raw_count != 1: + raise PipelineError(f'L1 does not contain imported match {match_id}') + if match_count != 1: + raise PipelineError(f'L2 does not contain imported match {match_id}') + if player_count != 10: + raise PipelineError( + f'Imported match has {player_count} players; expected 10' + ) + if round_count <= 0: + raise PipelineError('Imported match has no round facts') + + +def run_pipeline(job_id, match_id=None, replace=False): + store = JobStore() + job = store.get_job(job_id) + if not job: + raise PipelineError(f'Unknown ETL job: {job_id}') + + started = time.monotonic() + backup_dir = None + PIPELINE_LOCK.parent.mkdir(parents=True, exist_ok=True) + lock_file = PIPELINE_LOCK.open('w') + try: + try: + fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError as exc: + raise PipelineError('Another database pipeline is already running') from exc + + store.start_job(job_id, 'backup', 'Creating rollback snapshot') + backup_dir = create_backup_set(f'job-{job_id}') + store.append_log(job_id, f'Backup created: {backup_dir}\n') + + for stage, progress, script_path in STAGES: + _run_stage( + store, + job_id, + stage, + progress, + script_path, + replace=replace, + ) + + store.update_progress( + job_id, + 'validation', + 95, + 'Validating imported data', + ) + _validate_pipeline_output(match_id) + removed_backups = prune_backup_sets(keep=3) + if removed_backups: + store.append_log( + job_id, + f"Pruned old backups: {', '.join(removed_backups)}\n", + ) + duration = time.monotonic() - started + store.finish_job( + job_id, + True, + 'Pipeline completed and validated', + duration, + ) + return True + except Exception as exc: + store.append_log(job_id, f'\nPIPELINE FAILED: {exc}\n') + if backup_dir: + try: + restore_backup_set(backup_dir) + store.append_log(job_id, 'Rollback snapshot restored successfully\n') + except Exception as restore_exc: + store.append_log( + job_id, + f'ROLLBACK FAILED: {restore_exc}\n', + ) + exc = PipelineError(f'{exc}; rollback also failed: {restore_exc}') + store.finish_job( + job_id, + False, + str(exc), + time.monotonic() - started, + ) + return False + finally: + try: + fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN) + finally: + lock_file.close() + + +def _parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('--job-id', type=int, required=True) + parser.add_argument('--match-id') + parser.add_argument('--replace', action='store_true') + return parser.parse_args() + + +if __name__ == '__main__': + args = _parse_args() + succeeded = run_pipeline( + args.job_id, + match_id=args.match_id, + replace=args.replace, + ) + raise SystemExit(0 if succeeded else 1) diff --git a/requirements.txt b/requirements.txt index 0efa20b..0d3831e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,3 @@ -Flask -pandas -numpy -playwright -gunicorn -gevent -matplotlib +Flask>=3.0,<4 +gunicorn>=21,<24 +pytest>=8,<9 diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/tests/__init__.py @@ -0,0 +1 @@ + diff --git a/tests/test_integration.py b/tests/test_integration.py new file mode 100644 index 0000000..8aee864 --- /dev/null +++ b/tests/test_integration.py @@ -0,0 +1,620 @@ +import json +import io +import os +import shutil +import sqlite3 +import tempfile +import unittest +from pathlib import Path + +from web.config import Config + + +class ApplicationIntegrationTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.temp_dir = tempfile.TemporaryDirectory() + cls.original_web_path = Config.DB_WEB_PATH + Config.DB_WEB_PATH = os.path.join(cls.temp_dir.name, 'Web_App.sqlite') + shutil.copy2(cls.original_web_path, Config.DB_WEB_PATH) + + from web.app import create_app + + cls.app = create_app() + cls.app.config.update(TESTING=True) + cls.client = cls.app.test_client() + + with sqlite3.connect(Config.DB_WEB_PATH) as db: + raw_ids = db.execute( + """ + SELECT player_ids_json + FROM team_lineups + WHERE is_active = 1 + LIMIT 1 + """ + ).fetchone()[0] + cls.roster_ids = [str(value) for value in json.loads(raw_ids)] + + @classmethod + def tearDownClass(cls): + Config.DB_WEB_PATH = cls.original_web_path + cls.temp_dir.cleanup() + + def test_primary_pages_render(self): + paths = [ + '/', + '/matches/', + '/players/', + f'/players/{self.roster_ids[0]}', + '/teams/', + '/tactics/', + '/opponents/', + ] + for path in paths: + with self.subTest(path=path): + response = self.client.get(path) + self.assertEqual(response.status_code, 200) + self.assertGreater(len(response.data), 100) + + def test_admin_integrity_page_and_json_render(self): + with self.client.session_transaction() as session: + session['is_admin'] = True + + response = self.client.get('/admin/data-integrity') + self.assertEqual(response.status_code, 200) + self.assertIn('数据完整性中心'.encode('utf-8'), response.data) + + response = self.client.get('/admin/data-integrity?format=json') + self.assertEqual(response.status_code, 200) + report = response.get_json() + self.assertIn(report['overall_status'], {'pass', 'warn', 'fail'}) + self.assertGreater(report['counts']['matches'], 0) + self.assertEqual( + report['counts']['web_schema_version'], + Config.WEB_SCHEMA_VERSION, + ) + + response = self.client.get('/admin/import-match') + self.assertEqual(response.status_code, 200) + self.assertIn('比赛数据导入'.encode('utf-8'), response.data) + + def test_duplicate_match_upload_is_rejected_without_starting_job(self): + from database.paths import L1_DB + + with self.client.session_transaction() as session: + session['is_admin'] = True + with sqlite3.connect(str(L1_DB)) as db: + raw = db.execute( + """ + SELECT content + FROM raw_iframe_network + ORDER BY match_id + LIMIT 1 + """ + ).fetchone()[0] + + response = self.client.post( + '/admin/import-match', + data={ + 'capture': ( + io.BytesIO(raw.encode('utf-8')), + 'iframe_network.json', + ), + }, + content_type='multipart/form-data', + follow_redirects=True, + ) + self.assertEqual(response.status_code, 200) + self.assertIn(b'already imported with identical data', response.data) + + def test_player_search_works_across_l2_and_l3(self): + response = self.client.get('/players/?search=jAck') + self.assertEqual(response.status_code, 200) + self.assertIn(b'jAckY0987', response.data) + + def test_profile_keeps_all_primary_sections(self): + response = self.client.get(f'/players/{self.roster_ids[0]}') + self.assertEqual(response.status_code, 200) + for label in ( + '近期表现走势', + '能力八维图', + 'CORE (核心表现)', + '阶段表现', + '职业纪录', + '比赛记录', + '地图数据', + '留言板', + ): + with self.subTest(label=label): + self.assertIn(label.encode('utf-8'), response.data) + + def test_profile_period_api_and_trend_window(self): + steam_id = self.roster_ids[0] + response = self.client.get( + f'/players/{steam_id}/period_stats?period=last_20' + ) + self.assertEqual(response.status_code, 200) + period = response.get_json() + self.assertEqual(period['period_key'], 'last_20') + self.assertEqual(period['matches'], 20) + self.assertEqual(period['sample_reliable'], 1) + + response = self.client.get( + f'/players/{steam_id}/charts_data?period=last_10' + ) + self.assertEqual(response.status_code, 200) + chart = response.get_json() + self.assertEqual(chart['period']['period_key'], 'last_10') + self.assertLessEqual(len(chart['trend']['labels']), 10) + + def test_date_filter_uses_unix_timestamp_conversion(self): + from web.services.stats_service import StatsService + + with self.app.app_context(): + matches, total = StatsService.get_matches( + page=1, + per_page=20, + date_from='2025-01-01', + date_to='2026-12-31', + ) + self.assertGreater(total, 0) + self.assertGreater(len(matches), 0) + + def test_shared_matches_require_same_team(self): + from web.services.stats_service import StatsService + + selected_ids = self.roster_ids[:2] + with self.app.app_context(): + matches = StatsService.get_shared_matches(selected_ids) + self.assertGreater(len(matches), 0) + + l2 = sqlite3.connect(Config.DB_L2_PATH) + try: + for match in matches: + placeholders = ','.join('?' for _ in selected_ids) + team_count = l2.execute( + f""" + SELECT COUNT(DISTINCT team_id) + FROM fact_match_players + WHERE match_id = ? + AND steam_id_64 IN ({placeholders}) + """, + [match['match_id']] + selected_ids, + ).fetchone()[0] + self.assertEqual(team_count, 1) + finally: + l2.close() + + def test_opponent_list_only_contains_actual_opponents(self): + from web.services.opponent_service import OpponentService + + with self.app.app_context(): + opponents, total = OpponentService.get_opponent_list( + page=1, + per_page=20, + ) + self.assertGreater(total, 0) + self.assertGreater(len(opponents), 0) + + l2 = sqlite3.connect(Config.DB_L2_PATH) + try: + roster_ph = ','.join('?' for _ in self.roster_ids) + for opponent in opponents: + faced = l2.execute( + f""" + SELECT COUNT(*) + FROM fact_match_players opponent + JOIN fact_match_players roster + ON roster.match_id = opponent.match_id + AND roster.team_id != opponent.team_id + WHERE opponent.steam_id_64 = ? + AND roster.steam_id_64 IN ({roster_ph}) + """, + [opponent['steam_id_64']] + self.roster_ids, + ).fetchone()[0] + self.assertGreater(faced, 0) + finally: + l2.close() + + +class L3MartBuilderTests(unittest.TestCase): + def test_auxiliary_marts_build_on_database_copy(self): + from database.L3.L3_Builder import ( + _get_team_players, + _rebuild_auxiliary_marts, + ) + + with tempfile.TemporaryDirectory() as temp_dir: + l3_path = os.path.join(temp_dir, 'L3.db') + shutil.copy2(Config.DB_L3_PATH, l3_path) + l2 = sqlite3.connect(Config.DB_L2_PATH) + l2.row_factory = sqlite3.Row + l3 = sqlite3.connect(l3_path) + try: + _rebuild_auxiliary_marts( + l2, + l3, + sorted(_get_team_players()), + ) + l3.commit() + for table in ( + 'dm_player_match_history', + 'dm_player_map_stats', + 'dm_player_period_stats', + 'dm_player_records', + 'dm_player_weapon_stats', + ): + count = l3.execute( + f'SELECT COUNT(*) FROM {table}' + ).fetchone()[0] + self.assertGreater(count, 0) + self.assertEqual( + l3.execute('PRAGMA quick_check').fetchone()[0], + 'ok', + ) + finally: + l2.close() + l3.close() + + def test_spatial_processor_does_not_emit_fake_geometry_metrics(self): + from database.L3.L3_Builder import _get_team_players + from database.L3.processors.intelligence_processor import IntelligenceProcessor + + roster_ids = sorted(_get_team_players()) + placeholders = ','.join('?' for _ in roster_ids) + l2 = sqlite3.connect(Config.DB_L2_PATH) + try: + row = l2.execute( + f""" + SELECT attacker_steam_id + FROM fact_round_events + WHERE attacker_steam_id IN ({placeholders}) + AND attacker_pos_x IS NOT NULL + GROUP BY attacker_steam_id + ORDER BY COUNT(*) DESC + LIMIT 1 + """, + roster_ids, + ).fetchone() + self.assertIsNotNone(row) + features = IntelligenceProcessor._calculate_position_mastery( + str(row[0]), + l2, + ) + finally: + l2.close() + + for key in ( + 'int_pos_site_a_control_rate', + 'int_pos_site_b_control_rate', + 'int_pos_mid_control_rate', + 'int_pos_rotation_speed', + 'int_pos_lurk_tendency', + 'int_pos_site_anchor_score', + 'int_pos_retake_positioning', + 'int_pos_postplant_positioning', + 'int_pos_avg_distance_from_teammates', + ): + with self.subTest(key=key): + self.assertIsNone(features[key]) + self.assertIsNotNone(features['int_pos_position_diversity']) + + def test_percentiles_are_calculated_from_peer_scores(self): + from database.L3.L3_Builder import _update_percentiles + + with tempfile.TemporaryDirectory() as temp_dir: + l3_path = os.path.join(temp_dir, 'L3.db') + shutil.copy2(Config.DB_L3_PATH, l3_path) + l3 = sqlite3.connect(l3_path) + try: + player_ids = [ + row[0] for row in l3.execute( + """ + SELECT steam_id_64 + FROM dm_player_features + ORDER BY steam_id_64 + LIMIT 3 + """ + ) + ] + for steam_id, score in zip(player_ids, (10.0, 20.0, 30.0)): + l3.execute( + """ + UPDATE dm_player_features + SET score_overall = ? + WHERE steam_id_64 = ? + """, + [score, steam_id], + ) + _update_percentiles(l3, player_ids) + values = [ + row[0] for row in l3.execute( + f""" + SELECT tier_percentile + FROM dm_player_features + WHERE steam_id_64 IN ({','.join('?' for _ in player_ids)}) + ORDER BY score_overall + """, + player_ids, + ) + ] + self.assertEqual(values, [33.33, 66.67, 100.0]) + finally: + l3.close() + + def test_l3_backup_is_a_valid_sqlite_database(self): + from database.L3.L3_Builder import _backup_l3_database + + with tempfile.TemporaryDirectory() as temp_dir: + source = os.path.join(temp_dir, 'source.db') + backup = os.path.join(temp_dir, 'backup.db') + shutil.copy2(Config.DB_L3_PATH, source) + result_path = _backup_l3_database(source, backup) + self.assertEqual(result_path, backup) + with sqlite3.connect(backup) as db: + self.assertEqual(db.execute('PRAGMA quick_check').fetchone()[0], 'ok') + + +class FeatureFormulaTests(unittest.TestCase): + def test_empty_pace_inputs_do_not_receive_free_points(self): + from database.L3.processors.composite_processor import CompositeProcessor + + self.assertEqual(CompositeProcessor._calculate_pace_score({}), 0.0) + + def test_lower_map_and_elo_volatility_improves_stability_score(self): + from database.L3.processors.composite_processor import CompositeProcessor + + common = { + 'meta_rating_volatility': 0.2, + 'meta_loss_rating': 1.0, + 'meta_rating_consistency': 70, + 'int_pressure_tilt_resistance': 0.8, + 'meta_recent_form_rating': 1.15, + } + stable = dict(common, meta_map_stability=0.05, meta_elo_tier_stability=0.05) + volatile = dict(common, meta_map_stability=0.25, meta_elo_tier_stability=0.48) + self.assertGreater( + CompositeProcessor._calculate_stability_score(stable), + CompositeProcessor._calculate_stability_score(volatile), + ) + + def test_recent_form_uses_match_time_and_elo_stability_is_calculated(self): + from database.L3.L3_Builder import _get_team_players + from database.L3.processors.meta_processor import MetaProcessor + + steam_id = sorted(_get_team_players())[0] + l2 = sqlite3.connect(Config.DB_L2_PATH) + try: + expected_rows = l2.execute( + """ + SELECT p.rating + FROM fact_match_players p + JOIN fact_matches m ON m.match_id = p.match_id + WHERE p.steam_id_64 = ? + ORDER BY m.start_time DESC, p.match_id DESC + LIMIT 10 + """, + [steam_id], + ).fetchall() + expected = sum(row[0] for row in expected_rows) / len(expected_rows) + features = MetaProcessor._calculate_stability(steam_id, l2) + finally: + l2.close() + + self.assertAlmostEqual( + features['meta_recent_form_rating'], + round(expected, 3), + places=3, + ) + self.assertGreaterEqual(features['meta_elo_tier_stability'], 0) + self.assertNotEqual( + features['meta_elo_tier_stability'], + features['meta_rating_volatility'], + ) + + +class DatabaseGovernanceTests(unittest.TestCase): + def test_database_paths_are_absolute_and_exist(self): + from database.paths import L1_DB, L2_DB, L3_DB, WEB_DB + + for path in (L1_DB, L2_DB, L3_DB, WEB_DB): + with self.subTest(path=path): + self.assertTrue(path.is_absolute()) + self.assertTrue(path.exists()) + + def test_valid_capture_is_identified_from_network_urls(self): + from database.paths import L1_DB + from web.services.import_service import MatchImportService + + with sqlite3.connect(str(L1_DB)) as db: + match_id, raw = db.execute( + """ + SELECT match_id, content + FROM raw_iframe_network + ORDER BY match_id + LIMIT 1 + """ + ).fetchone() + result = MatchImportService.validate_capture(raw.encode('utf-8')) + self.assertEqual(result['match_id'], match_id) + self.assertGreaterEqual(result['successful_responses'], 2) + + def test_prepare_import_is_atomic_and_rejects_duplicate_queue(self): + import web.services.import_service as import_module + from database.paths import L1_DB + from web.services.import_service import ( + DuplicateMatchError, + MatchImportService, + ) + + with sqlite3.connect(str(L1_DB)) as db: + original_id, raw = db.execute( + """ + SELECT match_id, content + FROM raw_iframe_network + ORDER BY match_id + LIMIT 1 + """ + ).fetchone() + new_id = 'g161-99999999999999999999999' + raw = raw.replace(original_id, new_id).encode('utf-8') + + with tempfile.TemporaryDirectory() as temp_dir: + old_l1 = import_module.L1_DB + old_arena = import_module.OUTPUT_ARENA + old_web = Config.DB_WEB_PATH + fake_l1 = Path(temp_dir) / 'L1.db' + fake_web = Path(temp_dir) / 'Web_App.sqlite' + with sqlite3.connect(str(fake_l1)) as db: + db.execute( + """ + CREATE TABLE raw_iframe_network ( + match_id TEXT PRIMARY KEY, + content TEXT + ) + """ + ) + with sqlite3.connect(str(fake_web)) as db: + with open(Config.DB_WEB_SCHEMA_PATH, 'r', encoding='utf-8') as schema: + db.executescript(schema.read()) + import_module.L1_DB = fake_l1 + import_module.OUTPUT_ARENA = Path(temp_dir) / 'output_arena' + Config.DB_WEB_PATH = str(fake_web) + try: + prepared = MatchImportService.prepare_import( + raw, + 'iframe_network.json', + created_by='test', + ) + self.assertEqual(prepared['match_id'], new_id) + self.assertTrue(Path(prepared['source_path']).exists()) + with self.assertRaises(DuplicateMatchError): + MatchImportService.prepare_import( + raw, + 'iframe_network.json', + created_by='test', + ) + finally: + import_module.L1_DB = old_l1 + import_module.OUTPUT_ARENA = old_arena + Config.DB_WEB_PATH = old_web + + def test_pipeline_post_validation_accepts_current_databases(self): + from database.paths import L1_DB + from database.pipeline import _validate_pipeline_output + + with sqlite3.connect(str(L1_DB)) as db: + match_id = db.execute( + 'SELECT match_id FROM raw_iframe_network ORDER BY match_id LIMIT 1' + ).fetchone()[0] + _validate_pipeline_output(match_id) + + def test_job_store_tracks_progress_logs_and_completion(self): + from database.job_store import JobStore + + with tempfile.TemporaryDirectory() as temp_dir: + web_path = Path(temp_dir) / 'Web.sqlite' + with sqlite3.connect(str(web_path)) as db: + with open(Config.DB_WEB_SCHEMA_PATH, 'r', encoding='utf-8') as schema: + db.executescript(schema.read()) + + store = JobStore(web_path) + job_id = store.create_job( + 'test_pipeline', + match_id='g161-99999999999999999999999', + created_by='test', + ) + store.start_job(job_id, 'backup', 'Starting') + store.update_progress(job_id, 'l2', 55, 'Building L2') + store.append_log(job_id, 'line one\n') + store.finish_job(job_id, True, 'Done', 1.25) + + job = store.get_job(job_id) + self.assertEqual(job['status'], 'succeeded') + self.assertEqual(job['progress'], 100) + self.assertEqual(job['current_stage'], 'complete') + self.assertIn('line one', job['log_text']) + self.assertEqual(job['duration_seconds'], 1.25) + + def test_database_backup_and_restore_round_trip(self): + from database.maintenance import backup_database, restore_database + + with tempfile.TemporaryDirectory() as temp_dir: + source = Path(temp_dir) / 'source.db' + backup = Path(temp_dir) / 'backup.db' + with sqlite3.connect(str(source)) as db: + db.execute('CREATE TABLE values_table (value INTEGER)') + db.execute('INSERT INTO values_table VALUES (1)') + + backup_database(source, backup) + with sqlite3.connect(str(source)) as db: + db.execute('UPDATE values_table SET value = 2') + restore_database(backup, source) + + with sqlite3.connect(str(source)) as db: + value = db.execute( + 'SELECT value FROM values_table' + ).fetchone()[0] + self.assertEqual(value, 1) + + def test_high_frequency_queries_use_operational_indexes(self): + l2 = sqlite3.connect(Config.DB_L2_PATH) + try: + plans = { + 'player': l2.execute( + """ + EXPLAIN QUERY PLAN + SELECT * FROM fact_match_players + WHERE steam_id_64 = ? + """, + ['76561198330488905'], + ).fetchone()[3], + 'party': l2.execute( + """ + EXPLAIN QUERY PLAN + SELECT * FROM fact_match_players + WHERE match_id = ? AND match_team_id = ? + """, + ['match', 1], + ).fetchone()[3], + 'victim': l2.execute( + """ + EXPLAIN QUERY PLAN + SELECT * FROM fact_round_events + WHERE victim_steam_id = ? + """, + ['player'], + ).fetchone()[3], + } + finally: + l2.close() + + self.assertIn('idx_match_players_player_match', plans['player']) + self.assertIn('idx_match_players_party', plans['party']) + self.assertIn('idx_round_events_victim', plans['victim']) + + def test_player_records_reference_real_matches(self): + l3 = sqlite3.connect(Config.DB_L3_PATH) + l2 = sqlite3.connect(Config.DB_L2_PATH) + try: + records = l3.execute( + """ + SELECT match_id + FROM dm_player_records + WHERE match_id IS NOT NULL + """ + ).fetchall() + self.assertGreater(len(records), 0) + for (match_id,) in records: + exists = l2.execute( + 'SELECT 1 FROM fact_matches WHERE match_id = ?', + [match_id], + ).fetchone() + self.assertIsNotNone(exists) + finally: + l3.close() + l2.close() + + +if __name__ == '__main__': + unittest.main() diff --git a/web/app.py b/web/app.py index c317bbc..6dada33 100644 --- a/web/app.py +++ b/web/app.py @@ -1,20 +1,20 @@ import sys import os -# Add the project root directory to sys.path sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) -from flask import Flask, render_template +from flask import Flask from web.config import Config -from web.database import close_dbs +from web.database import close_dbs, initialize_web_db -def create_app(): + +def create_app(config_object=Config): app = Flask(__name__) - app.config.from_object(Config) - + app.config.from_object(config_object) + + initialize_web_db() app.teardown_appcontext(close_dbs) - - # Register Blueprints + from web.routes import main, matches, players, teams, tactics, admin, wiki, opponents app.register_blueprint(main.bp) app.register_blueprint(matches.bp) @@ -24,13 +24,10 @@ def create_app(): app.register_blueprint(admin.bp) app.register_blueprint(wiki.bp) app.register_blueprint(opponents.bp) - - @app.route('/') - def index(): - return render_template('home/index.html') - + return app + if __name__ == '__main__': app = create_app() app.run(debug=True, port=5000) diff --git a/web/config.py b/web/config.py index 4bbd83c..687e3ab 100644 --- a/web/config.py +++ b/web/config.py @@ -1,14 +1,25 @@ import os +from database.paths import L2_DB, L3_DB, WEB_DB, WEB_SCHEMA + + class Config: - SECRET_KEY = os.environ.get('SECRET_KEY') or 'yrtv-secret-key-dev' BASE_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) - - DB_L2_PATH = os.path.join(BASE_DIR, 'database', 'L2', 'L2.db') - DB_L3_PATH = os.path.join(BASE_DIR, 'database', 'L3', 'L3.db') - DB_WEB_PATH = os.path.join(BASE_DIR, 'database', 'Web', 'Web_App.sqlite') - - ADMIN_TOKEN = 'jackyyang0929' - + + SECRET_KEY = os.environ.get('SECRET_KEY', 'yrtv-dev-only-change-me') + ADMIN_TOKEN = os.environ.get('ADMIN_TOKEN', 'yrtv-admin-dev') + + DB_L2_PATH = str(L2_DB) + DB_L3_PATH = str(L3_DB) + DB_WEB_PATH = str(WEB_DB) + DB_WEB_SCHEMA_PATH = str(WEB_SCHEMA) + WEB_SCHEMA_VERSION = 2 + + MAX_CONTENT_LENGTH = 5 * 1024 * 1024 + SQLITE_TIMEOUT_SECONDS = 15 + SLOW_QUERY_THRESHOLD_SECONDS = float( + os.environ.get('SLOW_QUERY_THRESHOLD_SECONDS', '0.25') + ) + # Pagination ITEMS_PER_PAGE = 20 diff --git a/web/database.py b/web/database.py index 5982ab1..897d90b 100644 --- a/web/database.py +++ b/web/database.py @@ -1,7 +1,96 @@ +import os +import logging import sqlite3 +import time + from flask import g from web.config import Config +logger = logging.getLogger(__name__) + + +def _database_path(db_name): + paths = { + 'l2': Config.DB_L2_PATH, + 'l3': Config.DB_L3_PATH, + 'web': Config.DB_WEB_PATH, + } + try: + return paths[db_name] + except KeyError as exc: + raise ValueError(f"Unknown database: {db_name}") from exc + + +def initialize_web_db(): + """Create and migrate the small application-owned database.""" + os.makedirs(os.path.dirname(Config.DB_WEB_PATH), exist_ok=True) + db = sqlite3.connect( + Config.DB_WEB_PATH, + timeout=Config.SQLITE_TIMEOUT_SECONDS, + ) + try: + table_exists = db.execute( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name='team_lineups'" + ).fetchone() + if table_exists: + columns = { + row[1] for row in db.execute("PRAGMA table_info(team_lineups)") + } + if 'is_active' not in columns: + db.execute( + "ALTER TABLE team_lineups " + "ADD COLUMN is_active INTEGER NOT NULL DEFAULT 0" + ) + first_id = db.execute( + "SELECT id FROM team_lineups " + "ORDER BY created_at DESC, id DESC LIMIT 1" + ).fetchone() + if first_id: + db.execute( + "UPDATE team_lineups SET is_active = 1 WHERE id = ?", + first_id, + ) + else: + active_ids = [ + row[0] for row in db.execute( + "SELECT id FROM team_lineups WHERE is_active = 1 " + "ORDER BY created_at DESC, id DESC" + ) + ] + if not active_ids: + latest_id = db.execute( + "SELECT id FROM team_lineups " + "ORDER BY created_at DESC, id DESC LIMIT 1" + ).fetchone() + if latest_id: + db.execute( + "UPDATE team_lineups SET is_active = 1 WHERE id = ?", + latest_id, + ) + elif len(active_ids) > 1: + db.execute("UPDATE team_lineups SET is_active = 0") + db.execute( + "UPDATE team_lineups SET is_active = 1 WHERE id = ?", + [active_ids[0]], + ) + + with open(Config.DB_WEB_SCHEMA_PATH, 'r', encoding='utf-8') as schema_file: + db.executescript(schema_file.read()) + db.execute( + """ + INSERT OR IGNORE INTO schema_migrations (version, description) + VALUES (?, ?) + """, + [ + Config.WEB_SCHEMA_VERSION, + 'ETL jobs, match imports and active lineup governance', + ], + ) + db.commit() + finally: + db.close() + + def get_db(db_name): """ db_name: 'l2', 'l3', or 'web' @@ -10,18 +99,20 @@ def get_db(db_name): db = getattr(g, db_attr, None) if db is None: - if db_name == 'l2': - path = Config.DB_L2_PATH - elif db_name == 'l3': - path = Config.DB_L3_PATH - elif db_name == 'web': - path = Config.DB_WEB_PATH - else: - raise ValueError(f"Unknown database: {db_name}") - - # Connect with check_same_thread=False if needed for dev, but default is safer per thread - db = sqlite3.connect(path) + path = _database_path(db_name) + if db_name != 'web' and not os.path.exists(path): + raise RuntimeError( + f"{db_name.upper()} database does not exist: {path}. " + "Run the corresponding data builder first." + ) + db = sqlite3.connect( + path, + timeout=Config.SQLITE_TIMEOUT_SECONDS, + ) db.row_factory = sqlite3.Row + db.execute("PRAGMA busy_timeout = 15000") + if db_name != 'l3': + db.execute("PRAGMA foreign_keys = ON") setattr(g, db_attr, db) return db @@ -34,14 +125,34 @@ def close_dbs(e=None): db.close() def query_db(db_name, query, args=(), one=False): + started = time.perf_counter() cur = get_db(db_name).execute(query, args) - rv = cur.fetchall() - cur.close() + try: + rv = cur.fetchall() + finally: + cur.close() + duration = time.perf_counter() - started + if duration >= Config.SLOW_QUERY_THRESHOLD_SECONDS: + logger.warning( + "Slow query db=%s duration=%.3fs sql=%s", + db_name, + duration, + " ".join(query.split())[:500], + ) return (rv[0] if rv else None) if one else rv def execute_db(db_name, query, args=()): db = get_db(db_name) + started = time.perf_counter() cur = db.execute(query, args) db.commit() + duration = time.perf_counter() - started + if duration >= Config.SLOW_QUERY_THRESHOLD_SECONDS: + logger.warning( + "Slow write db=%s duration=%.3fs sql=%s", + db_name, + duration, + " ".join(query.split())[:500], + ) cur.close() return cur.lastrowid diff --git a/web/routes/admin.py b/web/routes/admin.py index ba9f47a..e8edcf1 100644 --- a/web/routes/admin.py +++ b/web/routes/admin.py @@ -1,16 +1,17 @@ -from flask import Blueprint, render_template, request, redirect, url_for, session, flash +from flask import Blueprint, render_template, request, redirect, url_for, session, flash, jsonify from web.config import Config from web.auth import admin_required from web.database import query_db -import os +from web.services.etl_service import EtlService +import hmac bp = Blueprint('admin', __name__, url_prefix='/admin') @bp.route('/login', methods=['GET', 'POST']) def login(): if request.method == 'POST': - token = request.form.get('token') - if token == Config.ADMIN_TOKEN: + token = request.form.get('token') or '' + if hmac.compare_digest(token, Config.ADMIN_TOKEN): session['is_admin'] = True return redirect(url_for('admin.dashboard')) else: @@ -27,19 +28,104 @@ def logout(): def dashboard(): return render_template('admin/dashboard.html') -from web.services.etl_service import EtlService +@bp.route('/data-integrity') +@admin_required +def data_integrity(): + from web.services.integrity_service import IntegrityService + + report = IntegrityService.build_report() + if request.args.get('format') == 'json': + return jsonify(report) + return render_template('admin/data_integrity.html', report=report) @bp.route('/trigger_etl', methods=['POST']) @admin_required def trigger_etl(): - script_name = request.form.get('script') - allowed = ['L1A.py', 'L2_Builder.py', 'L3_Builder.py'] - if script_name not in allowed: - return "Invalid script", 400 - - success, message = EtlService.run_script(script_name) - status_code = 200 if success else 500 - return message, status_code + from database.job_store import JobStore + + store = JobStore(Config.DB_WEB_PATH) + job_id = store.create_job( + 'manual_pipeline', + created_by='admin', + ) + try: + pid = EtlService.start_pipeline(job_id) + except Exception as exc: + store.finish_job(job_id, False, str(exc), 0) + return jsonify({'success': False, 'error': str(exc)}), 500 + return jsonify({'success': True, 'job_id': job_id, 'pid': pid}), 202 + + +@bp.route('/import-match', methods=['GET', 'POST']) +@admin_required +def import_match(): + from database.job_store import JobStore + from web.services.import_service import ( + DuplicateMatchError, + ImportValidationError, + MatchImportService, + ) + + store = JobStore(Config.DB_WEB_PATH) + if request.method == 'POST': + upload = request.files.get('capture') + if not upload or not upload.filename: + flash('请选择 iframe_network.json 文件。', 'error') + return redirect(url_for('admin.import_match')) + + prepared = None + try: + prepared = MatchImportService.prepare_import( + upload.read(), + upload.filename, + created_by='admin', + replace=request.form.get('replace') == '1', + ) + EtlService.start_pipeline( + prepared['job_id'], + match_id=prepared['match_id'], + replace=prepared['replace'], + ) + flash( + f"比赛 {prepared['match_id']} 已进入导入队列。", + 'success', + ) + return redirect(url_for( + 'admin.import_match', + job_id=prepared['job_id'], + )) + except (DuplicateMatchError, ImportValidationError) as exc: + flash(str(exc), 'warning') + except Exception as exc: + if prepared: + store.finish_job( + prepared['job_id'], + False, + f'Failed to start pipeline: {exc}', + 0, + ) + flash(f'启动导入失败：{exc}', 'error') + + selected_job = None + selected_job_id = request.args.get('job_id', type=int) + if selected_job_id: + selected_job = store.get_job(selected_job_id) + return render_template( + 'admin/import_match.html', + jobs=store.list_jobs(30), + selected_job=selected_job, + ) + + +@bp.route('/api/jobs/') +@admin_required +def api_job(job_id): + from database.job_store import JobStore + + job = JobStore(Config.DB_WEB_PATH).get_job(job_id) + if not job: + return jsonify({'error': 'Job not found'}), 404 + return jsonify(job) @bp.route('/sql', methods=['GET', 'POST']) @admin_required @@ -50,18 +136,21 @@ def sql_runner(): db_name = "l2" if request.method == 'POST': - query = request.form.get('query') + query = (request.form.get('query') or '').strip() db_name = request.form.get('db_name', 'l2') - - # Safety check - forbidden = ['DELETE', 'DROP', 'UPDATE', 'INSERT', 'ALTER', 'GRANT', 'REVOKE'] - if any(x in query.upper() for x in forbidden): - error = "Only SELECT queries allowed in Web Runner." + + statement = query.rstrip(';').strip() + if db_name not in {'l2', 'l3', 'web'}: + error = "Unknown database." + elif not statement.upper().startswith('SELECT '): + error = "Only SELECT queries are allowed." + elif ';' in statement: + error = "Only one SQL statement is allowed." else: try: - # Enforce limit if not present - if 'LIMIT' not in query.upper(): - query += " LIMIT 50" + query = statement + if 'LIMIT' not in statement.upper(): + query = f"{statement} LIMIT 50" rows = query_db(db_name, query) if rows: diff --git a/web/routes/main.py b/web/routes/main.py index 5384865..2b79137 100644 --- a/web/routes/main.py +++ b/web/routes/main.py @@ -1,6 +1,5 @@ from flask import Blueprint, render_template, request, jsonify from web.services.stats_service import StatsService -import time bp = Blueprint('main', __name__) @@ -18,18 +17,17 @@ def index(): return render_template('home/index.html', recent_matches=recent_matches, heatmap_data=heatmap_data, live_matches=live_matches) -from web.services.etl_service import EtlService - @bp.route('/parse_match', methods=['POST']) def parse_match(): url = request.form.get('url') if not url or '5eplay.com' not in url: - return jsonify({'success': False, 'message': 'Invalid 5EPlay URL'}) - - # Trigger L1A.py with URL argument - success, msg = EtlService.run_script('L1A.py', args=[url]) - - if success: - return jsonify({'success': True, 'message': 'Match parsing completed successfully!'}) - else: - return jsonify({'success': False, 'message': f'Error: {msg}'}) + return jsonify({'success': False, 'message': 'Invalid 5EPlay URL'}), 400 + + return jsonify({ + 'success': False, + 'message': ( + 'URL downloader is not included in this repository. ' + 'Place iframe_network.json under output_arena// ' + 'and run the L1/L2/L3 builders from Admin.' + ), + }), 501 diff --git a/web/routes/matches.py b/web/routes/matches.py index 026e6c7..f7d434f 100644 --- a/web/routes/matches.py +++ b/web/routes/matches.py @@ -33,19 +33,9 @@ def detail(match_id): rounds = StatsService.get_match_rounds(match_id) - # --- Roster Identification --- - # Fetch active roster to identify "Our Team" players - from web.services.web_service import WebService - lineups = WebService.get_lineups() - # Assume we use the first/active lineup - active_roster_ids = [] - if lineups: - try: - active_roster_ids = json.loads(lineups[0]['player_ids_json']) - except: - pass - - # Mark roster players (Ensure strict string comparison) + from web.services.team_context_service import TeamContextService + + active_roster_ids = TeamContextService.get_active_roster_ids() roster_set = set(str(uid) for uid in active_roster_ids) for p in players: p['is_in_roster'] = str(p['steam_id_64']) in roster_set diff --git a/web/routes/players.py b/web/routes/players.py index d05cbb4..1614878 100644 --- a/web/routes/players.py +++ b/web/routes/players.py @@ -1,15 +1,16 @@ from flask import Blueprint, render_template, request, jsonify, redirect, url_for, flash, current_app, session from web.services.stats_service import StatsService from web.services.feature_service import FeatureService +from web.services.player_profile_service import PlayerProfileService from web.services.web_service import WebService from web.database import execute_db, query_db from web.config import Config from datetime import datetime import os import json -from werkzeug.utils import secure_filename bp = Blueprint('players', __name__, url_prefix='/players') +ALLOWED_AVATAR_EXTENSIONS = {'.jpg', '.jpeg', '.png', '.webp'} @bp.route('/') def index(): @@ -41,7 +42,12 @@ def detail(steam_id): # Use steam_id as filename to ensure uniqueness per player # Preserve extension ext = os.path.splitext(file.filename)[1].lower() - if not ext: ext = '.jpg' + if ( + ext not in ALLOWED_AVATAR_EXTENSIONS + or not (file.mimetype or '').startswith('image/') + ): + flash('Avatar must be a JPG, PNG, or WebP image.', 'error') + return redirect(url_for('players.detail', steam_id=steam_id)) filename = f"{steam_id}{ext}" upload_folder = os.path.join(current_app.root_path, 'static', 'avatars') @@ -177,33 +183,9 @@ def detail(steam_id): history_asc = StatsService.get_player_trend(steam_id, limit=1000) history = history_asc[::-1] if history_asc else [] - # Calculate Map Stats - map_stats = {} - for match in history: - m_name = match['map_name'] - if m_name not in map_stats: - map_stats[m_name] = {'matches': 0, 'wins': 0, 'adr_sum': 0, 'rating_sum': 0} - - map_stats[m_name]['matches'] += 1 - if match['is_win']: - map_stats[m_name]['wins'] += 1 - map_stats[m_name]['adr_sum'] += (match['adr'] or 0) - map_stats[m_name]['rating_sum'] += (match['rating'] or 0) - - map_stats_list = [] - for m_name, data in map_stats.items(): - cnt = data['matches'] - map_stats_list.append({ - 'map_name': m_name, - 'matches': cnt, - 'win_rate': data['wins'] / cnt, - 'adr': data['adr_sum'] / cnt, - 'rating': data['rating_sum'] / cnt - }) - map_stats_list.sort(key=lambda x: x['matches'], reverse=True) - - # --- New: Recent Performance Stats --- - # recent_stats = StatsService.get_recent_performance_stats(steam_id) + map_stats_list = PlayerProfileService.get_map_stats(steam_id) + period_stats = PlayerProfileService.get_period_stats(steam_id) + records = PlayerProfileService.get_records(steam_id) return render_template('players/profile.html', player=player, @@ -213,6 +195,8 @@ def detail(steam_id): history=history, distribution=distribution, map_stats=map_stats_list, + period_stats=period_stats, + records=records, l2_stats=l2_stats, side_stats=side_stats) @@ -223,9 +207,13 @@ def like_comment(comment_id): @bp.route('//charts_data') def charts_data(steam_id): - # ... (existing code) ... - # Trend Data - trends = StatsService.get_player_trend(steam_id, limit=1000) + period_key = request.args.get('period', 'last_20') + if period_key not in PlayerProfileService.PERIOD_KEYS: + period_key = 'last_20' + trends = PlayerProfileService.get_period_history(steam_id, period_key) + if not trends: + trends = StatsService.get_player_trend(steam_id, limit=20) + period_summary = PlayerProfileService.get_period(steam_id, period_key) # Radar Data (Construct from features) features = FeatureService.get_player_features(steam_id) @@ -234,17 +222,11 @@ def charts_data(steam_id): # Task 1: Strict Team Average Calculation team_avg_radar = None - lineups = WebService.get_lineups() - if lineups: - target_lineup = None - try: - p_ids = [str(i) for i in json.loads(lineups[0].get("player_ids_json") or "[]")] - if str(steam_id) in p_ids: - target_lineup = p_ids - except: - target_lineup = None - - if target_lineup: + from web.services.team_context_service import TeamContextService + + active_roster_ids = TeamContextService.get_active_roster_ids() + target_lineup = active_roster_ids if str(steam_id) in active_roster_ids else None + if target_lineup: # Calculate strict average for this lineup team_sums = { 'score_aim': 0.0, 'score_defense': 0.0, 'score_utility': 0.0, @@ -303,9 +285,19 @@ def charts_data(steam_id): 'trend': {'labels': trend_labels, 'values': trend_values}, 'radar': radar_data, 'radar_dist': radar_dist, - 'team_avg_radar': team_avg_radar + 'team_avg_radar': team_avg_radar, + 'period': period_summary, }) + +@bp.route('//period_stats') +def period_stats(steam_id): + period_key = request.args.get('period', 'last_20') + period = PlayerProfileService.get_period(steam_id, period_key) + if not period: + return jsonify({'error': 'Unknown period or player'}), 404 + return jsonify(period) + # --- API for Comparison --- @bp.route('/api/search') def api_search(): diff --git a/web/routes/teams.py b/web/routes/teams.py index 8168ccb..1a2cb3b 100644 --- a/web/routes/teams.py +++ b/web/routes/teams.py @@ -67,14 +67,12 @@ def api_search(): @bp.route('/api/roster', methods=['GET', 'POST']) def api_roster(): - # Assume single team mode, always operating on ID=1 or the first lineup - lineups = WebService.get_lineups() - if not lineups: - # Auto-create default team if none exists + target_team = WebService.get_active_lineup() + if not target_team: WebService.save_lineup("My Team", "Default Roster", []) - lineups = WebService.get_lineups() - - target_team = dict(lineups[0]) # Get the latest one + target_team = WebService.get_active_lineup() + + target_team = dict(target_team) if request.method == 'POST': # Admin Check diff --git a/web/services/etl_service.py b/web/services/etl_service.py index fdec001..d2b9263 100644 --- a/web/services/etl_service.py +++ b/web/services/etl_service.py @@ -4,13 +4,51 @@ import sys from web.config import Config class EtlService: + SCRIPT_PATHS = { + 'L1A.py': os.path.join('database', 'L1', 'L1_Builder.py'), + 'L2_Builder.py': os.path.join('database', 'L2', 'L2_Builder.py'), + 'L3_Builder.py': os.path.join('database', 'L3', 'L3_Builder.py'), + } + + @staticmethod + def start_pipeline(job_id, match_id=None, replace=False): + script_path = os.path.join( + Config.BASE_DIR, + 'database', + 'pipeline.py', + ) + command = [ + sys.executable, + script_path, + '--job-id', + str(int(job_id)), + ] + if match_id: + command.extend(['--match-id', str(match_id)]) + if replace: + command.append('--replace') + + process = subprocess.Popen( + command, + cwd=Config.BASE_DIR, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + start_new_session=True, + ) + return process.pid + @staticmethod def run_script(script_name, args=None): """ - Executes an ETL script located in the ETL directory. + Executes an allow-listed data builder from its actual repository path. Returns (success, message) """ - script_path = os.path.join(Config.BASE_DIR, 'ETL', script_name) + relative_path = EtlService.SCRIPT_PATHS.get(script_name) + if not relative_path: + return False, f"Unsupported data script: {script_name}" + + script_path = os.path.join(Config.BASE_DIR, relative_path) if not os.path.exists(script_path): return False, f"Script not found: {script_path}" @@ -28,7 +66,7 @@ class EtlService: cwd=Config.BASE_DIR, capture_output=True, text=True, - timeout=300 # 5 min timeout + timeout=900 ) if result.returncode == 0: diff --git a/web/services/feature_service.py b/web/services/feature_service.py index 99eedf8..b29b24a 100644 --- a/web/services/feature_service.py +++ b/web/services/feature_service.py @@ -1,6 +1,6 @@ from __future__ import annotations -from typing import Any, Iterable +from typing import Any from web.database import query_db @@ -138,20 +138,47 @@ class FeatureService: } order_col = sort_map.get(sort_by, "core_avg_rating") - where = [] - args: list[Any] = [] if search: - where.append("steam_id_64 IN (SELECT steam_id_64 FROM dim_players WHERE username LIKE ?)") - args.append(f"%{search}%") - where_sql = f"WHERE {' AND '.join(where)}" if where else "" - - rows = query_db( - "l3", - f"SELECT * FROM dm_player_features {where_sql} ORDER BY {order_col} DESC LIMIT ? OFFSET ?", - args + [per_page, offset], - ) - total_row = query_db("l3", f"SELECT COUNT(*) as cnt FROM dm_player_features {where_sql}", args, one=True) - total = int(total_row["cnt"]) if total_row else 0 + dim_rows = query_db( + "l2", + """ + SELECT steam_id_64 + FROM dim_players + WHERE LOWER(username) LIKE LOWER(?) OR steam_id_64 LIKE ? + """, + [f"%{search}%", f"%{search}%"], + ) + matching_ids = [str(row["steam_id_64"]) for row in dim_rows] + rows = [] + for start in range(0, len(matching_ids), 500): + chunk = matching_ids[start:start + 500] + placeholders = ",".join("?" for _ in chunk) + rows.extend(query_db( + "l3", + f"SELECT * FROM dm_player_features " + f"WHERE steam_id_64 IN ({placeholders})", + chunk, + )) + rows = sorted( + rows, + key=lambda row: row[order_col] if row[order_col] is not None else float("-inf"), + reverse=True, + ) + total = len(rows) + rows = rows[offset:offset + per_page] + else: + rows = query_db( + "l3", + f"SELECT * FROM dm_player_features " + f"ORDER BY {order_col} DESC LIMIT ? OFFSET ?", + [per_page, offset], + ) + total_row = query_db( + "l3", + "SELECT COUNT(*) as cnt FROM dm_player_features", + one=True, + ) + total = int(total_row["cnt"]) if total_row else 0 players = [FeatureService._normalize_features(dict(r)) for r in rows] if rows else [] players = [p for p in players if p] @@ -160,19 +187,11 @@ class FeatureService: @staticmethod def get_roster_features_distribution(target_steam_id: str): - from web.services.web_service import WebService - import json + from web.services.team_context_service import TeamContextService - lineups = WebService.get_lineups() - roster_ids: list[str] = [] - - if lineups: - try: - p_ids = [str(i) for i in json.loads(lineups[0].get("player_ids_json") or "[]")] - if str(target_steam_id) in p_ids: - roster_ids = p_ids - except Exception: - roster_ids = [] + roster_ids = TeamContextService.get_active_roster_ids() + if str(target_steam_id) not in roster_ids: + roster_ids = [] if not roster_ids: return None @@ -202,7 +221,17 @@ class FeatureService: sample_keys = list(p.keys()) break - lower_is_better = {"int_timing_first_contact_time", "tac_avg_fd", "core_avg_match_duration"} + lower_is_better = { + "int_timing_first_contact_time", + "int_trade_response_time", + "tac_avg_fd", + "tac_fd_rate", + "core_avg_match_duration", + "core_dpr", + "meta_rating_volatility", + "meta_map_stability", + "meta_elo_tier_stability", + } result: dict[str, Any] = {} for m in sample_keys: @@ -224,16 +253,22 @@ class FeatureService: values = [] for p in stats_map.values(): v = (p or {}).get(m) + if v is None: + continue try: - values.append(float(v) if v is not None else 0.0) + values.append(float(v)) except (ValueError, TypeError): - values.append(0.0) + continue target_val_raw = (stats_map.get(target_steam_id) or {}).get(m) + if target_val_raw is None or not values: + result[m] = None + continue try: - target_val = float(target_val_raw) if target_val_raw is not None else 0.0 + target_val = float(target_val_raw) except (ValueError, TypeError): - target_val = 0.0 + result[m] = None + continue is_reverse = m not in lower_is_better # Sort values. For standard metrics, higher is better (reverse=True). @@ -251,9 +286,9 @@ class FeatureService: "val": target_val, "rank": rank, "total": len(values_sorted), - "min": min(values_sorted) if values_sorted else 0, - "max": max(values_sorted) if values_sorted else 0, - "avg": (sum(values_sorted) / len(values_sorted)) if values_sorted else 0, + "min": min(values_sorted), + "max": max(values_sorted), + "avg": sum(values_sorted) / len(values_sorted), "inverted": not is_reverse, } return result diff --git a/web/services/import_service.py b/web/services/import_service.py new file mode 100644 index 0000000..8462beb --- /dev/null +++ b/web/services/import_service.py @@ -0,0 +1,186 @@ +import hashlib +import json +import os +from pathlib import Path +import re +import sqlite3 +from typing import Any, Dict + +from database.job_store import JobStore +from database.paths import L1_DB, OUTPUT_ARENA +from web.config import Config + + +MATCH_ID_PATTERN = re.compile(r'\bg161-[0-9]{10,}\b') + + +class ImportValidationError(ValueError): + pass + + +class DuplicateMatchError(ImportValidationError): + pass + + +class MatchImportService: + @staticmethod + def validate_capture(raw_bytes: bytes) -> Dict[str, Any]: + if not raw_bytes: + raise ImportValidationError('Uploaded file is empty') + + try: + text = raw_bytes.decode('utf-8-sig') + except UnicodeDecodeError as exc: + raise ImportValidationError('Capture must be UTF-8 JSON') from exc + + try: + capture = json.loads(text) + except json.JSONDecodeError as exc: + raise ImportValidationError( + f'Invalid JSON at line {exc.lineno}, column {exc.colno}' + ) from exc + + if not isinstance(capture, list) or not capture: + raise ImportValidationError( + 'Capture root must be a non-empty list of network responses' + ) + + urls = [] + successful_responses = 0 + for index, item in enumerate(capture): + if not isinstance(item, dict): + raise ImportValidationError( + f'Capture item {index} must be an object' + ) + url = item.get('url') + if not isinstance(url, str) or not url: + raise ImportValidationError( + f'Capture item {index} has no URL' + ) + urls.append(url) + if item.get('status') == 200 and item.get('body') is not None: + successful_responses += 1 + + match_ids = sorted({ + match.group(0) + for url in urls + for match in MATCH_ID_PATTERN.finditer(url) + }) + if len(match_ids) != 1: + raise ImportValidationError( + f'Capture must reference exactly one match ID; found {match_ids}' + ) + if successful_responses < 2: + raise ImportValidationError( + 'Capture does not contain enough successful API responses' + ) + + match_id = match_ids[0] + has_match_data = any( + f'/api/data/match/{match_id}' in url for url in urls + ) + has_round_data = any( + f'/api/match/round/{match_id}' in url for url in urls + ) + if not has_match_data or not has_round_data: + missing = [] + if not has_match_data: + missing.append('match data') + if not has_round_data: + missing.append('round data') + raise ImportValidationError( + f"Capture is missing required endpoint(s): {', '.join(missing)}" + ) + + return { + 'match_id': match_id, + 'content_sha256': hashlib.sha256(raw_bytes).hexdigest(), + 'response_count': len(capture), + 'successful_responses': successful_responses, + 'text': text, + } + + @staticmethod + def _existing_l1_content(match_id: str): + if not L1_DB.exists(): + return None + db = sqlite3.connect(str(L1_DB)) + try: + row = db.execute( + """ + SELECT content + FROM raw_iframe_network + WHERE match_id = ? + """, + [match_id], + ).fetchone() + return row[0] if row else None + finally: + db.close() + + @staticmethod + def prepare_import( + raw_bytes: bytes, + original_filename: str, + created_by: str, + replace: bool = False, + ): + validation = MatchImportService.validate_capture(raw_bytes) + match_id = validation['match_id'] + content_hash = validation['content_sha256'] + + existing_content = MatchImportService._existing_l1_content(match_id) + if existing_content is not None: + existing_hash = hashlib.sha256( + existing_content.encode('utf-8') + ).hexdigest() + if existing_hash == content_hash: + raise DuplicateMatchError( + f'Match {match_id} is already imported with identical data' + ) + if not replace: + raise DuplicateMatchError( + f'Match {match_id} already exists with different data; ' + 'explicit replacement is required' + ) + + match_dir = OUTPUT_ARENA / match_id + match_dir.mkdir(parents=True, exist_ok=True) + destination = match_dir / 'iframe_network.json' + if destination.exists() and not replace: + current_hash = hashlib.sha256(destination.read_bytes()).hexdigest() + if current_hash == content_hash: + raise DuplicateMatchError( + f'Match {match_id} is already queued with identical data' + ) + raise DuplicateMatchError( + f'Pending capture already exists for {match_id}' + ) + + temporary = destination.with_suffix('.json.tmp') + temporary.write_bytes(raw_bytes) + os.replace(str(temporary), str(destination)) + + store = JobStore(Config.DB_WEB_PATH) + job_id = store.create_job( + 'match_import', + match_id=match_id, + input_path=str(destination), + created_by=created_by, + ) + store.upsert_match_import( + match_id, + content_hash, + str(destination), + 'queued', + job_id, + ) + return { + 'job_id': job_id, + 'match_id': match_id, + 'content_sha256': content_hash, + 'response_count': validation['response_count'], + 'source_path': str(destination), + 'original_filename': Path(original_filename or '').name, + 'replace': bool(replace), + } diff --git a/web/services/integrity_service.py b/web/services/integrity_service.py new file mode 100644 index 0000000..727662d --- /dev/null +++ b/web/services/integrity_service.py @@ -0,0 +1,511 @@ +from datetime import datetime, timezone +import json +import os +import sqlite3 + +from database.maintenance import backup_storage_status +from web.config import Config +from web.services.team_context_service import TeamContextService + + +class IntegrityService: + DATABASES = { + 'L2': Config.DB_L2_PATH, + 'L3': Config.DB_L3_PATH, + 'Web': Config.DB_WEB_PATH, + } + + @staticmethod + def _check(checks, name, status, detail, value=None): + checks.append({ + 'name': name, + 'status': status, + 'detail': detail, + 'value': value, + }) + + @staticmethod + def _connect(path): + db = sqlite3.connect(path, timeout=Config.SQLITE_TIMEOUT_SECONDS) + db.row_factory = sqlite3.Row + return db + + @staticmethod + def build_report(): + checks = [] + counts = {} + connections = {} + + try: + for name, path in IntegrityService.DATABASES.items(): + if not os.path.exists(path): + IntegrityService._check( + checks, + f'{name} database', + 'fail', + f'Missing file: {path}', + ) + continue + try: + db = IntegrityService._connect(path) + connections[name] = db + result = db.execute('PRAGMA quick_check').fetchone()[0] + IntegrityService._check( + checks, + f'{name} database', + 'pass' if result == 'ok' else 'fail', + f'quick_check: {result}', + os.path.getsize(path), + ) + except sqlite3.Error as exc: + IntegrityService._check( + checks, + f'{name} database', + 'fail', + str(exc), + ) + + l2 = connections.get('L2') + if l2: + IntegrityService._check_l2(l2, checks, counts) + + l3 = connections.get('L3') + roster_ids = TeamContextService.get_active_roster_ids() + counts['active_roster'] = len(roster_ids) + if l3: + IntegrityService._check_l3(l3, roster_ids, checks, counts) + + web = connections.get('Web') + if web: + IntegrityService._check_web(web, checks, counts) + + backup_status = backup_storage_status() + counts['backup_sets'] = backup_status['sets'] + counts['backup_bytes'] = backup_status['total_bytes'] + IntegrityService._check( + checks, + 'Backup retention', + 'warn' if backup_status['sets'] > 3 else 'pass', + ( + f"{backup_status['sets']} backup sets, " + f"{backup_status['total_bytes']:,} bytes" + ), + backup_status['sets'], + ) + finally: + for db in connections.values(): + db.close() + + status_order = {'pass': 0, 'warn': 1, 'fail': 2} + overall_status = max( + (check['status'] for check in checks), + key=lambda status: status_order[status], + default='fail', + ) + return { + 'generated_at': datetime.now(timezone.utc).isoformat(), + 'overall_status': overall_status, + 'counts': counts, + 'checks': checks, + 'totals': { + status: sum(1 for check in checks if check['status'] == status) + for status in ('pass', 'warn', 'fail') + }, + } + + @staticmethod + def _table_names(db): + return { + row[0] + for row in db.execute( + "SELECT name FROM sqlite_master WHERE type='table'" + ) + } + + @staticmethod + def _check_l2(db, checks, counts): + required_tables = { + 'dim_players', + 'fact_matches', + 'fact_match_teams', + 'fact_match_players', + 'fact_rounds', + 'fact_round_events', + 'fact_round_player_economy', + } + missing = sorted(required_tables - IntegrityService._table_names(db)) + IntegrityService._check( + checks, + 'L2 required tables', + 'fail' if missing else 'pass', + f"Missing: {', '.join(missing)}" if missing else 'All required tables exist', + ) + if missing: + return + + table_count_map = { + 'matches': 'fact_matches', + 'players': 'dim_players', + 'player_match_rows': 'fact_match_players', + 'rounds': 'fact_rounds', + 'events': 'fact_round_events', + 'economy_rows': 'fact_round_player_economy', + } + for key, table in table_count_map.items(): + counts[key] = db.execute(f'SELECT COUNT(*) FROM {table}').fetchone()[0] + + orphan_players = db.execute( + """ + SELECT COUNT(*) + FROM fact_match_players mp + LEFT JOIN fact_matches m ON m.match_id = mp.match_id + WHERE m.match_id IS NULL + """ + ).fetchone()[0] + IntegrityService._check( + checks, + 'Player-match referential integrity', + 'fail' if orphan_players else 'pass', + f'{orphan_players} player rows reference missing matches', + orphan_players, + ) + + orphan_events = db.execute( + """ + SELECT COUNT(*) + FROM fact_round_events e + LEFT JOIN fact_rounds r + ON r.match_id = e.match_id AND r.round_num = e.round_num + WHERE r.match_id IS NULL + """ + ).fetchone()[0] + IntegrityService._check( + checks, + 'Round-event referential integrity', + 'fail' if orphan_events else 'pass', + f'{orphan_events} events reference missing rounds', + orphan_events, + ) + + unusual_rosters = db.execute( + """ + SELECT COUNT(*) + FROM ( + SELECT match_id, COUNT(*) AS player_count + FROM fact_match_players + GROUP BY match_id + HAVING player_count != 10 + ) + """ + ).fetchone()[0] + IntegrityService._check( + checks, + 'Match player cardinality', + 'warn' if unusual_rosters else 'pass', + f'{unusual_rosters} matches do not contain exactly 10 players', + unusual_rosters, + ) + + missing_names = db.execute( + "SELECT COUNT(*) FROM dim_players WHERE username IS NULL OR TRIM(username) = ''" + ).fetchone()[0] + IntegrityService._check( + checks, + 'Player identity coverage', + 'warn' if missing_names else 'pass', + f'{missing_names} players have no username', + missing_names, + ) + + required_indexes = { + 'idx_match_players_player_match', + 'idx_match_players_match_team', + 'idx_match_players_party', + 'idx_round_events_victim', + 'idx_economy_player_match', + 'idx_matches_map_time', + } + existing_indexes = { + row[0] for row in db.execute( + "SELECT name FROM sqlite_master WHERE type = 'index'" + ) + } + missing_indexes = sorted(required_indexes - existing_indexes) + IntegrityService._check( + checks, + 'L2 operational indexes', + 'fail' if missing_indexes else 'pass', + ( + f"Missing: {', '.join(missing_indexes)}" + if missing_indexes else + 'All high-frequency query indexes exist' + ), + len(required_indexes) - len(missing_indexes), + ) + + @staticmethod + def _check_l3(db, roster_ids, checks, counts): + required_tables = { + 'dm_player_features', + 'dm_player_match_history', + 'dm_player_map_stats', + 'dm_player_period_stats', + 'dm_player_records', + 'dm_player_weapon_stats', + } + missing = sorted(required_tables - IntegrityService._table_names(db)) + IntegrityService._check( + checks, + 'L3 required tables', + 'fail' if missing else 'pass', + f"Missing: {', '.join(missing)}" if missing else 'All required tables exist', + ) + if missing: + return + + counts['l3_features'] = db.execute( + 'SELECT COUNT(*) FROM dm_player_features' + ).fetchone()[0] + counts['l3_history'] = db.execute( + 'SELECT COUNT(*) FROM dm_player_match_history' + ).fetchone()[0] + counts['l3_maps'] = db.execute( + 'SELECT COUNT(*) FROM dm_player_map_stats' + ).fetchone()[0] + counts['l3_weapons'] = db.execute( + 'SELECT COUNT(*) FROM dm_player_weapon_stats' + ).fetchone()[0] + counts['l3_periods'] = db.execute( + 'SELECT COUNT(*) FROM dm_player_period_stats' + ).fetchone()[0] + counts['l3_records'] = db.execute( + 'SELECT COUNT(*) FROM dm_player_records' + ).fetchone()[0] + + if roster_ids: + placeholders = ','.join('?' for _ in roster_ids) + covered = db.execute( + f""" + SELECT COUNT(DISTINCT steam_id_64) + FROM dm_player_features + WHERE steam_id_64 IN ({placeholders}) + """, + roster_ids, + ).fetchone()[0] + IntegrityService._check( + checks, + 'Active roster feature coverage', + 'pass' if covered == len(roster_ids) else 'fail', + f'{covered}/{len(roster_ids)} roster players have L3 features', + covered, + ) + + expected_history = db.execute( + f""" + SELECT COALESCE(SUM(total_matches), 0) + FROM dm_player_features + WHERE steam_id_64 IN ({placeholders}) + """, + roster_ids, + ).fetchone()[0] + actual_history = db.execute( + f""" + SELECT COUNT(*) + FROM dm_player_match_history + WHERE steam_id_64 IN ({placeholders}) + """, + roster_ids, + ).fetchone()[0] + IntegrityService._check( + checks, + 'Roster history completeness', + 'pass' if actual_history == expected_history else 'fail', + f'{actual_history}/{expected_history} player-match rows materialized', + actual_history, + ) + + score_rows = db.execute( + f""" + SELECT steam_id_64, score_overall, tier_percentile + FROM dm_player_features + WHERE steam_id_64 IN ({placeholders}) + AND score_overall > 0 + """, + roster_ids, + ).fetchall() + scores = [float(row['score_overall']) for row in score_rows] + invalid_percentiles = 0 + for row in score_rows: + expected = ( + sum(value <= float(row['score_overall']) for value in scores) + / len(scores) + * 100 + ) + actual = row['tier_percentile'] + if actual is None or abs(float(actual) - expected) > 0.011: + invalid_percentiles += 1 + IntegrityService._check( + checks, + 'Roster percentile correctness', + 'pass' if invalid_percentiles == 0 else 'warn', + f'{invalid_percentiles} eligible players have stale percentiles', + invalid_percentiles, + ) + + for table, label in ( + ('dm_player_period_stats', 'Roster period-stat coverage'), + ('dm_player_records', 'Roster record coverage'), + ): + covered = db.execute( + f""" + SELECT COUNT(DISTINCT steam_id_64) + FROM {table} + WHERE steam_id_64 IN ({placeholders}) + """, + roster_ids, + ).fetchone()[0] + IntegrityService._check( + checks, + label, + 'pass' if covered == len(roster_ids) else 'fail', + f'{covered}/{len(roster_ids)} roster players covered', + covered, + ) + + for key, label in ( + ('l3_history', 'Player match history mart'), + ('l3_maps', 'Player map stats mart'), + ('l3_weapons', 'Player weapon stats mart'), + ('l3_periods', 'Player period stats mart'), + ('l3_records', 'Player records mart'), + ): + value = counts[key] + IntegrityService._check( + checks, + label, + 'pass' if value else 'warn', + f'{value} rows', + value, + ) + + if roster_ids: + placeholders = ','.join('?' for _ in roster_ids) + scope_sql = f"AND steam_id_64 IN ({placeholders})" + scope_args = roster_ids + else: + scope_sql = '' + scope_args = [] + placeholder_rows = db.execute( + f""" + SELECT COUNT(*) + FROM dm_player_features + WHERE int_pos_site_a_control_rate = 0.33 + AND int_pos_site_b_control_rate = 0.33 + AND int_pos_mid_control_rate = 0.34 + {scope_sql} + """, + scope_args, + ).fetchone()[0] + IntegrityService._check( + checks, + 'Experimental spatial metrics', + 'warn' if placeholder_rows else 'pass', + f'{placeholder_rows} active-roster rows contain placeholder site-control values', + placeholder_rows, + ) + + @staticmethod + def _check_web(db, checks, counts): + required_tables = { + 'comments', + 'etl_jobs', + 'match_imports', + 'player_metadata', + 'schema_migrations', + 'strategy_boards', + 'team_lineups', + 'wiki_pages', + } + missing = sorted(required_tables - IntegrityService._table_names(db)) + IntegrityService._check( + checks, + 'Web required tables', + 'fail' if missing else 'pass', + f"Missing: {', '.join(missing)}" if missing else 'All required tables exist', + ) + if missing: + return + + counts['etl_jobs'] = db.execute( + 'SELECT COUNT(*) FROM etl_jobs' + ).fetchone()[0] + counts['match_imports'] = db.execute( + 'SELECT COUNT(*) FROM match_imports' + ).fetchone()[0] + + schema_version = db.execute( + 'SELECT COALESCE(MAX(version), 0) FROM schema_migrations' + ).fetchone()[0] + counts['web_schema_version'] = schema_version + IntegrityService._check( + checks, + 'Web schema version', + 'pass' if schema_version == Config.WEB_SCHEMA_VERSION else 'fail', + f'{schema_version}/{Config.WEB_SCHEMA_VERSION}', + schema_version, + ) + + foreign_key_errors = db.execute( + 'PRAGMA foreign_key_check' + ).fetchall() + IntegrityService._check( + checks, + 'Web foreign key integrity', + 'fail' if foreign_key_errors else 'pass', + f'{len(foreign_key_errors)} foreign key violations', + len(foreign_key_errors), + ) + + running_jobs = db.execute( + """ + SELECT COUNT(*) + FROM etl_jobs + WHERE status = 'running' + """ + ).fetchone()[0] + IntegrityService._check( + checks, + 'Pipeline concurrency', + 'warn' if running_jobs > 1 else 'pass', + f'{running_jobs} running pipeline jobs', + running_jobs, + ) + + lineups = db.execute( + 'SELECT id, player_ids_json, is_active FROM team_lineups' + ).fetchall() + counts['lineups'] = len(lineups) + invalid_lineups = 0 + for lineup in lineups: + try: + player_ids = json.loads(lineup['player_ids_json'] or '[]') + if not isinstance(player_ids, list): + invalid_lineups += 1 + except (TypeError, json.JSONDecodeError): + invalid_lineups += 1 + IntegrityService._check( + checks, + 'Lineup JSON validity', + 'fail' if invalid_lineups else 'pass', + f'{invalid_lineups} lineups contain invalid player ID JSON', + invalid_lineups, + ) + + active_count = sum(1 for lineup in lineups if lineup['is_active'] == 1) + IntegrityService._check( + checks, + 'Active lineup', + 'pass' if active_count == 1 else 'warn', + f'{active_count} active lineups configured', + active_count, + ) diff --git a/web/services/opponent_service.py b/web/services/opponent_service.py index efae06b..f92d4f6 100644 --- a/web/services/opponent_service.py +++ b/web/services/opponent_service.py @@ -1,19 +1,10 @@ from web.database import query_db -from web.services.web_service import WebService -import json +from web.services.team_context_service import TeamContextService class OpponentService: @staticmethod def _get_active_roster_ids(): - lineups = WebService.get_lineups() - active_roster_ids = [] - if lineups: - try: - raw_ids = json.loads(lineups[0]['player_ids_json']) - active_roster_ids = [str(uid) for uid in raw_ids] - except: - pass - return active_roster_ids + return TeamContextService.get_active_roster_ids() @staticmethod def get_opponent_list(page=1, per_page=20, sort_by='matches', search=None): @@ -21,30 +12,21 @@ class OpponentService: if not roster_ids: return [], 0 - # Placeholders roster_ph = ','.join('?' for _ in roster_ids) - - # 1. Identify Matches involving our roster (at least 1 member? usually 2 for 'team' match) - # Let's say at least 1 for broader coverage as requested ("1 match sample") - # But "Our Team" usually implies the entity. Let's stick to matches where we can identify "Us". - # If we use >=1, we catch solo Q matches of roster members. The user said "Non-team members or 1 match sample", - # but implied "facing different our team lineups". - # Let's use the standard "candidate matches" logic (>=2 roster members) to represent "The Team". - # OR, if user wants "Opponent Analysis" for even 1 match, maybe they mean ANY match in DB? - # "Left Top add Opponent Analysis... (non-team member or 1 sample)" - # This implies we analyze PLAYERS who are NOT us. - # Let's stick to matches where >= 1 roster member played, to define "Us" vs "Them". - - # Actually, let's look at ALL matches in DB, and any player NOT in active roster is an "Opponent". - # This covers "1 sample". - - # Query: - # Select all players who are NOT in active roster. - # Group by steam_id. - # Aggregate stats. - - where_clauses = [f"CAST(mp.steam_id_64 AS TEXT) NOT IN ({roster_ph})"] - args = list(roster_ids) + + where_clauses = [ + f"CAST(mp.steam_id_64 AS TEXT) NOT IN ({roster_ph})", + f""" + EXISTS ( + SELECT 1 + FROM fact_match_players roster_mp + WHERE roster_mp.match_id = mp.match_id + AND CAST(roster_mp.steam_id_64 AS TEXT) IN ({roster_ph}) + AND roster_mp.team_id != mp.team_id + ) + """, + ] + args = list(roster_ids) + list(roster_ids) if search: where_clauses.append("(LOWER(p.username) LIKE LOWER(?) OR mp.steam_id_64 LIKE ?)") @@ -61,16 +43,6 @@ class OpponentService: elif sort_by == 'win_rate': sort_sql = "win_rate DESC" - # Main Aggregation Query - # We need to join fact_matches to get match info (win/loss, elo) if needed, - # but fact_match_players has is_win (boolean) usually? No, it has team_id. - # We need to determine if THEY won. - # fact_match_players doesn't store is_win directly in schema (I should check schema, but stats_service calculates it). - # Wait, stats_service.get_player_trend uses `mp.is_win`? - # Let's check schema. `fact_match_players` usually has `match_id`, `team_id`. - # `fact_matches` has `winner_team`. - # So we join. - offset = (page - 1) * per_page sql = f""" @@ -151,10 +123,17 @@ class OpponentService: LEFT JOIN fact_match_teams fmt_gid ON mp.match_id = fmt_gid.match_id AND fmt_gid.group_id = mp.team_id LEFT JOIN fact_match_teams fmt_tid ON mp.match_id = fmt_tid.match_id AND fmt_tid.group_tid = mp.match_team_id WHERE CAST(mp.steam_id_64 AS TEXT) NOT IN ({roster_ph}) + AND EXISTS ( + SELECT 1 + FROM fact_match_players roster_mp + WHERE roster_mp.match_id = mp.match_id + AND CAST(roster_mp.steam_id_64 AS TEXT) IN ({roster_ph}) + AND roster_mp.team_id != mp.team_id + ) GROUP BY mp.steam_id_64 """ - rows = query_db('l2', sql, roster_ids) + rows = query_db('l2', sql, roster_ids + roster_ids) # Initialize Buckets elo_buckets = {'<1000': 0, '1000-1200': 0, '1200-1400': 0, '1400-1600': 0, '1600-1800': 0, '1800-2000': 0, '>2000': 0} @@ -216,13 +195,12 @@ class OpponentService: player = dict(info) player['avatar_url'] = StatsService.resolve_avatar_url(steam_id, player.get('avatar_url')) - # 2. Match History vs Us (All matches this player played) - # We define "Us" as matches where this player is an opponent. - # But actually, we just show ALL their matches in our DB, assuming our DB only contains matches relevant to us? - # Usually yes, but if we have a huge DB, we might want to filter by "Contains Roster Member". - # For now, show all matches in DB for this player. - - sql_history = """ + roster_ids = OpponentService._get_active_roster_ids() + if not roster_ids: + return None + roster_ph = ','.join('?' for _ in roster_ids) + + sql_history = f""" SELECT m.match_id, m.start_time, m.map_name, m.score_team1, m.score_team2, m.winner_team, mp.team_id, mp.match_team_id, mp.rating, mp.kd_ratio, mp.adr, mp.kills, mp.deaths, @@ -236,9 +214,16 @@ class OpponentService: LEFT JOIN fact_match_teams fmt_gid ON mp.match_id = fmt_gid.match_id AND fmt_gid.group_id = mp.team_id LEFT JOIN fact_match_teams fmt_tid ON mp.match_id = fmt_tid.match_id AND fmt_tid.group_tid = mp.match_team_id WHERE mp.steam_id_64 = ? + AND EXISTS ( + SELECT 1 + FROM fact_match_players roster_mp + WHERE roster_mp.match_id = mp.match_id + AND CAST(roster_mp.steam_id_64 AS TEXT) IN ({roster_ph}) + AND roster_mp.team_id != mp.team_id + ) ORDER BY m.start_time DESC """ - history = query_db('l2', sql_history, [steam_id]) + history = query_db('l2', sql_history, [steam_id] + roster_ids) # 3. Aggregation by ELO elo_buckets = { @@ -389,11 +374,18 @@ class OpponentService: LEFT JOIN fact_match_teams fmt_gid ON mp.match_id = fmt_gid.match_id AND fmt_gid.group_id = mp.team_id LEFT JOIN fact_match_teams fmt_tid ON mp.match_id = fmt_tid.match_id AND fmt_tid.group_tid = mp.match_team_id WHERE CAST(mp.steam_id_64 AS TEXT) NOT IN ({roster_ph}) + AND EXISTS ( + SELECT 1 + FROM fact_match_players roster_mp + WHERE roster_mp.match_id = mp.match_id + AND CAST(roster_mp.steam_id_64 AS TEXT) IN ({roster_ph}) + AND roster_mp.team_id != mp.team_id + ) AND m.map_name IS NOT NULL AND m.map_name <> '' GROUP BY m.map_name ORDER BY matches DESC """ - rows = query_db('l2', sql, roster_ids) + rows = query_db('l2', sql, roster_ids + roster_ids) results = [] for r in rows: d = dict(r) diff --git a/web/services/player_profile_service.py b/web/services/player_profile_service.py new file mode 100644 index 0000000..74f4808 --- /dev/null +++ b/web/services/player_profile_service.py @@ -0,0 +1,123 @@ +from web.database import query_db + + +class PlayerProfileService: + PERIOD_KEYS = ( + 'career', + 'last_10', + 'last_20', + 'last_30', + 'days_30', + 'days_90', + ) + + @staticmethod + def get_period_stats(steam_id): + rows = query_db( + 'l3', + """ + SELECT * + FROM dm_player_period_stats + WHERE steam_id_64 = ? + ORDER BY CASE period_key + WHEN 'career' THEN 1 + WHEN 'last_10' THEN 2 + WHEN 'last_20' THEN 3 + WHEN 'last_30' THEN 4 + WHEN 'days_30' THEN 5 + WHEN 'days_90' THEN 6 + ELSE 99 + END + """, + [steam_id], + ) + return [dict(row) for row in rows] + + @staticmethod + def get_period(steam_id, period_key): + if period_key not in PlayerProfileService.PERIOD_KEYS: + return None + row = query_db( + 'l3', + """ + SELECT * + FROM dm_player_period_stats + WHERE steam_id_64 = ? AND period_key = ? + """, + [steam_id, period_key], + one=True, + ) + return dict(row) if row else None + + @staticmethod + def get_records(steam_id): + rows = query_db( + 'l3', + """ + SELECT * + FROM dm_player_records + WHERE steam_id_64 = ? + ORDER BY CASE record_key + WHEN 'highest_rating' THEN 1 + WHEN 'most_kills' THEN 2 + WHEN 'highest_adr' THEN 3 + WHEN 'highest_kd' THEN 4 + WHEN 'most_headshots' THEN 5 + WHEN 'longest_win_streak' THEN 6 + ELSE 99 + END + """, + [steam_id], + ) + return [dict(row) for row in rows] + + @staticmethod + def get_period_history(steam_id, period_key): + period = PlayerProfileService.get_period(steam_id, period_key) + if not period: + return [] + rows = query_db( + 'l3', + """ + SELECT + match_date AS start_time, + rating, + kd_ratio, + adr, + kast, + match_id, + map_name, + is_win, + match_sequence AS match_index + FROM dm_player_match_history + WHERE steam_id_64 = ? + AND match_date BETWEEN ? AND ? + ORDER BY match_date, match_id + """, + [steam_id, period['period_start'], period['period_end']], + ) + return [dict(row) for row in rows] + + @staticmethod + def get_map_stats(steam_id): + rows = query_db( + 'l3', + """ + SELECT + map_name, + matches, + wins, + win_rate, + avg_rating AS rating, + avg_kd AS kd, + avg_adr AS adr, + avg_kast AS kast, + best_rating, + worst_rating + FROM dm_player_map_stats + WHERE steam_id_64 = ? + ORDER BY matches DESC, map_name + """, + [steam_id], + ) + return [dict(row) for row in rows] diff --git a/web/services/stats_service.py b/web/services/stats_service.py index 23738a8..b3c6a76 100644 --- a/web/services/stats_service.py +++ b/web/services/stats_service.py @@ -1,4 +1,4 @@ -from web.database import query_db, execute_db +from web.database import query_db from flask import current_app, url_for import os @@ -13,7 +13,7 @@ class StatsService: try: # Check local file first (User Request: "directly associate if exists") base = os.path.join(current_app.root_path, 'static', 'avatars') - for ext in ('.jpg', '.png', '.jpeg'): + for ext in ('.jpg', '.png', '.jpeg', '.webp'): fname = f"{steam_id}{ext}" fpath = os.path.join(base, fname) if os.path.exists(fpath): @@ -38,18 +38,9 @@ class StatsService: 'round_stats': [{'type', 'count', 'wins', 'win_rate'}] } """ - # 1. Get Active Roster - from web.services.web_service import WebService - import json - - lineups = WebService.get_lineups() - active_roster_ids = [] - if lineups: - try: - raw_ids = json.loads(lineups[0]['player_ids_json']) - active_roster_ids = [str(uid) for uid in raw_ids] - except: - pass + from web.services.team_context_service import TeamContextService + + active_roster_ids = TeamContextService.get_active_roster_ids() if not active_roster_ids: return {} @@ -60,21 +51,23 @@ class StatsService: placeholders = ','.join('?' for _ in active_roster_ids) - # Step A: Get Candidate Match IDs (matches with >= 2 roster players) - # Also get the team_id of our players in that match to determine win candidate_sql = f""" - SELECT mp.match_id, MAX(mp.team_id) as our_team_id + SELECT mp.match_id, mp.team_id as our_team_id, + COUNT(DISTINCT mp.steam_id_64) as roster_count FROM fact_match_players mp WHERE CAST(mp.steam_id_64 AS TEXT) IN ({placeholders}) - GROUP BY mp.match_id + GROUP BY mp.match_id, mp.team_id HAVING COUNT(DISTINCT mp.steam_id_64) >= 2 + ORDER BY mp.match_id, roster_count DESC, mp.team_id """ candidate_rows = query_db('l2', candidate_sql, active_roster_ids) if not candidate_rows: return {} - candidate_map = {row['match_id']: row['our_team_id'] for row in candidate_rows} + candidate_map = {} + for row in candidate_rows: + candidate_map.setdefault(row['match_id'], row['our_team_id']) match_ids = list(candidate_map.keys()) match_placeholders = ','.join('?' for _ in match_ids) @@ -221,11 +214,15 @@ class StatsService: args.append(map_name) if date_from: - where_clauses.append("start_time >= ?") + where_clauses.append( + "start_time >= CAST(strftime('%s', ?) AS INTEGER)" + ) args.append(date_from) if date_to: - where_clauses.append("start_time <= ?") + where_clauses.append( + "start_time < CAST(strftime('%s', date(?, '+1 day')) AS INTEGER)" + ) args.append(date_to) where_str = " AND ".join(where_clauses) @@ -270,109 +267,51 @@ class StatsService: party_rows = query_db('l2', party_sql, match_ids) party_map = {row['match_id']: row['max_party'] for row in party_rows} - # --- New: Determine "Our Team" Result --- - # Logic: Check if any player from `active_roster` played in these matches. - # Use WebService to get the active roster - from web.services.web_service import WebService - import json - - lineups = WebService.get_lineups() - active_roster_ids = [] - if lineups: - try: - # Load IDs and ensure they are all strings for DB comparison consistency - raw_ids = json.loads(lineups[0]['player_ids_json']) - active_roster_ids = [str(uid) for uid in raw_ids] - except: - pass + from web.services.team_context_service import TeamContextService + + active_roster_ids = TeamContextService.get_active_roster_ids() # If no roster, we can't determine "Our Result" if not active_roster_ids: result_map = {} else: - # 1. Get UIDs for Roster Members involved in these matches - # We query fact_match_players to ensure we get the UIDs actually used in these matches roster_placeholders = ','.join('?' for _ in active_roster_ids) - uid_sql = f""" - SELECT DISTINCT steam_id_64, uid + roster_team_sql = f""" + SELECT match_id, team_id, + COUNT(DISTINCT steam_id_64) as roster_count FROM fact_match_players WHERE match_id IN ({placeholders}) AND CAST(steam_id_64 AS TEXT) IN ({roster_placeholders}) + GROUP BY match_id, team_id """ - combined_args_uid = match_ids + active_roster_ids - uid_rows = query_db('l2', uid_sql, combined_args_uid) - - # Set of "Our UIDs" (as strings) - our_uids = set() - for r in uid_rows: - if r['uid']: - our_uids.add(str(r['uid'])) - - # 2. Get Group UIDs and Winner info from fact_match_teams - # We need to know which group contains our UIDs - teams_sql = f""" - SELECT fmt.match_id, fmt.group_id, fmt.group_uids, m.winner_team - FROM fact_match_teams fmt - JOIN fact_matches m ON fmt.match_id = m.match_id - WHERE fmt.match_id IN ({placeholders}) - """ - teams_rows = query_db('l2', teams_sql, match_ids) - - # 3. Determine Result per Match + roster_team_rows = query_db( + 'l2', + roster_team_sql, + match_ids + active_roster_ids, + ) + winner_by_match = { + str(match['match_id']): match['winner_team'] + for match in matches + } + teams_by_match = {} + for row in roster_team_rows: + teams_by_match.setdefault(str(row['match_id']), []).append( + row['team_id'] + ) + result_map = {} - - # Group data by match - match_groups = {} # match_id -> {group_id: [uids...], winner: int} - - for r in teams_rows: - mid = r['match_id'] - gid = r['group_id'] - uids_str = r['group_uids'] or "" - # Split and clean UIDs - uids = set(str(u).strip() for u in uids_str.split(',') if u.strip()) - - if mid not in match_groups: - match_groups[mid] = {'groups': {}, 'winner': r['winner_team']} - - match_groups[mid]['groups'][gid] = uids - - # Analyze - for mid, data in match_groups.items(): - winner_gid = data['winner'] - groups = data['groups'] - - our_in_winner = False - our_in_loser = False - - # Check each group - for gid, uids in groups.items(): - # Intersection of Our UIDs and Group UIDs - common = our_uids.intersection(uids) - if common: - if gid == winner_gid: - our_in_winner = True - else: - our_in_loser = True - - if our_in_winner and not our_in_loser: - result_map[mid] = 'win' - elif our_in_loser and not our_in_winner: - result_map[mid] = 'loss' - elif our_in_winner and our_in_loser: - result_map[mid] = 'mixed' - else: - # Fallback: If UID matching failed (maybe missing UIDs), try old team_id method? - # Or just leave it as None (safe) - pass + for match_id, team_ids in teams_by_match.items(): + unique_team_ids = set(team_ids) + if len(unique_team_ids) > 1: + result_map[match_id] = 'mixed' + continue + our_team_id = next(iter(unique_team_ids)) + result_map[match_id] = ( + 'win' + if str(our_team_id) == str(winner_by_match.get(match_id)) + else 'loss' + ) - # Convert to dict to modify - matches = [dict(m) for m in matches] - for m in matches: - m['avg_elo'] = elo_map.get(m['match_id'], 0) - m['max_party'] = party_map.get(m['match_id'], 1) - m['our_result'] = result_map.get(m['match_id']) - - # Convert to dict to modify matches = [dict(m) for m in matches] for m in matches: m['avg_elo'] = elo_map.get(m['match_id'], 0) @@ -542,33 +481,20 @@ class StatsService: @staticmethod def get_shared_matches(steam_ids): - # Find matches where ALL steam_ids were present if not steam_ids or len(steam_ids) < 1: return [] - + + steam_ids = list(dict.fromkeys(str(steam_id) for steam_id in steam_ids)) placeholders = ','.join('?' for _ in steam_ids) count = len(steam_ids) - - # We need to know which team the players were on to determine win/loss - # Assuming they were on the SAME team for "shared experience" - # If count=1, it's just match history - - # Query: Get matches where all steam_ids are present - # Also join to get team_id to check if they were on the same team (optional but better) - # For simplicity in v1: Just check presence in the match. - # AND check if the player won. - - # We need to return: match_id, map_name, score, result (Win/Loss) - # "Result" is relative to the lineup. - # If they were on the winning team, it's a Win. - + sql = f""" SELECT m.match_id, m.start_time, m.map_name, m.score_team1, m.score_team2, m.winner_team, - MAX(mp.team_id) as player_team_id -- Just take one team_id (assuming same) + mp.team_id as player_team_id FROM fact_matches m JOIN fact_match_players mp ON m.match_id = mp.match_id WHERE mp.steam_id_64 IN ({placeholders}) - GROUP BY m.match_id + GROUP BY m.match_id, mp.team_id HAVING COUNT(DISTINCT mp.steam_id_64) = ? ORDER BY m.start_time DESC """ @@ -580,14 +506,7 @@ class StatsService: results = [] for r in rows: - # Determine if Win - # winner_team in DB is 'Team 1' or 'Team 2' usually, or the team name. - # fact_matches.winner_team stores the NAME of the winner? Or 'team1'/'team2'? - # Let's check how L2_Builder stores it. Usually it stores the name. - # But fact_match_players.team_id stores the name too. - - # Logic: If m.winner_team == mp.team_id, then Win. - is_win = (r['winner_team'] == r['player_team_id']) + is_win = str(r['winner_team']) == str(r['player_team_id']) # If winner_team is NULL or empty, it's a draw? if not r['winner_team']: @@ -628,7 +547,31 @@ class StatsService: """ l3_rows = query_db("l3", l3_sql, [steam_id, limit]) if l3_rows: - return l3_rows + history = [dict(row) for row in l3_rows] + match_ids = [row['match_id'] for row in history] + placeholders = ','.join('?' for _ in match_ids) + party_rows = query_db( + "l2", + f""" + SELECT me.match_id, COUNT(p.steam_id_64) AS party_size + FROM fact_match_players me + LEFT JOIN fact_match_players p + ON p.match_id = me.match_id + AND p.match_team_id = me.match_team_id + AND me.match_team_id > 0 + WHERE me.steam_id_64 = ? + AND me.match_id IN ({placeholders}) + GROUP BY me.match_id + """, + [steam_id] + match_ids, + ) + party_map = { + row['match_id']: max(int(row['party_size'] or 0), 1) + for row in party_rows + } + for row in history: + row['party_size'] = party_map.get(row['match_id'], 1) + return history sql = """ SELECT * FROM ( @@ -729,19 +672,10 @@ class StatsService: Calculates rank and distribution of the target player within the active roster. Now covers all L3 Basic Features for Detailed Panel. """ - from web.services.web_service import WebService from web.services.feature_service import FeatureService - import json - - # 1. Get Active Roster IDs - lineups = WebService.get_lineups() - active_roster_ids = [] - if lineups: - try: - raw_ids = json.loads(lineups[0]['player_ids_json']) - active_roster_ids = [str(uid) for uid in raw_ids] - except: - pass + from web.services.team_context_service import TeamContextService + + active_roster_ids = TeamContextService.get_active_roster_ids() if not active_roster_ids: return None @@ -851,33 +785,38 @@ class StatsService: "basic_avg_rating", "basic_avg_kd", "basic_avg_adr", "basic_avg_kast", "basic_avg_rws", ] - lower_is_better = [] + lower_is_better = { + "int_timing_first_contact_time", + "int_trade_response_time", + "tac_avg_fd", + "tac_fd_rate", + "core_avg_match_duration", + "core_dpr", + "meta_rating_volatility", + "meta_map_stability", + "meta_elo_tier_stability", + } result = {} for m in metrics: values = [] - non_numeric = False for p in stats_map.values(): raw = (p or {}).get(m) if raw is None: - raw = 0 + continue try: values.append(float(raw)) - except Exception: - non_numeric = True - break + except (TypeError, ValueError): + continue raw_target = (stats_map.get(target_steam_id) or {}).get(m) if raw_target is None: - raw_target = 0 + result[m] = None + continue try: target_val = float(raw_target) - except Exception: - non_numeric = True - target_val = 0 - - if non_numeric: + except (TypeError, ValueError): result[m] = None continue diff --git a/web/services/team_context_service.py b/web/services/team_context_service.py new file mode 100644 index 0000000..d3f7f1e --- /dev/null +++ b/web/services/team_context_service.py @@ -0,0 +1,36 @@ +import json + +from web.services.web_service import WebService + + +class TeamContextService: + """Single source of truth for the private team's active roster.""" + + @staticmethod + def get_active_lineup(): + lineup = WebService.get_active_lineup() + return dict(lineup) if lineup else None + + @staticmethod + def get_active_roster_ids(): + lineup = TeamContextService.get_active_lineup() + if not lineup: + return [] + + try: + raw_ids = json.loads(lineup.get('player_ids_json') or '[]') + except (TypeError, json.JSONDecodeError): + return [] + + if not isinstance(raw_ids, list): + return [] + + seen = set() + roster_ids = [] + for raw_id in raw_ids: + steam_id = str(raw_id).strip() + if steam_id and steam_id not in seen: + seen.add(steam_id) + roster_ids.append(steam_id) + return roster_ids + diff --git a/web/services/web_service.py b/web/services/web_service.py index 590f456..0c5298a 100644 --- a/web/services/web_service.py +++ b/web/services/web_service.py @@ -53,17 +53,39 @@ class WebService: sql = "UPDATE team_lineups SET name=?, description=?, player_ids_json=? WHERE id=?" return execute_db('web', sql, [name, description, ids_json, lineup_id]) else: - sql = "INSERT INTO team_lineups (name, description, player_ids_json) VALUES (?, ?, ?)" - return execute_db('web', sql, [name, description, ids_json]) + active = 0 if WebService.get_active_lineup() else 1 + sql = """ + INSERT INTO team_lineups + (name, description, player_ids_json, is_active) + VALUES (?, ?, ?, ?) + """ + return execute_db('web', sql, [name, description, ids_json, active]) @staticmethod def get_lineups(): - return query_db('web', "SELECT * FROM team_lineups ORDER BY created_at DESC") + return query_db( + 'web', + "SELECT * FROM team_lineups ORDER BY is_active DESC, created_at DESC, id DESC", + ) @staticmethod def get_lineup(lineup_id): return query_db('web', "SELECT * FROM team_lineups WHERE id = ?", [lineup_id], one=True) + @staticmethod + def get_active_lineup(): + lineup = query_db( + 'web', + "SELECT * FROM team_lineups WHERE is_active = 1 ORDER BY id LIMIT 1", + one=True, + ) + if lineup: + return lineup + return query_db( + 'web', + "SELECT * FROM team_lineups ORDER BY created_at DESC, id DESC LIMIT 1", + one=True, + ) # --- Users / Auth --- @staticmethod diff --git a/web/templates/admin/dashboard.html b/web/templates/admin/dashboard.html index 0959fba..cb0ddd2 100644 --- a/web/templates/admin/dashboard.html +++ b/web/templates/admin/dashboard.html @@ -12,9 +12,8 @@

数据管线 (ETL)

- - - + 上传并导入比赛 +
@@ -23,6 +22,7 @@

工具箱

+ 数据完整性中心 SQL Runner Manage Wiki
@@ -31,20 +31,23 @@
+{% endif %} +{% endblock %} diff --git a/web/templates/players/_career_dashboard.html b/web/templates/players/_career_dashboard.html new file mode 100644 index 0000000..15f0ec8 --- /dev/null +++ b/web/templates/players/_career_dashboard.html @@ -0,0 +1,88 @@ +
+
+
+
+

阶段表现

+

窗口按该玩家最新一场比赛向前计算

+
+ +
+ + {% for period in period_stats %} +
+
+
+
Matches
+
{{ period.matches }}
+
+
+
Rating
+
{{ '%.2f'|format(period.avg_rating or 0) }}
+
+
+
K/D
+
{{ '%.2f'|format(period.avg_kd or 0) }}
+
+
+
ADR
+
{{ '%.1f'|format(period.avg_adr or 0) }}
+
+
+
KAST
+
{{ '%.1f%%'|format((period.avg_kast or 0) * 100) }}
+
+
+
Win Rate
+
+ {{ '%.0f%%'|format((period.win_rate or 0) * 100) }} +
+
+
+
+ {% if period.sample_reliable %} + 样本充足 + {% else %} + 样本不足，仅供参考 + {% endif %} +
+
+ {% else %} +
暂无阶段统计
+ {% endfor %} +
+ +
+
+

职业纪录

+

每项纪录都可追溯到具体比赛

+
+
+ {% for record in records %} + +
{{ record.record_label }}
+
+ {% if record.record_key in ['most_kills', 'most_headshots', 'longest_win_streak'] %} + {{ record.record_value|int }} + {% elif record.record_key == 'highest_adr' %} + {{ '%.1f'|format(record.record_value) }} + {% else %} + {{ '%.2f'|format(record.record_value) }} + {% endif %} +
+
{{ record.map_name }}
+ + {% else %} +
暂无职业纪录
+ {% endfor %} +
+
+
diff --git a/web/templates/players/profile.html b/web/templates/players/profile.html index 538bff5..df7c829 100644 --- a/web/templates/players/profile.html +++ b/web/templates/players/profile.html @@ -213,6 +213,8 @@ + {% include "players/_career_dashboard.html" %} +
@@ -221,8 +223,11 @@

📈 近期表现走势 (Performance Trend)

-
- +
+ + + +
@@ -256,7 +261,7 @@
{% macro detail_item(label, value, key, format_str='{:.2f}', sublabel=None, count_label=None) %} - {% set dist = distribution[key] if distribution else None %} + {% set dist = distribution[key] if distribution and distribution[key] else None %}
{{ label }} @@ -273,7 +278,11 @@
- {{ format_str.format(value if value is not none else 0) }} + {% if value is none %} + N/A + {% else %} + {{ format_str.format(value) }} + {% endif %} {% if sublabel %} {{ sublabel }} @@ -834,6 +843,7 @@ {% block scripts %}