"""P0 数据一致性测试： - 审核入库岗位跨图谱重建持久化（真实区间/来源/场景行业边/演化事件）； - 节点编辑跨重建保留且与 /api/jobs 的 JobDefinition 一致； - JD 手动导入（文件上传 / 文本粘贴 / 坏行清单 三路径）； - 访问令牌空串清除端到端； - 语义权重相对路径（相对工程根解析）检测。 """ from __future__ import annotations import json import time from datetime import date import pytest from fastapi.testclient import TestClient from app.api.routes import app from app.config import load_config from app.graph.tkg import TemporalKG from app.models import JDRecord @pytest.fixture(scope="module") def client(graph_path): with TestClient(app) as c: # 触发 startup 加载图谱 yield c @pytest.fixture(autouse=True) def _restore_kg(): """本模块的 API 调用会改内存图谱；每个用例结束后从真实快照重载，避免串扰。""" yield import app.api.routes as routes from app.config import resolve gp = resolve(routes._cfg, "graph_path") if gp.exists(): routes._state["kg"] = TemporalKG.load(gp) def _jd(jid: str, title: str, pub: str, req: list[str], pref: list[str] | None = None) -> dict: text = (f"岗位名称：{title}\n岗位职责：\n1、负责相关研发工作\n" f"任职要求：\n1、熟悉 {'、'.join(req)}\n" f"加分项：\n1、了解 {'、'.join(pref or ['团队协作'])}\n") return {"jid": jid, "job_title": title, "company": f"测试公司{jid}", "city": "深圳", "industry": "互联网/软件", "publish_date": pub, "text": text} JAVA_RECORDS = [ _jd("j1", "Java开发工程师", "2022-01-15", ["Java", "MySQL"]), _jd("j2", "Java开发工程师", "2022-04-15", ["Java", "Redis"]), _jd("j3", "Java开发工程师", "2023-07-15", ["Java", "MySQL"]), ] def _tmp_cfg(tmp_path, records) -> dict: """最小数据集 + 全部状态路径指向 tmp 的 pipeline 配置。""" jd_dir = tmp_path / "jd_raw" jd_dir.mkdir(exist_ok=True) (jd_dir / "batch_2022.jsonl").write_text( "\n".join(json.dumps(r, ensure_ascii=False) for r in records) + "\n", encoding="utf-8") cfg = load_config() cfg["jd_raw_dir"] = str(jd_dir) cfg["graph_path"] = str(tmp_path / "graph.json") cfg["state_db_path"] = str(tmp_path / "state.db") cfg["audit_path"] = str(tmp_path / "audit.jsonl") cfg["pending_path"] = str(tmp_path / "pending.json") return cfg def _wait_task(client, task_id: str) -> dict: for _ in range(150): t = client.get(f"/api/tasks/{task_id}").json()["data"] if t["status"] in ("done", "failed"): return t time.sleep(0.2) raise AssertionError("任务超时未完成") # ---------- 1. 审核入库真实化 + 跨重建持久化 ---------- def test_accept_real_data_and_persists_across_rebuild(client, tmp_path, monkeypatch): """accept 用候选簇真实统计入库；pipeline 重建后岗位仍在（含场景/行业边与演化事件）。""" import app.api.routes as routes from app.discover.cluster import save_pending from app.models import JobDefinition from app.services.pipeline import run_pipeline from app.storage import db as state_db cfg = _tmp_cfg(tmp_path, JAVA_RECORDS) db_path = tmp_path / "state.db" monkeypatch.setitem(routes._cfg, "state_db_path", str(db_path)) monkeypatch.setitem(routes._cfg, "audit_path", str(tmp_path / "audit.jsonl")) monkeypatch.setitem(routes._cfg, "graph_path", str(tmp_path / "graph_api.json")) cand = JobDefinition( job_id="cand_大模型工程师", title="大模型工程师", category="新岗位候选", industries=["人工智能"], required_skills=["Python", "PyTorch"], preferred_skills=["Docker"], scenarios=["智能客服"], status="pending", expected_years=3, source_count=12, company_count=5, support_source_ids=["import_20260801.jsonl#1", "import_20260801.jsonl#2"], support_windows=["2025Q4", "2025Q3"]) save_pending([cand], db_path) r = client.post("/api/review", json={ "candidate_id": "cand_大模型工程师", "action": "accept", "operator": "张老师"}) assert r.status_code == 200 # 当前图谱：真实区间（首个支持窗口 2025Q3 起）与真实来源、场景/行业边、新增事件 g = client.get("/api/graph").json()["data"] req_edges = [e for e in g["edges"] if e["source"] == "job_大模型工程师" and e.get("rel") == "REQUIRES"] assert len(req_edges) == 3 assert all(e["intervals"][0] == ["2025Q3", None] for e in req_edges) assert all(set(e["sources"]) == {"import_20260801.jsonl#1", "import_20260801.jsonl#2"} for e in req_edges) others = [e for e in g["edges"] if e["source"] == "job_大模型工程师"] assert any(e["rel"] == "APPLIES_TO" and e["target"] == "场景::智能客服" for e in others) assert any(e["rel"] == "BELONGS_TO" and e["target"] == "行业::人工智能" for e in others) evs = [e for e in g["events"] if e["job_id"] == "job_大模型工程师"] assert len(evs) == 3 assert all(e["change_type"] == "新增" and e["window"] == "2025Q3" for e in evs) # 覆写层已记录（跨重建持久化的依据） assert state_db.get_override(db_path, "job_大模型工程师")["kind"] == "job_accept" # 全量重建：数据集中根本没有该岗位（2023 后新岗位），仅靠覆写层重放强制纳入 stats = run_pipeline(cfg) assert stats["stages"]["overrides"]["jobs"] == 1 # 已入库岗位不会重新出现在待审核分区 assert state_db.get_candidate(db_path, "cand_大模型工程师") is None kg = TemporalKG.load(tmp_path / "graph.json") assert "job_大模型工程师" in kg.jobs job = kg.jobs["job_大模型工程师"] assert job.status == "active" and job.title == "大模型工程师" edge = kg.g["job_大模型工程师"]["Python"] assert edge["intervals"][0] == ["2025Q3", None] assert set(edge["sources"]) == {"import_20260801.jsonl#1", "import_20260801.jsonl#2"} assert kg.g.has_edge("job_大模型工程师", "场景::智能客服") assert kg.g.has_edge("job_大模型工程师", "行业::人工智能") assert any(ev.change_type == "新增" and ev.window == "2025Q3" for ev in kg.events_of("job_大模型工程师")) def test_reject_clears_override(client, tmp_path, monkeypatch): """驳回：清理覆写层记录，重建后岗位不复活。""" import app.api.routes as routes from app.discover.cluster import save_pending from app.models import JobDefinition from app.storage import db as state_db db_path = tmp_path / "state.db" monkeypatch.setitem(routes._cfg, "state_db_path", str(db_path)) monkeypatch.setitem(routes._cfg, "audit_path", str(tmp_path / "audit.jsonl")) monkeypatch.setitem(routes._cfg, "graph_path", str(tmp_path / "graph_api.json")) cand = JobDefinition(job_id="cand_短命岗位", title="短命岗位", required_skills=["Python"], status="pending", support_source_ids=["x.jsonl#1"], support_windows=["2025Q2"]) save_pending([cand], db_path) state_db.save_override(db_path, "cand_短命岗位", {"kind": "job_accept", "job": cand.model_dump()}) r = client.post("/api/review", json={ "candidate_id": "cand_短命岗位", "action": "reject", "operator": "张老师", "reason": "证据不足"}) assert r.status_code == 200 assert state_db.get_override(db_path, "cand_短命岗位") is None assert state_db.get_candidate(db_path, "cand_短命岗位") is None def test_discover_candidate_carries_support_evidence(): """新岗位发现：候选带支撑 source_id / 季度窗口 / 行业，供审核入库真实化。""" from app.discover import cluster from app.models import ExtractionResult records, extractions = [], {} for i in range(15): # ≥ MIN_CLUSTER_SIZE(15)，5 家企业 ≥ MIN_COMPANIES(5) r = JDRecord(jid=f"c{i}", job_title="提示词工程师", company=f"公司{i % 5}", industry="人工智能", publish_date=date(2025, i % 6 + 1, 1), text="负责大模型提示词设计与优化", source_id=f"t.jsonl#{i + 1}") records.append(r) extractions[r.jid] = ExtractionResult( jid=r.jid, job_title="提示词工程师", required_skills=["Python", "提示工程"], scenarios=["智能客服"]) cands = cluster.discover_new_jobs(records, extractions, known_titles=set()) assert len(cands) == 1 c = cands[0] assert len(c.support_source_ids) == 15 assert min(c.support_windows) == "2025Q1" assert c.industries == ["人工智能"] # ---------- 2. 节点编辑同步 JobDefinition + 跨重建保留 ---------- def test_node_update_syncs_jobdef_and_persists(client, tmp_path, monkeypatch): """节点编辑同步 kg.jobs；重建后编辑保留且 /api/jobs 定义一致。""" import app.api.routes as routes from app.services.pipeline import run_pipeline cfg = _tmp_cfg(tmp_path, JAVA_RECORDS) db_path = tmp_path / "state.db" monkeypatch.setitem(routes._cfg, "state_db_path", str(db_path)) monkeypatch.setitem(routes._cfg, "audit_path", str(tmp_path / "audit.jsonl")) monkeypatch.setitem(routes._cfg, "graph_path", str(tmp_path / "graph_api.json")) r = client.post("/api/graph/node/update", json={ "node_id": "job_Java开发工程师", "updates": {"title": "Java后端架构师", "level": "高级"}, "operator": "张老师", "reason": "校准"}) assert r.status_code == 200 node = r.json()["data"]["node"] assert node["title"] == "Java后端架构师" and node["level"] == "高级" assert node["expected_years"] == 6 # 级别同步回年限（高级→6） # /api/jobs 的 JobDefinition 同步一致 d = client.get("/api/jobs/job_Java开发工程师").json()["data"]["definition"] assert d["title"] == "Java后端架构师" and d["expected_years"] == 6 # 重建：节点编辑经覆写层重放保留，JobDefinition 同样一致 run_pipeline(cfg) kg = TemporalKG.load(tmp_path / "graph.json") assert "job_Java开发工程师" in kg.jobs # 2022 年出现 → 既有岗位正常重建 attrs = kg.g.nodes["job_Java开发工程师"] assert attrs["title"] == "Java后端架构师" and attrs["level"] == "高级" assert attrs["expected_years"] == 6 jd = kg.jobs["job_Java开发工程师"] assert jd.title == "Java后端架构师" and jd.expected_years == 6 # ---------- 3. JD 手动导入（文件 / 文本 / 坏行） ---------- def _import_setup(tmp_path, monkeypatch): import app.api.routes as routes cfg = _tmp_cfg(tmp_path, JAVA_RECORDS) monkeypatch.setitem(routes._cfg, "jd_raw_dir", cfg["jd_raw_dir"]) monkeypatch.setitem(routes._cfg, "graph_path", cfg["graph_path"]) monkeypatch.setitem(routes._cfg, "state_db_path", cfg["state_db_path"]) monkeypatch.setitem(routes._cfg, "audit_path", cfg["audit_path"]) monkeypatch.setitem(routes._cfg, "pending_path", cfg["pending_path"]) return cfg def test_jd_import_text_paste(client, tmp_path, monkeypatch): """JSON 文本粘贴：合格记录落 import_*.jsonl（保留来源标记），坏行返回并写 logs，触发重建。""" _import_setup(tmp_path, monkeypatch) lines = [ json.dumps(_jd("n1", "Java开发工程师", "2025-03-01", ["Java"]), ensure_ascii=False), "", # 空行 → 跳过 json.dumps(_jd("n2", "Python开发工程师", "2025-05-01", ["Python"]), ensure_ascii=False), "{不是合法 json", # 坏行 ] r = client.post("/api/jd/import", json={"text": "\n".join(lines)}) assert r.status_code == 200 data = r.json()["data"] assert data["imported"] == 2 and data["skipped"] == 1 assert len(data["bad_lines"]) == 1 and data["bad_lines"][0]["line"] == 4 jd_dir = tmp_path / "jd_raw" imported = jd_dir / data["file"] assert imported.exists() and data["file"].startswith("import_") saved = [json.loads(x) for x in imported.read_text(encoding="utf-8").splitlines()] assert {s["jid"] for s in saved} == {"n1", "n2"} # 坏行写 logs from app.config import ROOT assert (ROOT / "logs" / data["bad_log"]).exists() # 异步重建完成，导入统计随任务可查 t = _wait_task(client, data["task_id"]) assert t["status"] == "done" and t["import"]["imported"] == 2 def test_jd_import_file_upload(client, tmp_path, monkeypatch): """multipart 文件上传（.jsonl）：与文本粘贴同一入库链路。""" _import_setup(tmp_path, monkeypatch) content = "\n".join([ json.dumps(_jd("f1", "Java开发工程师", "2025-01-10", ["Java"]), ensure_ascii=False), json.dumps({"jid": "f2"}), # 缺必需字段 → 坏行 ]).encode("utf-8") r = client.post("/api/jd/import", files={"file": ("extra.jsonl", content, "application/octet-stream")}) assert r.status_code == 200 data = r.json()["data"] assert data["imported"] == 1 and len(data["bad_lines"]) == 1 assert data["bad_lines"][0]["line"] == 2 t = _wait_task(client, data["task_id"]) assert t["status"] == "done" def test_jd_import_csv_and_bad_ext(client, tmp_path, monkeypatch): """CSV 表头映射导入；不支持的扩展名 400。""" _import_setup(tmp_path, monkeypatch) csv_text = ("jid,job_title,company,publish_date,text\n" "c1,Java开发工程师,示例科技,2025-02-01,熟悉Java后端开发\n" "c2,缺日期岗位,示例科技,,熟悉Python\n") # publish_date 缺失 → 坏行 r = client.post("/api/jd/import", files={"file": ("jd.csv", csv_text.encode("utf-8"), "text/csv")}) assert r.status_code == 200 data = r.json()["data"] assert data["imported"] == 1 and len(data["bad_lines"]) == 1 _wait_task(client, data["task_id"]) bad = client.post("/api/jd/import", files={"file": ("jd.exe", b"MZ", "application/octet-stream")}) assert bad.status_code == 400 def test_jd_import_rebuild_only(client, tmp_path, monkeypatch): """不带内容调用保持旧行为：仅触发全量重建。""" _import_setup(tmp_path, monkeypatch) r = client.post("/api/jd/import", json={}) assert r.status_code == 200 data = r.json()["data"] assert "imported" not in data and data["task_id"] t = _wait_task(client, data["task_id"]) assert t["status"] == "done" # ---------- 4. 访问令牌空串清除端到端 ---------- def test_token_clear_end_to_end(client, tmp_path, monkeypatch): """设置令牌 → 鉴权生效 → 空串清除 → 鉴权解除（config.json 同步）。""" import app.config as cfgmod import app.api.routes as routes monkeypatch.setattr(cfgmod, "CONFIG_JSON", tmp_path / "config.json") monkeypatch.setattr(routes, "load_runtime_settings", cfgmod.load_runtime_settings) r = client.post("/api/config", json={"access_token": "tok-xyz"}) assert r.status_code == 200 and r.json()["data"]["has_access_token"] is True assert client.get("/api/jobs").status_code == 401 assert client.get("/api/jobs", headers={"X-Access-Token": "tok-xyz"}).status_code == 200 # 空串清除（前端令牌框 dirty 后留空提交的场景） r2 = client.post("/api/config", json={"access_token": ""}, headers={"X-Access-Token": "tok-xyz"}) assert r2.status_code == 200 assert r2.json()["data"]["has_access_token"] is False assert "access_token" not in routes._cfg assert "access_token" not in cfgmod.load_runtime_settings() assert client.get("/api/jobs").status_code == 200 # 无需令牌即可访问 # ---------- 5. 语义权重相对路径 ---------- def test_relative_weights_path(tmp_path, monkeypatch): """embed_model_path 支持相对路径（相对工程根解析），绝对路径配置自动兼容。""" import app.config as cfgmod from app.match import semantic monkeypatch.setattr(cfgmod, "ROOT", tmp_path) w = tmp_path / "models" / "bge-m3" w.mkdir(parents=True) for f in ["config.json", "model.safetensors", "tokenizer.json"]: (w / f).write_text("{}") st = semantic.status({"embed_model_path": "models/bge-m3"}) assert st["configured"] is True and st["weights_present"] is True # 既有绝对路径配置不受影响 st2 = semantic.status({"embed_model_path": str(w)}) assert st2["weights_present"] is True assert semantic.resolve_weights_path("models/bge-m3") == w assert semantic.resolve_weights_path(str(w)) == w def test_accept_normalizes_candidate_id_single_node(client, tmp_path, monkeypatch): """候选转正归一化：accept 后全图只有 job_<岗位名> 一个节点，边/事件挂在其下。""" import app.api.routes as routes from app.services.overrides import normalize_job_id from app.models import JobDefinition kg = routes._get_kg() job = JobDefinition(job_id="cand_测试岗位X", title="测试岗位X", category="测试", required_skills=["Python"], preferred_skills=["Docker"], scenarios=["教育"], industries=[], expected_years=3, status="active", support_source_ids=["batch_2024.jsonl#1"], support_windows=["2024Q1"]) assert normalize_job_id(job) == "job_测试岗位X" job.job_id = normalize_job_id(job) # 调用方负责归一化（与 routes 一致） from app.services.overrides import apply_accepted_job apply_accepted_job(kg, job) # 全图唯一节点：无 cand_ 残留，REQUIRES 边挂在归一化 id 下 assert "job_测试岗位X" in kg.g and "cand_测试岗位X" not in kg.g assert kg.g.has_edge("job_测试岗位X", "Python") assert not list(kg.g.edges("cand_测试岗位X")) # 清理测试数据 kg.g.remove_node("job_测试岗位X") kg.jobs.pop("job_测试岗位X", None)