真实数据集扩产与被动语料接入
- scripts/expand_real_dataset.py:AutoWeakS 全量 2215 条 + 500 条分层评测集 + 简历扩量(seed=2026 可复现) - pipeline 被动语料机制(passive_sources):真实无日期 JD 参与接入/清洗/抽取验证(2915 条命中率 99.8%),不进图谱/演化/发现 - 新岗位发现第四指标'差异度'与双年窗口增长率口径 - 评测新增 real_jd_500 维度(99.91%);测试会话隔离 jd_extra_dirs 防止本机配置拖慢套件
This commit is contained in:
@@ -129,6 +129,25 @@ def run_pipeline(config: dict | None = None) -> dict:
|
||||
stats["stages"]["extract"] = {"extracted": len(extractions),
|
||||
"extractor": extractor.name}
|
||||
|
||||
# 3.5 被动语料分流:真实但未标日期的数据集(如 autoweaks,统一标记 2019-06-01),
|
||||
# 参与接入/清洗/抽取验证(检验真实语料上的抽取鲁棒性与词表命中率),
|
||||
# 但不进图谱/演化/新岗位发现——避免统一日期造成 2019 虚假演化尖峰、
|
||||
# 以及"单年份→增长率视为达标"导致的待审核候选洪泛。
|
||||
passive = {s.strip() for s in str(cfg.get("passive_sources") or "").split(",") if s.strip()}
|
||||
if passive:
|
||||
passive_records = [r for r in records if r.source in passive]
|
||||
if passive_records:
|
||||
hit = sum(1 for r in passive_records
|
||||
if extractions.get(r.jid) and extractions[r.jid].required_skills)
|
||||
stats["stages"]["real_corpus"] = {
|
||||
"records": len(passive_records), "sources": sorted(passive),
|
||||
"skill_hit_rate": round(hit / max(1, len(passive_records)), 4),
|
||||
}
|
||||
logger.info("被动语料 %d 条(%s):词表技能命中率 %.1f%%,不进图谱/演化/发现",
|
||||
len(passive_records), "/".join(sorted(passive)),
|
||||
100 * hit / max(1, len(passive_records)))
|
||||
records = [r for r in records if r.source not in passive]
|
||||
|
||||
# 4. 图谱写入:既有岗位定义 + 观察流
|
||||
first_years: dict[str, int] = {}
|
||||
for r in records:
|
||||
|
||||
Reference in New Issue
Block a user