Compare commits

..
16 Commits
Author SHA1 Message Date
RRRosmontis c6908b5441 fix(bilibili): hk 出口 WAF 412 改用官方 API 直取 + 重建文档补齐
- bilibili.py: view -> playurl -> 媒体段 的 API 直取主路径,绕开页面
  WAF/JS 挑战;yt-dlp 保留为后备;cookie 仅后备路径需要
- docs: 迁移步骤补 /var/cache/nginx 与 ffmpeg 两个重建坑位
- 实测: 公网 200/1.3MB/6.5s
2026-10-22 17:28:12 +08:00
RRRosmontis 4dee7a3c0c chore: 仓库与线上一致性补齐
- pixiv.py: 补录线上 2026-09-17 版(图源错误码处理/fetch_illust_url 返回 api_status)
- nginx/00-limits.conf: 限流 zone 定义入库(站点 conf 引用, 缺失将导致 nginx -t 失败)
- nginx/00-websocket.conf: connection_upgrade map 入库(同上)
- docs: 迁移第 4 步改为套用 nginx/ 三件套; 记录一致性核对结果
2026-10-21 18:56:58 +08:00
RRRosmontis 67aa9be3da feat(nginx): openapi 全站启用按 IP 限流(轻防滥用档)
- 00-limits.conf 新增 limit_req_zone perip_req 10r/s + 429 状态码
  (zone 定义属服务器配置, 不入库)
- 站点 conf 443 server 块挂 limit_req burst=20 nodelay + limit_conn perip 10
- 域名直连无 CDN, binary_remote_addr 即真实 IP; 后端全绑 127.0.0.1 无死角
- 验证: 30 并发部分 429, 桶恢复后正常
2026-09-23 15:05:14 +08:00
RRRosmontis cba14eae25 docs: 部署与运维补充 attempts 逐次记录说明 2026-09-23 12:03:46 +08:00
RRRosmontis c3f8d64f0e feat(dufe): auto 模式 attempts 改为逐次完整记录
- entry.attempts 由计数改为列表: 每次尝试 {n, success, message}
- relogin 后立即重试的那次带 "relogin": true 标记
- 成功/失败(重试用尽)两种情况都完整携带全部尝试记录
2026-09-23 12:03:30 +08:00
RRRosmontis 9b9c501fa9 feat(dufe): 新增 mode=auto 自动签到,非签到码逻辑以 bot 为基准同步
- mode=auto: 通知列表(getMessageAll+busiMessage/list)发现签到活动
  (bot ginkgo 同款过滤, 按 signInId 去重) → 逐个自动签到
- 连续重试: 首次+最多5次间隔3s (对齐 bot MAX_SIGNIN_ATTEMPTS/SIGNIN_BACKOFF),
  token 失效重登录一次立即重试; 限流中断本轮; 结果聚合 data.results 列表
- 基准同步 (bot ginkgo): 移植限流防护体系(check_response/is_rate_limited/
  safe_error/_LoginErrors/_request), parse_course_url courseId 放宽 [0-9A-Za-z]+,
  commit_sign_in 错误包装, fetch_sign_in_info 增加 check_response 与
  sess/token 注入 (auto 模式请求内复用登录态)
- 验证: 免凭据单测 + 假凭据 401 线路测试 + bj 真账密只读联调
  (notices=20/busi=18/活动12 类型解析正确)
2026-09-23 11:53:12 +08:00
RRRosmontis 9ecbbbc6d8 feat(jm): 新增 JM 漫画下载打包服务(9004,移植自 qqbot jm 模块)
- source/jm/jm.py:aiohttp + jmcomic,GET /jm/get_comic?id=xxx[&range=起-止]
  下载至 cache/jm 隐藏暂存目录 → webp 转 jpg 打包 ZIP → HTTP 直返 → 600s 延时删除
  单次上限 30 章;per-album 锁防并发;cache_sweep 兜底清理;/jm/cache_info 运维接口
- source/main.py:聚合层注册 /jm/get_comic(新增 JM_BACKEND,复用 proxy())
- systemd/jm.service:独立系统服务(Restart=always,日志 /var/log/jm.log)
- README / 部署与运维文档同步更新
2026-09-23 10:58:48 +08:00
RRRosmontis 9ba194ae97 fix(dufe): 修复 base64 %3D 容错与 commit 判定 fail-open
ginkgo_sign_in.py:
- 顶部新增 import urllib.parse
- _b64_decode: 先 urllib.parse.unquote 再做 空格->+ 替换
  (iOS 快捷指令会把 base64 的 = 编码成 %3D,旧实现直接 binascii.Error;
    顺序必须是 unquote 在 空格->+ 之前,否则 %20 会被误当成被 form-urlencoded 吃掉的 +)
- commit 判定改为 fail-closed:要求 commit_info 是 dict 且 success is True,
  否则一律返回 msg="sign-in failed: <原因>";
  do_commit=false 时不检查 commit 字段,msg 仍为 "ok"
- reason 取值修正:非 dict 真值(字符串/数字)不再触发 AttributeError 500,
  统一回落到 "no commit result returned"
- parse_course_url 报错文案改为
  "URL is not a sign-in page (missing /signin/<signInId>/<type>): <url>"

文档:README 版本要点与 docs/部署与运维.md 变更记录同步
2026-09-13 11:08:40 +08:00
RRRosmontis a436bf58b7 docs: 补充 dufe 服务(端口 9005、端点 /dufe/ginkgostu_sign_in、单元与变更记录)
- README: 架构图/端口表/目录结构/部署启用列表/验证命令/版本要点
- docs/部署与运维.md: 对外入口表、进程与端口表、目录结构、日志、变更记录
2026-09-13 03:17:19 +08:00
RRRosmontis 371d3a3624 feat(dufe): 新增银杏签到码查询服务,并在聚合层注册端点
- source/dufe/ginkgostu_sign_in/ginkgo_sign_in.py: 签到码查询服务
  (Flask + waitress 16 线程,监听 127.0.0.1:9005,端点 GET|POST /dufe/ginkgostu_sign_in)
- systemd/ginkgo_sign_in.service: 常驻服务单元,Restart=always,日志 /var/log/ginkgo_sign_in.log
- source/main.py: 新增 DUFE_BACKEND 与 handle_dufe_sign_in,
  并新增 proxy_forward()(保留请求方法与请求体,支持 POST);
  原有 proxy()(固定 GET,供 pixiv/bilibili 用)未做任何改动
2026-09-13 03:17:19 +08:00
RRRosmontis ca26977233 docs: 更正服务器副本的说明(备份说明.txt 为同内容纯文本副本,非历史版本) 2026-09-13 02:27:13 +08:00
RRRosmontis 5a43bce45d fix: 补齐 99543c9 遗漏的内容改动(该提交的 git add 因 pathspec 报错未生效)
上一个 refactor 提交只落到 main.py 的重命名,端口与路径的实际改动漏掉了,此提交补齐:
- source/main.py: PIXIV_BACKEND 9088->9007, BILIBILI_BACKEND 19990->9006, port 9888->9008
- nginx/openapi.rrrosmontis.icu.conf: 3 处 proxy_pass -> 127.0.0.1:9008
- systemd/openapi.service: WorkingDirectory/ExecStart 同步为 source/
2026-09-13 02:26:54 +08:00
RRRosmontis 1b2071d78b docs: 更新 README 与线上部署运维说明,新增 .gitignore
- README: 新架构图/端口表/目录结构/双缓存机制/waitress/部署与验证步骤/版本要点
- docs/部署与运维.md: 线上服务器的实际路径、缓存策略、迁移 8 步、日志与变更记录
- .gitignore: 忽略 source/cache/、tokens.json、cookies.txt、__pycache__
2026-09-13 02:26:35 +08:00
RRRosmontis 212adb1e25 perf(pixiv): 改用 waitress 线程池,缓存路径改为相对路径
- IMAGE_DIR 由硬编码 /home/pixiv/images 改为 (Path(__file__).parent/'../cache/pixiv/images').resolve()
  (即 source/cache/pixiv/images),mkdir 补 parents=True
- systemd/pixiv.service: ExecStart 改为 waitress-serve --host=127.0.0.1 --port=9007 --threads=8 pixiv:app
  替代 Flask 开发服务器(thread-per-request 无上限,突发流量下线程爆炸)
- __main__ 兜底加 threaded=True,手动运行时不串行
2026-09-13 02:26:34 +08:00
RRRosmontis 5400dc0a86 feat(bilibili): 新增 BV 本地缓存,修复视频暂存目录泄漏
- 新增 CACHE_DIR(../cache/bilibili),以 BV 号为键缓存 {BV}.mp4,命中即 FileResponse、不再调用 yt-dlp
- 未命中时下载到缓存目录内的隐藏暂存目录 .dl-XXXX,成功后 os.replace 原子落盘
- finally 中 shutil.rmtree(tmpdir):原实现只清理 cookie 临时文件,导致每次取视频都把完整视频
  永久留在 /tmp(无上限磁盘泄漏)
- 淘汰策略:TTL 7 天未访问过期 + 总量超过 2 GiB 按 mtime 从最旧开始删除
- 新增运维接口 GET /bilibili/cache_info(缓存目录/上限/文件数/总量/剩余磁盘)
- 缓存 HIT/MISS/STORE 日志加 flush=True,实时可见
2026-09-13 02:26:34 +08:00
RRRosmontis 99543c9935 refactor: 主入口 main.py 上移到 source/,端口重排为 9008/9007/9006
- main.py: source/openapi/main.py -> source/main.py
- 端口: 聚合层 9888->9008, pixiv 后端 9088->9007, bilibili 后端 19990->9006
- nginx/openapi.rrrosmontis.icu.conf: 三处 proxy_pass 指向 127.0.0.1:9008
- systemd/openapi.service: WorkingDirectory/ExecStart 同步为 source/
2026-09-13 02:26:34 +08:00
16 changed files with 2206 additions and 151 deletions
+14
View File
@@ -0,0 +1,14 @@
# 运行时缓存(不入库)
source/cache/
# 凭据与密钥
source/pixiv/tokens.json
source/bilibili/cookies.txt
# Python
__pycache__/
*.pyc
# 杂项
*.swp
.DS_Store
+139 -17
View File
@@ -2,34 +2,156 @@
统一接口服务。线上入口:`https://openapi.rrrosmontis.icu` 统一接口服务。线上入口:`https://openapi.rrrosmontis.icu`
聚合层把 **pixiv 图片服务**、**bilibili 视频服务**、**dufe 银杏签到码查询服务**与 **jm 漫画下载服务**收敛到单一对外入口。五个后端进程全部只监听 `127.0.0.1`,对外仅由 nginx 暴露 80/443——即"一个入口、零暴露端口"。
## 架构
```
公网 443/80 nginx (server_name openapi.rrrosmontis.icu)
├── /pixiv/get_image?id=xxx -> 127.0.0.1:9008 main.py 聚合层 -> 127.0.0.1:9007 pixiv
├── /bilibili/get_video?bv=xxx -> 127.0.0.1:9008 main.py 聚合层 -> 127.0.0.1:9006 bilibili
├── /dufe/ginkgostu_sign_in -> 127.0.0.1:9008 main.py 聚合层 -> 127.0.0.1:9005 dufe 签到码查询(GET|POST)
├── /jm/get_comic?id=xxx -> 127.0.0.1:9008 main.py 聚合层 -> 127.0.0.1:9004 jm
├── /img-master/ /img-original/ -> nginx 直接反代 i.pximg.net(无本地后端)
└── / -> 127.0.0.1:9008(302 跳转文章页)
```
## 端口与进程
| 服务 | 监听 | 入口源码 | 说明 |
|---|---|---|---|
| openapi | `127.0.0.1:9008` | `source/main.py`(aiohttp) | 聚合层,纯转发,**唯一入口** |
| pixiv | `127.0.0.1:9007` | `source/pixiv/pixiv.py`(Flask + waitress) | 图片代理 |
| bilibili | `127.0.0.1:9006` | `source/bilibili/bilibili.py`(aiohttp,官方 API 直取 + yt-dlp 后备) | 视频代理 |
| dufe 签到 | `127.0.0.1:9005` | `source/dufe/ginkgostu_sign_in/ginkgo_sign_in.py`(Flask + waitress 16 线程) | 银杏(DUFE)签到码查询,`GET\|POST /dufe/ginkgostu_sign_in`;`&mode=auto` 时拉取通知列表自动签到(与 bot ginkgo 同逻辑,连续重试,结果列表聚合返回) |
| jm | `127.0.0.1:9004` | `source/jm/jm.py`(aiohttp + jmcomic) | 漫画下载打包(移植自 qqbot jm 模块),`GET /jm/get_comic?id=xxx[&range=起-止]` |
| pixiv-token-refresh | — | `source/pixiv/refresh_token.py` | token 刷新守护(每 30–60 分钟随机) |
| bilibili-refresh | — | `source/bilibili/refresh_cookie.py` | cookie 刷新守护(每 50–70 分钟随机) |
> 历史端口 `9888 / 9088 / 19990` 已于 2026-09-12 废弃,全部改为上述 `900x`。
> pixiv 早期误绑 `0.0.0.0:9088` 对公网裸奔(绕过 nginx 的日志与限流),现已收口为 `127.0.0.1`。
## 结构 ## 结构
``` ```
source/ source/
├── openapi/main.py # 聚合层 (127.0.0.1:9888) /pixiv/get_image /bilibili/get_video ├── main.py # 聚合层 (127.0.0.1:9008),五条路由:/ /pixiv/get_image /bilibili/get_video /dufe/ginkgostu_sign_in /jm/get_comic
├── pixiv/ ├── pixiv/
│ ├── pixiv.py # Pixiv 图片服务 (9088) │ ├── pixiv.py # Pixiv 图片服务 (127.0.0.1:9007)
│ ├── refresh_token.py # Pixiv OAuth token 刷新守护 │ ├── refresh_token.py # Pixiv OAuth token 刷新守护
│ └── tokens.template.json # Pixiv token 模板 │ └── tokens.template.json # token 模板(实际 tokens.json 不入库)
└── bilibili/ ├── bilibili/
├── bilibili.py # bilibili 视频服务 (19990) │ ├── bilibili.py # bilibili 视频服务 (127.0.0.1:9006)
├── refresh_cookie.py # cookie 定时刷新守护 │ ├── refresh_cookie.py # cookie 定时刷新守护
├── qr_login.py # 扫码登录获取 cookie │ ├── qr_login.py # 扫码登录获取 cookie
└── qr_login_fast.py │ └── qr_login_fast.py
├── dufe/
│ └── ginkgostu_sign_in/
│ └── ginkgo_sign_in.py # 银杏(DUFE)签到码查询 (127.0.0.1:9005, GET|POST)
├── jm/
│ └── jm.py # JM 漫画下载打包服务 (127.0.0.1:9004)
└── cache/ # 运行时生成,不入库
├── pixiv/images # 图片缓存
├── bilibili/ # 视频缓存
└── jm/ # jm 临时 ZIP(TTL 10 分钟,返回后删除)
nginx/ nginx/
└── openapi.rrrosmontis.icu.conf # nginx 统一入口(/pixiv/ /bilibili/ /img-master/ /img-original/ 图片反代) ├── openapi.rrrosmontis.icu.conf # 统一入口(/pixiv/ /bilibili/ /img-master/ /img-original/,其余路径由 location / 兜底至聚合层;含按 IP 限流)
├── 00-limits.conf # 限流 zone(limit_req_zone rate=10r/s、limit_conn_zone、429)——缺失会导致 nginx -t 失败
└── 00-websocket.conf # $connection_upgrade map 定义——缺失会导致 nginx -t 失败
systemd/ systemd/
├── openapi.service # 聚合层 ├── openapi.service # 聚合层
├── pixiv.service # Pixiv 图片服务 ├── pixiv.service # Pixiv 图片服务(waitress --threads=8)
├── bilibili.service # bilibili 视频服务 ├── bilibili.service # bilibili 视频服务
├── bilibili-refresh.service # cookie 刷新守护(每 50–70 分钟随机刷新) ├── bilibili-refresh.service # cookie 刷新守护
└── pixiv-token-refresh.service # Pixiv token 刷新守护(每 30–60 分钟随机刷新) ├── pixiv-token-refresh.service # Pixiv token 刷新守护
├── ginkgo_sign_in.service # 银杏(DUFE)签到码查询服务
└── jm.service # JM 漫画下载打包服务
docs/
└── 部署与运维.md # 线上服务器的路径、缓存、迁移步骤与变更记录
``` ```
## 缓存机制
**pixiv(图片)**:`check_cache()` 以 glob 命中即直接回文件;未命中按 `官方 App API → pixiv.re → pixiv.cat` 三级回退抓取,落盘用 `.tmp` + `os.replace` 原子写。
缓存目录不再硬编码,由模块位置推导:
```python
IMAGE_DIR = (Path(__file__).resolve().parent / "../cache/pixiv/images").resolve()
IMAGE_DIR.mkdir(parents=True, exist_ok=True)
```
即 `source/cache/pixiv/images`;线上常用 `/home/images` 软链指向它,便于访问。
**bilibili(视频)**:以 **BV 号为键**缓存到 `source/cache/bilibili/{BV}.mp4`
- 命中即直接 `FileResponse`,**不再下载**
- 未命中时优先走 **官方 API 直取**(view → playurl → 媒体段;2026-10-22 起 hk 出口被页面 WAF 挑战,yt-dlp 页面提取失效),失败再回退 yt-dlp
- 未命中才下载,且先落到缓存目录内的隐藏暂存目录 `.dl-XXXX`,成功后 `os.replace` 原子落盘
- **无论成功 / 失败 / 超时都会 `shutil.rmtree(tmpdir)`** —— 早期实现只清理 cookie 临时文件、从不清理视频暂存目录,导致每次取视频都把完整视频永久留在 `/tmp`(无上限磁盘泄漏),已修复
- 淘汰策略:单文件 `CACHE_TTL_SECONDS`(默认 7 天)未访问即过期;总量超过 `CACHE_MAX_BYTES`(默认 2 GiB)时按 mtime 从最旧开始删除
- 命中/落盘会在日志打印 `cache HIT / MISS / STORE`(已 `flush=True`,实时可见)
运维接口(只绑本机):
```bash
curl -s 127.0.0.1:9006/bilibili/cache_info
# {cache_dir, cache_max_bytes, cache_ttl_seconds, count, total_bytes, free_bytes, files[]}
```
**jm(漫画)**:**非持久缓存**,与 bilibili 不同——ZIP 仅作临时产物
- 请求 `id`(+ 可选 `range=起-止`,单次 ≤30 章)→ jmcomic 下载到 `cache/jm/` 内隐藏暂存目录 `.dl-<id>-XXXX` → webp 统一转 jpg 打包 ZIP(`{标题}_{id}_ch{起}-{止}.zip`)→ HTTP 返回
- 打包完成即 `rmtree` 图片暂存目录;ZIP 在返回后 `ZIP_TTL_SECONDS`(默认 600 秒)延时删除
- TTL 窗口内重复请求同一 ZIP 直接复用并重新计时;每次请求附带 `_cache_sweep()` 兜底清理过期 ZIP(防进程重启残留)
- 同一本子串行处理(per-album asyncio.Lock),并发请求同本返回 409
- 运维接口:`curl -s 127.0.0.1:9004/jm/cache_info`
## 部署 ## 部署
1. token:复制 `source/pixiv/tokens.template.json` 为 `source/pixiv/tokens.json`,填入实际 pixiv token 1. **Python 环境**:各服务共用一个环境,需含 `flask`、`waitress`、`aiohttp`、`httpx`、`yt-dlp`、`jmcomic`、`pillow`;系统包另需 **ffmpeg**(yt-dlp 合流用,API 直取路径不需要)
2. bilibili cookie:放置 `source/bilibili/cookies.txt`(`name=value` 格式),或运行 `qr_login.py` 扫码生成 2. **凭据**:
3. 服务:将 `systemd/*.service` 安装到 `/etc/systemd/system/`,启用并启动 openapi / pixiv / bilibili / bilibili-refresh / pixiv-token-refresh(`systemctl enable --now`) - 复制 `source/pixiv/tokens.template.json` 为 `source/pixiv/tokens.json`(`chmod 600`)并填入实际 pixiv token
4. nginx:将 `nginx/openapi.rrrosmontis.icu.conf` 放入 `/etc/nginx/conf.d/`,`nginx -t && systemctl reload nginx` - 放置 `source/bilibili/cookies.txt`(`name=value` 格式),或运行 `qr_login.py` 扫码生成
5. 定时任务:无 cron/timer——bilibili-refresh(每 50–70 分钟刷 cookie)与 pixiv-token-refresh(每 30–60 分钟刷 Pixiv OAuth token,轮换后的 refresh_token 原子落盘)均为常驻守护,内部循环随机间隔,systemd `Restart=always` 兜底 - jm 无需凭据,但要求部署机可直连源站(jmcomic 会自动更新 API 域名)
3. **服务**:将 `systemd/*.service` 安装到 `/etc/systemd/system/`,然后
`systemctl daemon-reload && systemctl enable --now openapi pixiv bilibili ginkgo_sign_in jm bilibili-refresh pixiv-token-refresh`
4. **nginx**:将 `nginx/openapi.rrrosmontis.icu.conf` 放入 `/etc/nginx/conf.d/`(需存在 `$connection_upgrade` 的 map 定义),`nginx -t && systemctl reload nginx`
5. **软链**(可选):`ln -s <部署根>/source/cache/pixiv/images /home/images`
6. **定时任务**:无 cron/timer——两个刷新守护均为常驻循环(内部随机间隔),systemd `Restart=always` 兜底
## 验证
```bash
curl -k "https://openapi.rrrosmontis.icu/pixiv/get_image?id=147113746"
curl -k "https://openapi.rrrosmontis.icu/bilibili/get_video?bv=BV16a41157Q4"
curl -kI "https://openapi.rrrosmontis.icu/bilibili/get_video?bv=BV16a41157Q4" # 只看头部,省流量
curl -k "https://openapi.rrrosmontis.icu/dufe/ginkgostu_sign_in" # 无参应返回 400 JSON
curl -k "https://openapi.rrrosmontis.icu/jm/get_comic?id=350234" -o jm.zip # 返回 ZIP
curl -s 127.0.0.1:9006/bilibili/cache_info
curl -s 127.0.0.1:9004/jm/cache_info
```
端口暴露自检(五个后端都应"不可达",仅 443/80 通):
```bash
for p in 9004 9005 9006 9007 9008; do timeout 3 bash -c "cat < /dev/null > /dev/tcp/<host>/$p" 2>/dev/null \
&& echo "$p 可连通(异常)" || echo "$p 不可达(正常)"; done
```
## 版本要点
- **2026-09-23**
- dufe 签到服务新增 **`mode=auto`**:拉取通知列表(getMessageAll + busiMessage/list)→ 按 bot ginkgo 同款规则发现签到活动(互动消息 + 文本含"签到" + 可解析课程链接,按 signInId 去重)→ 逐个自动签到;**连续重试**(首次 + 最多 5 次,间隔 3s,对齐 bot `MAX_SIGNIN_ATTEMPTS=6 / SIGNIN_BACKOFF`;token 疑似失效时重登录一次立即重试,对齐 bot refresh 语义);**所有结果聚合为 `data.results` 列表**返回(每活动含 success/message/attempts/title/signInCode 等)
- 非签到码签到逻辑与 bot ginkgo 比对后以 bot 为基准同步:移植限流防护体系(`check_response` / `is_rate_limited` / `safe_error` / `_LoginErrors`)、`parse_course_url` 的 courseId 放宽至 `[0-9A-Za-z]+`、`commit_sign_in` 的错误包装、`fetch_sign_in_info` 的 check_response 与外部登录态注入(`sess`/`token` 参数,auto 模式请求内复用同一登录态)
- **2026-09-23**(早些):新增 **jm 漫画下载打包服务**(`source/jm/jm.py`,aiohttp + jmcomic,`127.0.0.1:9004`,端点 `GET /jm/get_comic?id=xxx[&range=起-止]`),移植自 qqbot 的 jm 模块(去掉 QQ/代理轮换/上传 cf 逻辑,改为 HTTP 直返 ZIP;hk 可直连源站无需代理)。ZIP 为临时产物:返回后 600 秒延时删除,TTL 窗口内同名请求直接复用。聚合层 `main.py` 注册路由(新增 `JM_BACKEND`,复用既有 `proxy()`),nginx 无需改动(`location /` 兜底)
- **2026-09-13**:新增 **dufe 银杏签到码查询服务**(`source/dufe/ginkgostu_sign_in/ginkgo_sign_in.py`,Flask + waitress 16 线程,`127.0.0.1:9005`,端点 `GET|POST /dufe/ginkgostu_sign_in`),并在聚合层 `main.py` 注册该路由——为此新增 `proxy_forward()`(保留请求方法与请求体,原 `proxy()` 保持固定 GET 不变,零改动既有行为)。同期修复该服务两处缺陷:base64 参数补 `urllib.parse.unquote`(兼容 iOS 快捷指令的 `%3D` 编码),`commit` 判定改为 fail-closed(须 `commit.success is True` 才算签到成功)
- **2026-09-12**
- 端口重排 `9888/9088/19990 → 9008/9007/9006`,全部仅绑 `127.0.0.1`;pixiv 由 `0.0.0.0:9088` 收口
- 主入口 `main.py` 从 `source/openapi/` 上移到 `source/`,systemd `WorkingDirectory` 同步为 `source/`
- pixiv 缓存路径由硬编码 `/home/pixiv/images` 改为相对 `../cache/pixiv/images`
- pixiv 改由 **waitress 线程池**承载(`--threads=8`),替代 Flask 开发服务器(thread-per-request 无上限)
- bilibili 新增 **BV 真缓存**(`../cache/bilibili`,2 GiB / 7 天淘汰),并修复视频暂存目录永不清理导致的 `/tmp` 泄漏
- 新增 `/bilibili/cache_info` 运维接口;缓存日志加 `flush=True`
- **2026-09-02**:新增 pixiv token 刷新守护;补 `/img-original/` 反代(App API 原图路径)
- **2026-07-31**:三服务迁移落位(Python 环境合并为单一 `general`)
+200
View File
@@ -0,0 +1,200 @@
# 线上部署与运维(hk)
> 记录线上服务器 `hk`(`103.24.218.119:10526`)的实际路径、缓存策略与迁移步骤。
> 架构与代码结构见根目录 [README](../README.md)。
## 对外入口
统一入口:`https://openapi.rrrosmontis.icu`(nginx 监听 80/443)
| 路径 | 链路 |
|---|---|
| `/pixiv/get_image?id=xxx` | `127.0.0.1:9008`(main.py 聚合层)→ `127.0.0.1:9007`(pixiv) |
| `/bilibili/get_video?bv=xxx` | `127.0.0.1:9008`(main.py 聚合层)→ `127.0.0.1:9006`(bilibili) |
| `/dufe/ginkgostu_sign_in` | `127.0.0.1:9008`(main.py 聚合层)→ `127.0.0.1:9005`(dufe 签到码查询,支持 GET\|POST;`&mode=auto` 自动签到) |
| `/jm/get_comic?id=xxx[&range=起-止]` | `127.0.0.1:9008`(main.py 聚合层)→ `127.0.0.1:9004`(jm 漫画下载打包) |
| `/img-master/`、`/img-original/` | nginx 直接反代 `i.pximg.net`(无本地后端) |
| `/` | `127.0.0.1:9008`(302 跳转文章页) |
- 其余在服站点:`sqy.bu4.ren`、`xxx.aeolianchi.me`
- 已删除的 vhost:`kim1.work`、`zxx.aeolianchi.me`(2026-09-12)
- 历史域名 `api.bu4.ren` / `pixiv.bu4.ren` / `apixiv.bu4.ren` 已合并进 `openapi.rrrosmontis.icu`
## 进程与端口
五个服务**全部只绑 `127.0.0.1`**,对外仅 nginx 的 443/80。
| systemd 单元 | 源码 | 监听 | 说明 |
|---|---|---|---|
| `openapi.service` | `source/main.py` | `:9008` | 聚合层(aiohttp,纯转发,唯一入口) |
| `pixiv.service` | `source/pixiv/pixiv.py` | `:9007` | Flask,由 waitress(8 线程)承载 |
| `bilibili.service` | `source/bilibili/bilibili.py` | `:9006` | aiohttp,官方 API 直取 + yt-dlp 后备 |
| `ginkgo_sign_in.service` | `source/dufe/ginkgostu_sign_in/ginkgo_sign_in.py` | `:9005` | 银杏(DUFE)签到码查询,Flask + waitress(16 线程),`GET\|POST` |
| `jm.service` | `source/jm/jm.py` | `:9004` | JM 漫画下载打包,aiohttp + jmcomic |
| `pixiv-token-refresh.service` | `source/pixiv/refresh_token.py` | — | 常驻刷新 Pixiv OAuth token |
| `bilibili-refresh.service` | `source/bilibili/refresh_cookie.py` | — | 常驻刷新 bilibili cookie |
## 目录结构
```
/home/openapi/
├── source/
│ ├── main.py 聚合层源码(2026-09-12 从 source/openapi/ 上移至此)
│ ├── pixiv/
│ │ ├── pixiv.py 图像代理(Flask)
│ │ ├── refresh_token.py token 刷新守护
│ │ └── tokens.json Pixiv OAuth token(敏感,权限 600,不入库)
│ ├── bilibili/
│ │ ├── bilibili.py 视频代理(aiohttp,API 直取 + yt-dlp 后备)
│ │ ├── cookies.txt bilibili cookie(敏感,不入库)
│ │ ├── refresh_cookie.py cookie 刷新守护
│ │ └── qr_login*.py 扫码登录工具(cookie 失效时人工重登用)
│ ├── dufe/
│ │ └── ginkgostu_sign_in/
│ │ └── ginkgo_sign_in.py 银杏(DUFE)签到码查询服务(127.0.0.1:9005,GET|POST)
│ ├── jm/
│ │ └── jm.py JM 漫画下载打包服务(127.0.0.1:9004)
│ └── cache/ 运行时缓存(不入库)
│ ├── pixiv/images pixiv 图片缓存(代码中为 ../cache/pixiv/images)
│ ├── bilibili/ bilibili 视频缓存(上限 2 GiB / TTL 7 天)
│ └── jm/ jm 临时 ZIP(TTL 10 分钟,返回后删除)
├── systemd/ 7 个单元文件的归档副本(与 /etc/systemd/system/ 保持一致)
├── nginx/ 宝塔导出的历史存档,非生效路径
├── cert/ SSL 证书
├── env/ Python 环境(general + env_general.tar.gz 整包)
└── 备份说明.txt 本文档在服务器上的纯文本副本(随本文档同步维护)
/home/images -> /home/openapi/source/cache/pixiv/images 便捷软链
```
## 缓存机制
**pixiv(图片)**:`check_cache()` 以 glob 命中即直接回文件;未命中按 `官方 App API → pixiv.re → pixiv.cat` 三级回退抓取,落盘用 `.tmp` + `os.replace` 原子写。
**bilibili(视频)**:以 BV 号为键缓存到 `source/cache/bilibili/{BV}.mp4`
- 命中即复用、不再下载
- 未命中时优先走 **官方 API 直取**(view → playurl → 媒体段,不受页面 WAF 影响);失败回退 yt-dlp,均写入缓存目录内的隐藏暂存目录 `.dl-XXXX`,成功后原子 rename 落盘
- 无论成功 / 失败 / 超时都 `rmtree` 该暂存目录
(2026-09-12 修复:原实现从不清理 tmpdir,每次取视频都会把完整视频永久留在 `/tmp`,属无上限磁盘泄漏)
- 淘汰策略:单文件 7 天未访问过期 + 总量超过 2 GiB 时按 mtime 从最旧开始删
- 运维接口:`curl -s 127.0.0.1:9006/bilibili/cache_info`
**jm(漫画)**:**非持久临时产物**,移植自 qqbot 的 jm 模块(bj `C:/call/bot/utility/jm.py`)
- `GET /jm/get_comic?id=xxx[&range=起-止]`:jmcomic 下载 → `cache/jm/` 内隐藏暂存目录 `.dl-<id>-XXXX` → webp 统一转 jpg 打包 ZIP(`{标题}_{id}_ch{起}-{止}.zip`)→ HTTP 直返
- 单次上限 30 章(与 qqbot 一致,同时避免超过聚合层 300s 超时);总章节 ≤30 可省略 range 自动全量
- 打包完成即 `rmtree` 图片暂存目录;ZIP 在返回后 600 秒(`ZIP_TTL_SECONDS`)延时删除
- TTL 窗口内同名 ZIP 重复请求直接复用并 touch 重新计时;每次请求附带 `_cache_sweep()` 兜底清理过期 ZIP(防进程重启残留)
- 同一本子串行处理(per-album asyncio.Lock),并发请求同本返回 409
- hk 直连源站,无需代理(jmcomic 自动更新 API 域名);qqbot 侧的代理轮换 / 上传 cf / QQ 回复逻辑均未带入
- 运维接口:`curl -s 127.0.0.1:9004/jm/cache_info`
## nginx
| 文件 | 作用 |
|---|---|
| `/etc/nginx/conf.d/openapi.rrrosmontis.icu.conf` | 统一入口站点(`proxy_pass` → `127.0.0.1:9008`;含按 IP 限流,见下) |
| `/etc/nginx/conf.d/00-limits.conf` | 限流 zone 定义(`perip` 连接数 / `perip_req` 请求速率)+ 429 状态码 |
| `/etc/nginx/conf.d/00-websocket.conf` | `$connection_upgrade` map 定义(**必须存在**) |
| `/home/openapi/nginx/**` | 宝塔导出的历史存档,非生效路径 |
**按 IP 限流(2026-09-23 启用,轻防滥用档)**:域名 DNS 直连本机(无 CDN),`$binary_remote_addr` 即真实客户端 IP,后端全绑 `127.0.0.1` 使 nginx 成为唯一入口、限流无死角。参数:`limit_req_zone ... rate=10r/s` + `burst=20 nodelay`,`limit_conn perip 10`;超出返回 **429**。验证:30 并发请求中部分被拒(429),桶恢复后正常。调整档位改 `/etc/nginx/conf.d/00-limits.conf`(速率)与站点 conf(burst/并发)后 `nginx -t && systemctl reload nginx`。
## Python 环境
- `/home/openapi/env/general` —— 五服务共用(flask / waitress / aiohttp / httpx / yt-dlp / jmcomic / pillow)
- `/home/openapi/env/env_general.tar.gz` —— 环境整包(迁移用)
- 注意:pixiv 现由 **waitress** 承载(`pip install waitress`,纯 Python 线程池),不再使用 Flask 开发服务器
(开发服务器是 thread-per-request 无上限,受控线程池更适合长期运行)
## 迁移到新服务器
1. **环境**:解压 `env/env_general.tar.gz`,直接使用其 `bin/python`(依赖齐全,含 waitress);若整包过旧需补 `pip install jmcomic`;另装系统包 `ffmpeg`(yt-dlp 合流)
2. **源码**:恢复 `source/{main.py,pixiv/,bilibili/,dufe/,jm/}` 到 `/home/openapi/source/`
3. **systemd**:恢复 `systemd/*.service` 到 `/etc/systemd/system/`,`daemon-reload` + `enable` + `start`
4. **nginx**:先建缓存目录 `mkdir -p /var/cache/nginx`(站点 conf 的 `proxy_cache_path` 需要父目录存在,否则 `nginx -t` 报 mkdir 失败),再套用 `nginx/` 下三个文件到 `/etc/nginx/conf.d/`:`openapi.rrrosmontis.icu.conf` + `00-websocket.conf`(`$connection_upgrade` map,**必须**)+ `00-limits.conf`(限流 zone,**必须**,否则站点 conf 的 `limit_req zone=perip_req` 会使 `nginx -t` 失败),再 `nginx -t && systemctl reload nginx`
5. **SSL**:恢复 `cert/<域名>/` 到 `/etc/nginx/cert/<域名>/`
6. **凭据**:放回 `source/pixiv/tokens.json`(600) 与 `source/bilibili/cookies.txt`
7. **软链**:`ln -s /home/openapi/source/cache/pixiv/images /home/images`
8. **验证**:
```bash
curl -k "https://openapi.rrrosmontis.icu/pixiv/get_image?id=xxx"
curl -k "https://openapi.rrrosmontis.icu/bilibili/get_video?bv=BV16a41157Q4"
curl -k "https://openapi.rrrosmontis.icu/dufe/ginkgostu_sign_in" # 无参应返回 400 JSON
curl -k "https://openapi.rrrosmontis.icu/jm/get_comic?id=350234" -o jm.zip # 返回 ZIP
```
## 日志
```
/var/log/openapi.log /var/log/pixiv.log /var/log/bilibili.log
/var/log/ginkgo_sign_in.log /var/log/bilibili_refresh.log /var/log/pixiv_token_refresh.log
/var/log/jm.log
```
缓存命中/落盘会在 `bilibili.log` 打印 `cache HIT / MISS / STORE`(已加 `flush`,实时可见)。
## 变更记录
**2026-10-22(服务器重装重建 + bilibili 出口 WAF 修复)**
- hk 重装为 Ubuntu 22.04,按本文件从零重建:nginx + acme.sh(Let's Encrypt ECC)+ Python venv + 7 个 systemd 单元 + 凭据恢复;补充恢复时的两个坑位:
- `proxy_cache_path /var/cache/nginx/...` 要求父目录存在 → 迁移步骤补 `mkdir -p /var/cache/nginx`
- `yt-dlp` 合流需要系统包 `ffmpeg`
- **bilibili 修复**:hk 出口访问 `www.bilibili.com` 视频页被 WAF 下发 412/JS 挑战(浏览器 UA 触发,API 客户端 UA 可过),页面无 `__INITIAL_STATE__`,yt-dlp 页面提取必然失败;改为**官方 API 直取**(`x/web-interface/view` → `x/player/playurl` → 媒体段下载),yt-dlp 保留后备;实测公网 200/1.3MB/6.5s
- jm/dufe/pixiv/openapi 重建后实测正常;证书由 acme.sh 自动续期(cron 每 6h 检查)
**2026-09-23(仓库与线上一致性核对 + 完整性补齐)**
- 逐文件 md5 核对:源码/main.py、systemd 全部 7 个单元、ginkgo/jm/bilibili/refresh 各模块、站点 nginx conf 均与仓库一致
- **补录漂移**:`source/pixiv/pixiv.py` 线上版为 2026-09-17 的"图源错误码处理"增强(官方 API 404/401/403/429 分别返回明确错误、`fetch_illust_url` 返回 api_status、不再对确定性错误做无意义备用源重试),此前未入库;本次以线上版为准补录
- **仓库补齐恢复依赖**:新增 `nginx/00-limits.conf`(限流 zone,站点 conf 引用 `perip_req`/`perip`,缺失会使 `nginx -t` 失败)与 `nginx/00-websocket.conf`(`$connection_upgrade` map,同样必须);迁移步骤第 4 步已同步
- 凭据快照另存于本机备份目录(`tokens.json`、`cookies.txt`,0600),不入库
**2026-09-23(dufe mode=auto + bot 基准同步)**
- dufe 签到服务新增 **`&mode=auto`**(url 参数可省略):SSO 登录一次后复用登录态 → 拉取通知列表(`GET /student/message/getMessageAll` pageSize=20/readFlag=0 + `POST /student/busiMessage/list`)→ 按 bot ginkgo 同款规则发现活动(仅互动消息、文本含"签到"、jumpUrl 可解析为 `/course/detail/.../signin/<id>/<type>`,query 不入库,按 signInId 去重)→ 按时间序逐个自动签到
- **连续重试**:每活动首次 + 最多 5 次重试、间隔 3 秒(`MAX_SIGNIN_ATTEMPTS=6` / `SIGNIN_BACKOFF=(3,3,3,3,3)`,对齐 bot);上游错误/登录错时重登录一次立即重试(对齐 bot `_call_sign_in_local` 的 refresh 语义);**除限流外任何未确认成功都消耗重试次数**(对齐 bot `complete_signin`);限流即中断本轮,已完成结果照常返回(`data.rate_limited=true`)
- **结果聚合**:`data.results` 列表逐活动返回(activityKey/signInId/type/typeName/title/signInCode/status/success/message/attempts),另附 total/succeeded/failed/activities 计数;**`attempts` 为逐次完整记录列表**(每次尝试 `{"n", "success", "message"}`,relogin 后立即重试的那次带 `"relogin": true`;成功与重试用尽两种情况都携带全部尝试)
- **与 bot ginkgo 非签到码签到逻辑比对后以 bot 为基准同步**:
- 移植限流防护体系:`RateLimitedError` / `_RATE_MESSAGE` / `_RATE_CODES` / `is_rate_limited` / `_valid_retry_after` / `_LoginErrors` / `check_response` / `safe_error` / `_request`(原实现完全无限流检测)
- `parse_course_url`:courseId 由 `\d+` 放宽为 `[0-9A-Za-z]+`,报错文案对齐
- `commit_sign_in`:经 `_request` 包装(`check_response`),失败消息 "request to InTheSign failed: %s" % safe_error(e)(原为 "upstream request failed")
- `fetch_sign_in_info`:getSignInInfo 响应过 `check_response`;新增 `sess`/`token` 参数注入(auto 模式请求内复用登录态,省去每活动一次 SSO);补齐 "Malformed getSignInInfo response" 非 dict 校验
- 验证:免凭据单测(发现/过滤正反用例)+ 假凭据 401 线路测试 + bj 真账密**只读**联调(登录 OK,notices=20/busi=18,发现 12 个活动含 1 个普通签到,类型/标题解析正确);未做真实提交(避免与运行中的 bot 轮询交叉签到)
**2026-09-23(早些)**
- 新增 **jm 漫画下载打包服务**:`source/jm/jm.py` + `jm.service`(`127.0.0.1:9004`,aiohttp + jmcomic,端点 `GET /jm/get_comic?id=xxx[&range=起-止]`)
- 移植自 qqbot 的 jm 模块(bj `C:/call/bot/utility/jm.py`):保留章节范围解析 / 文件名清洗 / webp→jpg 打包逻辑;去掉 QQ 回复、代理轮换(hk 可直连源站)、上传 cf 文件服务器三部分,改为 HTTP 直返 ZIP
- ZIP 为临时产物:返回后 600 秒延时删除(`asyncio` 延时任务 + 每次请求 `_cache_sweep()` 兜底清理,防进程重启残留);TTL 窗口内同名请求直接复用
- 单次上限 30 章(`MAX_CHAPTERS`,与 qqbot 一致;也避免超过聚合层 `proxy()` 的 300s 超时)
- 在聚合层 `main.py` 注册该端点:新增 `JM_BACKEND`,复用既有 `proxy()`(固定 GET),零改动既有行为
- nginx **无需改动**:`location /` 兜底已把 `/jm/...` 转发给 `127.0.0.1:9008`(与 dufe 相同)
- `env/general` 新增依赖 `jmcomic`(2.7.7,`pip install jmcomic`)
**2026-09-13**
- 新增 **dufe 银杏签到码查询服务**:`source/dufe/ginkgostu_sign_in/ginkgo_sign_in.py` + `ginkgo_sign_in.service`(`127.0.0.1:9005`,Flask + waitress 16 线程,端点 `GET|POST /dufe/ginkgostu_sign_in`)
- 在聚合层 `main.py` 注册该端点:新增 `DUFE_BACKEND` 与 `proxy_forward()`(保留请求方法/请求体;原 `proxy()` 固定 GET 行为未改动)
- nginx **无需改动**:`location /` 兜底已把 `/dufe/...` 转发给 `127.0.0.1:9008`
- 修复 `ginkgo_sign_in.py` 两处缺陷:
- `_b64_decode` 先做 `urllib.parse.unquote`(iOS 快捷指令会把 base64 的 `=` 编码成 `%3D`,旧实现直接 `binascii.Error`);顺序为先 unquote、再 `" "→"+"`
- `commit` 判定由 fail-open 改为 fail-closed:须 `isinstance(dict)` 且 `success is True`,否则一律判失败并返回 `msg = "sign-in failed: <原因>"`
- `parse_course_url` 报错文案改为 `URL is not a sign-in page (missing /signin/<signInId>/<type>): <url>`
**2026-09-12**
- 端口重排 `9888 / 9088 / 19990 → 9008 / 9007 / 9006`(全部仅 `127.0.0.1`)
- pixiv 曾误绑 `0.0.0.0:9088` 对外裸奔(绕过 nginx 日志与限流),已收口为 `127.0.0.1`
- 主入口 `main.py` 由 `source/openapi/` 上移到 `source/`;systemd `WorkingDirectory` 同步
- pixiv 的 `IMAGE_DIR` 由硬编码 `/home/pixiv/images` 改为相对 `../cache/pixiv/images`,并建 `/home/images` 软链
- bilibili 新增真缓存(`../cache/bilibili`)并修复 `/tmp` 泄漏,新增 `cache_info` 接口
- pixiv 改用 waitress 8 线程
- 删除 `kim1.work` / `zxx.aeolianchi.me` 两个 vhost
- 清理历史 `.bak` / `__pycache__` / 空日志等冗余文件
**2026-09-02** 新增 `pixiv-token-refresh`;补 `/img-original/` 反代(App API 原图路径)
**2026-07-31** 三服务迁移落位(Python 环境合并为单一 `general`)
+8
View File
@@ -0,0 +1,8 @@
limit_conn_zone $binary_remote_addr zone=perip:10m;
limit_conn_zone $server_name zone=perserver:10m;
# 2026-09-23 新增:按 IP 请求速率限制(轻防滥用档)
# openapi 域名 DNS 直连本机(无 CDN),$binary_remote_addr 即真实客户端 IP
limit_req_zone $binary_remote_addr zone=perip_req:10m rate=10r/s;
limit_req_status 429;
limit_conn_status 429;
+4
View File
@@ -0,0 +1,4 @@
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
+9 -5
View File
@@ -1,5 +1,5 @@
# openapi.rrrosmontis.icu 统一入口(原 api/pixiv/apixiv.bu4.ren 合并) # openapi.rrrosmontis.icu 统一入口(原 api/pixiv/apixiv.bu4.ren 合并)
# /pixiv/ /bilibili/ -> openapi 聚合层 127.0.0.1:9888 (main.py) # /pixiv/ /bilibili/ -> openapi 聚合层 127.0.0.1:9008 (main.py)
# /img-master/ /img-original/ -> i.pximg.net 图片反代(原 apixiv.bu4.ren 逻辑,方案二) # /img-master/ /img-original/ -> i.pximg.net 图片反代(原 apixiv.bu4.ren 逻辑,方案二)
# 2026-09-02 补 /img-original/:App API 原图 URL 为 /img-original/ 前缀,此前未配置导致 404 # 2026-09-02 补 /img-original/:App API 原图 URL 为 /img-original/ 前缀,此前未配置导致 404
@@ -31,6 +31,10 @@ server {
add_header Strict-Transport-Security "max-age=31536000"; add_header Strict-Transport-Security "max-age=31536000";
error_page 497 https://$host$request_uri; error_page 497 https://$host$request_uri;
# ---- 按 IP 限流(轻防滥用档,2026-09-23):10r/s burst 20、并发连接 10,超出 429 ----
limit_req zone=perip_req burst=20 nodelay;
limit_conn perip 10;
resolver 8.8.8.8 1.1.1.1 ipv6=off valid=300s; resolver 8.8.8.8 1.1.1.1 ipv6=off valid=300s;
# ---- 图片反代 i.pximg.net(原 apixiv.bu4.ren 逻辑,路径前缀 /img-master/)---- # ---- 图片反代 i.pximg.net(原 apixiv.bu4.ren 逻辑,路径前缀 /img-master/)----
@@ -85,7 +89,7 @@ server {
# ---- pixiv(openapi 聚合层 /pixiv/get_image)---- # ---- pixiv(openapi 聚合层 /pixiv/get_image)----
location ^~ /pixiv/ { location ^~ /pixiv/ {
proxy_pass http://127.0.0.1:9888; proxy_pass http://127.0.0.1:9008;
proxy_set_header Host $http_host; proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Real-Port $remote_port; proxy_set_header X-Real-Port $remote_port;
@@ -104,7 +108,7 @@ server {
# ---- bilibili(openapi 聚合层 /bilibili/get_video)---- # ---- bilibili(openapi 聚合层 /bilibili/get_video)----
location ^~ /bilibili/ { location ^~ /bilibili/ {
proxy_pass http://127.0.0.1:9888; proxy_pass http://127.0.0.1:9008;
proxy_set_header Host $http_host; proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Real-Port $remote_port; proxy_set_header X-Real-Port $remote_port;
@@ -121,9 +125,9 @@ server {
proxy_set_header Connection $connection_upgrade; proxy_set_header Connection $connection_upgrade;
} }
# ---- 其余(根路径 / )→ 9888(main.py:跳转文章页)---- # ---- 其余(根路径 / )→ 9008(main.py:跳转文章页)----
location / { location / {
proxy_pass http://127.0.0.1:9888; proxy_pass http://127.0.0.1:9008;
proxy_set_header Host $http_host; proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Real-Port $remote_port; proxy_set_header X-Real-Port $remote_port;
+204 -27
View File
@@ -3,7 +3,9 @@ import sys
import base64 import base64
import os import os
import re import re
import shutil
import tempfile import tempfile
import time
from pathlib import Path from pathlib import Path
import httpx import httpx
@@ -12,6 +14,27 @@ from aiohttp import web
COOKIE_FILE = str(Path(__file__).parent / "cookies.txt") COOKIE_FILE = str(Path(__file__).parent / "cookies.txt")
UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36" UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
# 2026-10-22: hk 出口被 bilibili WAF 下发 412 / JS 挑战页(浏览器 UA 触发),
# 页面里没有可用的 __INITIAL_STATE__,yt-dlp 的页面提取必然失败。
# 实测 API 客户端 UA(yt-dlp/x、curl/x)可通过,故主路径改为官方 API 直取;
# yt-dlp 保留为后备(其他出口/未来 WAF 策略变化时仍可工作)。
API_UA = "yt-dlp/2026.10.21"
VIEW_API = "https://api.bilibili.com/x/web-interface/view"
PLAYURL_API = "https://api.bilibili.com/x/player/playurl"
# ---------------- 本地缓存(与 pixiv 同级:source/cache/bilibili) ----------------
CACHE_DIR = (Path(__file__).resolve().parent / "../cache/bilibili").resolve()
CACHE_DIR.mkdir(parents=True, exist_ok=True)
CACHE_MAX_BYTES = 2 * 1024 ** 3 # 缓存总量上限 2 GiB
CACHE_TTL_SECONDS = 7 * 24 * 3600 # 7 天未访问即淘汰
VIDEO_EXTS = (".mp4", ".flv", ".mkv", ".webm")
CONTENT_TYPES = {
".mp4": "video/mp4",
".flv": "video/x-flv",
".mkv": "video/x-matroska",
".webm": "video/webm",
}
def _prepare_cookie_file(): def _prepare_cookie_file():
"""Convert simple name=value cookies.txt to Netscape format for yt-dlp.""" """Convert simple name=value cookies.txt to Netscape format for yt-dlp."""
@@ -36,6 +59,65 @@ def _prepare_cookie_file():
return tmp.name return tmp.name
def _cache_file(bvid):
"""缓存命中则返回文件路径(同时刷新 mtime 作为 LRU 依据),否则 None。"""
for f in CACHE_DIR.glob(f"{bvid}.*"):
if f.is_file() and f.suffix.lower() in VIDEO_EXTS:
try:
os.utime(f, None)
except OSError:
pass
return f
return None
def _cache_files():
out = []
for f in CACHE_DIR.iterdir():
if f.is_file() and f.suffix.lower() in VIDEO_EXTS:
out.append(f)
return out
def _cache_sweep():
"""先按 TTL 淘汰过期文件,再按总量上限从最旧(mtime 最小)开始淘汰。"""
now = time.time()
for f in _cache_files():
try:
if now - f.stat().st_mtime > CACHE_TTL_SECONDS:
f.unlink()
except OSError:
pass
files = _cache_files()
total = 0
for f in files:
try:
total += f.stat().st_size
except OSError:
pass
if total > CACHE_MAX_BYTES:
for f in sorted(files, key=lambda p: p.stat().st_mtime):
if total <= CACHE_MAX_BYTES:
break
try:
size = f.stat().st_size
f.unlink()
total -= size
except OSError:
pass
def _file_response(path, bvid):
return web.FileResponse(
path=path,
headers={
"Content-Type": CONTENT_TYPES.get(path.suffix.lower(), "application/octet-stream"),
"Content-Disposition": f'attachment; filename="{path.name}"',
},
)
async def resolve_bvid(link): async def resolve_bvid(link):
# 直接是 BV 号 # 直接是 BV 号
m = re.search(r"(BV[a-zA-Z0-9]{10})", link) m = re.search(r"(BV[a-zA-Z0-9]{10})", link)
@@ -70,6 +152,60 @@ def decode_url(url_b64):
return base64.b64decode(url_b64.encode()).decode() return base64.b64decode(url_b64.encode()).decode()
def _api_headers():
return {"User-Agent": API_UA, "Referer": "https://www.bilibili.com/"}
def _api_get(url, params, what="bilibili api"):
import requests
r = requests.get(url, params=params, headers=_api_headers(), timeout=30)
r.raise_for_status()
j = r.json()
if not isinstance(j, dict) or j.get("code") != 0:
raise RuntimeError("%s failed: code=%s message=%s" % (
what, (j or {}).get("code"), (j or {}).get("message")))
return j.get("data") or {}
def _download_via_api(bvid, work_dir):
"""官方 API 直取(view -> playurl -> 媒体段),返回落盘路径。
hk 出口被页面 WAF 挑战时 yt-dlp 无法工作,此路径只走 api.bilibili.com
与媒体 CDN,实测不受该挑战影响。"""
import requests
data = _api_get(VIEW_API, {"bvid": bvid}, "view")
cid = data.get("cid")
if not cid:
raise RuntimeError("view api returned no cid")
play = _api_get(PLAYURL_API, {
"bvid": bvid, "cid": cid, "qn": 80, "fnval": 0, "fourk": 1,
}, "playurl")
durls = play.get("durl") or []
if not durls:
raise RuntimeError("playurl api returned no durl segments")
out = Path(work_dir) / f"{bvid}.mp4"
try:
with open(out, "wb") as f:
for seg in durls:
with requests.get(seg["url"], headers=_api_headers(),
stream=True, timeout=120) as r:
r.raise_for_status()
for chunk in r.iter_content(chunk_size=256 * 1024):
if chunk:
f.write(chunk)
if out.stat().st_size == 0:
raise RuntimeError("empty media file")
return out
except Exception:
try:
out.unlink(missing_ok=True)
except OSError:
pass
raise
async def get_video(url_b64=None, bv=None): async def get_video(url_b64=None, bv=None):
if bv: if bv:
bvid = bv bvid = bv
@@ -90,11 +226,35 @@ async def get_video(url_b64=None, bv=None):
except ValueError as e: except ValueError as e:
return web.json_response({"error": str(e)}, status=400) return web.json_response({"error": str(e)}, status=400)
# 1) 缓存命中:直接回文件,不再调用 yt-dlp
cached = _cache_file(bvid)
if cached:
print(f"[bilibili] cache HIT {bvid} -> {cached.name} ({cached.stat().st_size} B)", flush=True)
return _file_response(cached, bvid)
print(f"[bilibili] cache MISS {bvid}, downloading...", flush=True)
# 2) 下载到缓存目录内的隐藏临时目录(同盘,才能原子 rename 落盘)
tmpdir = tempfile.mkdtemp(prefix=".dl-", dir=CACHE_DIR)
cookie_tmp = None
try:
# 2a) 主路径:官方 API 直取(绕开页面 WAF,不依赖 cookie/yt-dlp)
video_path = None
api_err = None
try:
video_path = await asyncio.to_thread(_download_via_api, bvid, tmpdir)
print(f"[bilibili] api download OK {bvid}", flush=True)
except Exception as e:
api_err = e
print(f"[bilibili] api download failed {bvid}: {e}; fallback yt-dlp", flush=True)
# 2b) 后备:yt-dlp(需要 cookie 文件)
if video_path is None:
cookie_tmp = _prepare_cookie_file() cookie_tmp = _prepare_cookie_file()
if not cookie_tmp: if not cookie_tmp:
return web.json_response({"error": "missing bilibili cookie"}, status=500) return web.json_response(
{"error": "download failed", "detail": "api path failed and no cookie for yt-dlp fallback: %s" % api_err},
tmpdir = tempfile.mkdtemp(prefix="bilibili_") status=502,
)
output_template = os.path.join(tmpdir, "video.%(ext)s") output_template = os.path.join(tmpdir, "video.%(ext)s")
cmd = [ cmd = [
os.path.join(os.path.dirname(sys.executable), "yt-dlp"), os.path.join(os.path.dirname(sys.executable), "yt-dlp"),
@@ -106,29 +266,23 @@ async def get_video(url_b64=None, bv=None):
"-o", output_template, "-o", output_template,
f"https://www.bilibili.com/video/{bvid}", f"https://www.bilibili.com/video/{bvid}",
] ]
try:
proc = await asyncio.create_subprocess_exec( proc = await asyncio.create_subprocess_exec(
*cmd, *cmd,
stdout=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
) )
try: try:
stdout, stderr = await asyncio.wait_for( stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=300)
proc.communicate(), timeout=300
)
except asyncio.TimeoutError: except asyncio.TimeoutError:
proc.kill() proc.kill()
await proc.wait() await proc.wait()
return web.json_response({"error": "download timeout"}, status=504) return web.json_response({"error": "download timeout"}, status=504)
finally:
if cookie_tmp and os.path.exists(cookie_tmp):
os.unlink(cookie_tmp)
video_files = [p for p in Path(tmpdir).glob("*") if p.is_file()] video_files = [p for p in Path(tmpdir).glob("*") if p.is_file()]
if proc.returncode != 0 and not video_files: if proc.returncode != 0 and not video_files:
err_msg = stderr.decode()[-1000:] if stderr else "unknown error" err_msg = stderr.decode()[-1000:] if stderr else "unknown error"
return web.json_response( return web.json_response(
{"error": "download failed", "detail": err_msg}, {"error": "download failed", "detail": "api: %s | yt-dlp: %s" % (api_err, err_msg)},
status=502, status=502,
) )
@@ -138,24 +292,22 @@ async def get_video(url_b64=None, bv=None):
video_path = max(video_files, key=lambda p: p.stat().st_size) video_path = max(video_files, key=lambda p: p.stat().st_size)
ext = video_path.suffix.lower() ext = video_path.suffix.lower()
content_types = { final = CACHE_DIR / f"{bvid}{ext}"
".mp4": "video/mp4", try:
".flv": "video/x-flv", os.replace(video_path, final) # 同盘原子落盘
".mkv": "video/x-matroska", except OSError:
".webm": "video/webm", shutil.move(str(video_path), str(final))
} _cache_sweep()
content_type = content_types.get(ext, "application/octet-stream") print(f"[bilibili] cache STORE {bvid} -> {final.name} ({final.stat().st_size} B)", flush=True)
return _file_response(final, bvid)
return web.FileResponse(
path=video_path,
headers={
"Content-Type": content_type,
"Content-Disposition": f'attachment; filename="{video_path.name}"',
},
)
except Exception as e: except Exception as e:
print(f"[bilibili] Error: {e}") print(f"[bilibili] Error: {e}")
return web.json_response({"error": str(e)}, status=500) return web.json_response({"error": str(e)}, status=500)
finally:
# 关键修复:无论成功/失败/超时都回收临时目录(原实现会把整个视频永久留在 /tmp)
shutil.rmtree(tmpdir, ignore_errors=True)
if cookie_tmp and os.path.exists(cookie_tmp):
os.unlink(cookie_tmp)
async def handle_bilibili(request): async def handle_bilibili(request):
@@ -168,8 +320,33 @@ async def handle_bilibili(request):
return await get_video(url_b64=url_b64) return await get_video(url_b64=url_b64)
async def handle_cache_info(request):
files = []
total = 0
for f in _cache_files():
try:
st = f.stat()
except OSError:
continue
files.append({"name": f.name, "size": st.st_size, "mtime": int(st.st_mtime)})
total += st.st_size
return web.json_response(
{
"cache_dir": str(CACHE_DIR),
"cache_max_bytes": CACHE_MAX_BYTES,
"cache_ttl_seconds": CACHE_TTL_SECONDS,
"count": len(files),
"total_bytes": total,
"free_bytes": shutil.disk_usage(CACHE_DIR).free,
"files": sorted(files, key=lambda x: -x["size"]),
}
)
app = web.Application() app = web.Application()
app.router.add_get("/bilibili/get_video", handle_bilibili) app.router.add_get("/bilibili/get_video", handle_bilibili)
app.router.add_get("/bilibili/cache_info", handle_cache_info)
if __name__ == "__main__": if __name__ == "__main__":
web.run_app(app, host="127.0.0.1", port=19990) print(f"Cache directory: {CACHE_DIR}")
web.run_app(app, host="127.0.0.1", port=9006)
File diff suppressed because it is too large Load Diff
+280
View File
@@ -0,0 +1,280 @@
# -*- coding: utf-8 -*-
# jm.py — JM 漫画下载 API 模块
# 流程: 源站下载 -> 本地临时目录打包 ZIP -> HTTP 返回 -> 延时删除
# 移植自 qqbot 的 jm 模块(bj C:/call/bot/utility/jm.py),
# 去掉 QQ 机器人 / 代理轮换 / 上传 cf 逻辑,改为与 pixiv/bilibili 一致的 HTTP 直返模式。
# hk 服务器可直连源站,无需代理。
import asyncio
import shutil
import tempfile
import time
import unicodedata
import zipfile
from pathlib import Path
from aiohttp import web
# ---------------- 本地临时缓存(与 pixiv/bilibili 同级:source/cache/jm) ----------------
CACHE_DIR = (Path(__file__).resolve().parent / "../cache/jm").resolve()
CACHE_DIR.mkdir(parents=True, exist_ok=True)
MAX_CHAPTERS = 30 # 单次最多章节(与 qqbot 一致,同时避免超过聚合层 300s 超时)
ZIP_TTL_SECONDS = 600 # ZIP 返回后保留 10 分钟再删除
PORT = 9004
_image_exts = (".webp", ".jpg", ".png")
# 同一本子防并发(album_id -> asyncio.Lock)
_locks = {}
_locks_guard = asyncio.Lock()
def _sanitize_filename(name):
"""清理文件名非法字符,保留中英文/数字/常见符号"""
cleaned = unicodedata.normalize("NFKC", name or "")
result = []
for ch in cleaned:
if ch.isalnum() or ch in ("_", "-", ".", "(", ")", "[", "]", "(", ")", "【", "】"):
result.append(ch)
elif ch.isspace():
result.append("_")
else:
result.append("-")
s = "".join(result)
while "__" in s:
s = s.replace("__", "_")
while "--" in s:
s = s.replace("--", "-")
return s[:80].strip("_-") or "unnamed"
def _zip_files():
return [f for f in CACHE_DIR.glob("*.zip") if f.is_file()]
def _cache_sweep():
"""兜底清理:进程重启后残留的过期 ZIP(mtime 超过 TTL 即删)"""
now = time.time()
for f in _zip_files():
try:
if now - f.stat().st_mtime > ZIP_TTL_SECONDS:
f.unlink()
print(f"[jm] sweep expired zip: {f.name}", flush=True)
except OSError:
pass
async def _delete_later(path, delay):
"""返回响应后延时删除 ZIP"""
await asyncio.sleep(delay)
try:
Path(path).unlink(missing_ok=True)
print(f"[jm] ttl delete: {Path(path).name}", flush=True)
except OSError:
pass
class _ParamError(RuntimeError):
"""参数类错误,返回 400"""
def _resolve_album_plan(album_id, range_start, range_end):
"""获取元数据并确定章节范围与 ZIP 文件名。
返回 (title, episodes, range_start, range_end, zip_name),失败抛异常"""
from jmcomic import JmModuleConfig
option = JmModuleConfig.option_class().default()
client = option.build_jm_client()
album = client.get_album_detail(int(album_id))
title = getattr(album, "title", "") or str(album_id)
episodes = getattr(album, "episode_list", []) or []
total = len(episodes)
if total == 0:
raise RuntimeError("未找到任何章节")
if range_start is None:
if total > MAX_CHAPTERS:
raise _ParamError(
f"共 {total} 章, 超过单次上限 {MAX_CHAPTERS} 章, 请用 range 参数分段下载 (如 range=1-{MAX_CHAPTERS})"
)
range_start, range_end = 1, total
else:
if range_start < 1 or range_end < range_start:
raise _ParamError("章节范围不合法")
if range_end - range_start + 1 > MAX_CHAPTERS:
raise _ParamError(f"章节跨度超过 {MAX_CHAPTERS} 章, 请缩小范围")
if range_start > total:
raise _ParamError(f"起始章节超出总章节数 {total}")
range_end = min(range_end, total)
zip_name = f"{_sanitize_filename(title)}_{album_id}_ch{range_start}-{range_end}.zip"
return title, episodes, range_start, range_end, zip_name
def _download_and_pack(album_id, plan):
"""下载并打包 ZIP。成功返回 (zip_path, image_count, ok_ch, fail_ch),失败抛异常"""
from jmcomic import JmModuleConfig
from PIL import Image
title, episodes, range_start, range_end, zip_name = plan
work_dir = Path(tempfile.mkdtemp(prefix=f".dl-{album_id}-", dir=CACHE_DIR))
try:
# 逐章下载(hk 直连源站,无需代理)
ok_count = 0
fail_count = 0
for idx in range(range_start - 1, range_end):
photo_id = episodes[idx][0]
try:
dl_option = JmModuleConfig.option_class().construct({
"dir_rule": {"rule": "Bd_Pid", "base_dir": str(work_dir)},
"download": {"image": {"suffix": ".webp"}},
})
dler = JmModuleConfig.downloader_class()(dl_option)
dler.download_photo(int(photo_id))
ok_count += 1
print(f"[jm] 章节下载完成: {album_id}/{photo_id}", flush=True)
except Exception as e:
fail_count += 1
print(f"[jm] 章节下载失败: {album_id}/{photo_id}: {e}", flush=True)
# 收集图片
images = []
for ext in _image_exts:
images.extend(sorted(work_dir.rglob(f"*{ext}")))
if not images:
raise RuntimeError("未下载到任何图片")
# 打包 ZIP (webp 统一转 jpg, 无密码)
zip_path = CACHE_DIR / zip_name
tmp_dir = Path(tempfile.mkdtemp(prefix=".pack-", dir=CACHE_DIR))
try:
packed = []
for i, p in enumerate(images, 1):
out = tmp_dir / f"{i:04d}.jpg"
try:
img = Image.open(str(p))
if img.mode != "RGB":
img = img.convert("RGB")
img.save(str(out), "JPEG", quality=95)
except Exception:
out = tmp_dir / f"{i:04d}{p.suffix.lower()}"
shutil.copyfile(str(p), str(out))
packed.append(out)
with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as zf:
for j in packed:
zf.write(j, j.name)
finally:
shutil.rmtree(tmp_dir, ignore_errors=True)
print(f"[jm] 打包完成: {zip_name} ({len(images)} 张, 成功 {ok_count} 章 / 失败 {fail_count} 章)", flush=True)
return zip_path, len(images), ok_count, fail_count
finally:
# 无论成功/失败都回收图片临时目录
shutil.rmtree(work_dir, ignore_errors=True)
def _file_response(zip_path):
return web.FileResponse(
path=zip_path,
headers={
"Content-Type": "application/zip",
"Content-Disposition": f'attachment; filename="{zip_path.name}"',
},
)
async def handle_jm(request):
album_id = request.query.get("id", "").strip()
if not album_id or not album_id.isdigit():
return web.json_response({"error": "missing or invalid id parameter"}, status=400)
range_start = range_end = None
range_str = request.query.get("range", "").strip()
if range_str:
import re
m = re.fullmatch(r"(\d{1,4})-(\d{1,4})", range_str)
if not m:
return web.json_response(
{"error": "invalid range format, expected: start-end (e.g. 1-30)"}, status=400
)
range_start, range_end = int(m.group(1)), int(m.group(2))
_cache_sweep()
# 同本子串行,防止并发重复下载
async with _locks_guard:
lock = _locks.setdefault(album_id, asyncio.Lock())
if lock.locked():
return web.json_response(
{"error": "same album is being processed, please retry later"}, status=409
)
async with lock:
try:
plan = await asyncio.to_thread(
_resolve_album_plan, album_id, range_start, range_end
)
except _ParamError as e:
return web.json_response({"error": str(e)}, status=400)
except Exception as e:
return web.json_response(
{"error": "failed to fetch album info", "detail": str(e)[:200]}, status=502
)
zip_name = plan[4]
existing = CACHE_DIR / zip_name
if existing.exists():
# TTL 窗口内重复请求:直接复用并重新计时
try:
existing.touch()
except OSError:
pass
print(f"[jm] reuse zip: {zip_name}", flush=True)
asyncio.create_task(_delete_later(existing, ZIP_TTL_SECONDS))
return _file_response(existing)
try:
zip_path, img_count, ok_c, fail_c = await asyncio.to_thread(
_download_and_pack, album_id, plan
)
except Exception as e:
return web.json_response(
{"error": "download or pack failed", "detail": str(e)[:200]}, status=502
)
asyncio.create_task(_delete_later(zip_path, ZIP_TTL_SECONDS))
return _file_response(zip_path)
async def handle_cache_info(request):
files = []
total = 0
for f in _zip_files():
try:
st = f.stat()
except OSError:
continue
files.append({"name": f.name, "size": st.st_size, "mtime": int(st.st_mtime)})
total += st.st_size
return web.json_response(
{
"cache_dir": str(CACHE_DIR),
"zip_ttl_seconds": ZIP_TTL_SECONDS,
"max_chapters": MAX_CHAPTERS,
"count": len(files),
"total_bytes": total,
"free_bytes": shutil.disk_usage(CACHE_DIR).free,
"files": sorted(files, key=lambda x: -x["size"]),
}
)
app = web.Application()
app.router.add_get("/jm/get_comic", handle_jm)
app.router.add_get("/jm/cache_info", handle_cache_info)
if __name__ == "__main__":
print(f"Cache directory: {CACHE_DIR}")
web.run_app(app, host="127.0.0.1", port=PORT)
+107
View File
@@ -0,0 +1,107 @@
from aiohttp import web, ClientSession, ClientTimeout
PIXIV_BACKEND = "http://127.0.0.1:9007"
BILIBILI_BACKEND = "http://127.0.0.1:9006"
DUFE_BACKEND = "http://127.0.0.1:9005"
JM_BACKEND = "http://127.0.0.1:9004"
async def handle_pixiv(request):
pid = request.query.get("id", "")
if not pid:
return web.json_response({"error": "missing id parameter"}, status=400)
target = f"{PIXIV_BACKEND}/{pid}"
return await proxy(request, target)
async def handle_bilibili(request):
qs = request.query_string
target = f"{BILIBILI_BACKEND}/bilibili/get_video"
if qs:
target += "?" + qs
return await proxy(request, target)
async def handle_dufe_sign_in(request):
# ginkgo_sign_in.service(127.0.0.1:9005),端点 /dufe/ginkgostu_sign_in,支持 GET|POST
target = f"{DUFE_BACKEND}/dufe/ginkgostu_sign_in"
if request.query_string:
target += "?" + request.query_string
return await proxy_forward(request, target)
async def handle_jm(request):
qs = request.query_string
target = f"{JM_BACKEND}/jm/get_comic"
if qs:
target += "?" + qs
return await proxy(request, target)
async def proxy(request, target):
timeout = ClientTimeout(total=300)
async with ClientSession(timeout=timeout) as session:
try:
async with session.get(target) as resp:
body = await resp.read()
headers = {}
for key in ("Content-Type", "Content-Disposition", "Cache-Control"):
if key in resp.headers:
headers[key] = resp.headers[key]
return web.Response(
body=body,
status=resp.status,
headers=headers,
)
except Exception as e:
return web.json_response(
{"error": "backend unreachable", "detail": str(e)}, status=502
)
async def proxy_forward(request, target):
# 保留请求方法与请求体的转发,供需要 POST 的后端使用(dufe 签到码查询)
timeout = ClientTimeout(total=300)
data = None
fwd_headers = {}
if request.method not in ("GET", "HEAD"):
data = await request.read()
ctype = request.headers.get("Content-Type")
if ctype:
fwd_headers["Content-Type"] = ctype
async with ClientSession(timeout=timeout) as session:
try:
async with session.request(
request.method, target, data=data, headers=fwd_headers
) as resp:
body = await resp.read()
headers = {}
for key in ("Content-Type", "Content-Disposition", "Cache-Control"):
if key in resp.headers:
headers[key] = resp.headers[key]
return web.Response(
body=body,
status=resp.status,
headers=headers,
)
except Exception as e:
return web.json_response(
{"error": "backend unreachable", "detail": str(e)}, status=502
)
async def handle_root(request):
raise web.HTTPFound("https://rrrosmontis.icu/articles/index.html?id=00003")
app = web.Application()
app.router.add_get("/", handle_root)
app.router.add_get("/pixiv/get_image", handle_pixiv)
app.router.add_get("/bilibili/get_video", handle_bilibili)
app.router.add_get("/dufe/ginkgostu_sign_in", handle_dufe_sign_in)
app.router.add_post("/dufe/ginkgostu_sign_in", handle_dufe_sign_in)
app.router.add_get("/jm/get_comic", handle_jm)
if __name__ == "__main__":
web.run_app(app, host="127.0.0.1", port=9008)
-55
View File
@@ -1,55 +0,0 @@
from aiohttp import web, ClientSession, ClientTimeout
PIXIV_BACKEND = "http://127.0.0.1:9088"
BILIBILI_BACKEND = "http://127.0.0.1:19990"
async def handle_pixiv(request):
pid = request.query.get("id", "")
if not pid:
return web.json_response({"error": "missing id parameter"}, status=400)
target = f"{PIXIV_BACKEND}/{pid}"
return await proxy(request, target)
async def handle_bilibili(request):
qs = request.query_string
target = f"{BILIBILI_BACKEND}/bilibili/get_video"
if qs:
target += "?" + qs
return await proxy(request, target)
async def proxy(request, target):
timeout = ClientTimeout(total=300)
async with ClientSession(timeout=timeout) as session:
try:
async with session.get(target) as resp:
body = await resp.read()
headers = {}
for key in ("Content-Type", "Content-Disposition", "Cache-Control"):
if key in resp.headers:
headers[key] = resp.headers[key]
return web.Response(
body=body,
status=resp.status,
headers=headers,
)
except Exception as e:
return web.json_response(
{"error": "backend unreachable", "detail": str(e)}, status=502
)
async def handle_root(request):
raise web.HTTPFound("https://rrrosmontis.icu/articles/index.html?id=00003")
app = web.Application()
app.router.add_get("/", handle_root)
app.router.add_get("/pixiv/get_image", handle_pixiv)
app.router.add_get("/bilibili/get_video", handle_bilibili)
if __name__ == "__main__":
web.run_app(app, host="127.0.0.1", port=9888)
+28 -13
View File
@@ -12,8 +12,8 @@ USER_AGENT = "PixivIOSApp/7.13.0 (iOS 14.6; iPhone)"
urllib3.disable_warnings() urllib3.disable_warnings()
IMAGE_DIR = Path("/home/pixiv/images") IMAGE_DIR = (Path(__file__).resolve().parent / "../cache/pixiv/images").resolve()
IMAGE_DIR.mkdir(exist_ok=True) IMAGE_DIR.mkdir(parents=True, exist_ok=True)
app = Flask(__name__) app = Flask(__name__)
@@ -31,9 +31,11 @@ def get_access_token():
def fetch_illust_url(pid, page): def fetch_illust_url(pid, page):
"""返回 (original_url, api_status)。api_status: 200 成功 / 404 作品不存在 /
401|403 认证失败 / 429 限流 / 0 网络等临时错误 / -1 数据异常。"""
access_token = get_access_token() access_token = get_access_token()
if not access_token: if not access_token:
return None return None, 403
try: try:
resp = requests.get( resp = requests.get(
f"https://{API_HOST}/v1/illust/detail", f"https://{API_HOST}/v1/illust/detail",
@@ -47,16 +49,17 @@ def fetch_illust_url(pid, page):
verify=False verify=False
) )
if resp.status_code != 200: if resp.status_code != 200:
return None return None, resp.status_code
data = resp.json() data = resp.json()
if data.get("error"): if data.get("error"):
return None return None, -1
illust = data.get("illust") illust = data.get("illust")
if not illust: if not illust:
return None return None, -1
page_count = illust.get("page_count", 1) page_count = illust.get("page_count", 1)
if page >= page_count: if page >= page_count:
return None # 页码超出:作品存在但请求的分页不存在,按 404 语义处理
return None, 404
original_url = None original_url = None
if page_count > 1: if page_count > 1:
meta_pages = illust.get("meta_pages") meta_pages = illust.get("meta_pages")
@@ -69,10 +72,12 @@ def fetch_illust_url(pid, page):
if not original_url: if not original_url:
original_url = illust.get("image_urls", {}).get("large") original_url = illust.get("image_urls", {}).get("large")
if not original_url: if not original_url:
return None return None, -1
return original_url.replace("i.pximg.net", IMAGE_PROXY_DOMAIN) return original_url.replace("i.pximg.net", IMAGE_PROXY_DOMAIN), 200
except requests.RequestException:
return None, 0 # 网络等临时错误:可继续尝试备用源
except Exception: except Exception:
return None return None, -1
def make_re_url(pid, page): def make_re_url(pid, page):
@@ -146,9 +151,19 @@ def serve_image(path):
return send_file(cached_file, mimetype=mime, max_age=3600) return send_file(cached_file, mimetype=mime, max_age=3600)
urls_to_try = [] urls_to_try = []
official_url = fetch_illust_url(pid, page) official_url, api_status = fetch_illust_url(pid, page)
if official_url: if official_url:
urls_to_try.append(official_url) urls_to_try.append(official_url)
# 明确的作品级错误直接返回,不再做无意义的备用源重试:
# 404 = 作品不存在/已删除/分页不存在(官方 API 已裁决)
# 401/403 = token 失效(refresh 由独立服务负责,这里明确报错)
# 429 = pixiv API 限流(备用源同样是 pixiv 数据,重试无意义)
if api_status == 404:
return jsonify(error="作品不存在或已删除", code="not_found"), 404
if api_status in (401, 403):
return jsonify(error="pixiv 认证失败,token 可能失效", code="auth_error"), 502
if api_status == 429:
return jsonify(error="pixiv API 限流,请稍后再试", code="rate_limited"), 503
urls_to_try.append(make_re_url(pid, page)) urls_to_try.append(make_re_url(pid, page))
urls_to_try.append(make_cat_url(pid, page)) urls_to_try.append(make_cat_url(pid, page))
@@ -157,7 +172,7 @@ def serve_image(path):
if img_path: if img_path:
return send_file(img_path, mimetype=mime, max_age=3600) return send_file(img_path, mimetype=mime, max_age=3600)
return jsonify(error="Failed to fetch image from all sources"), 500 return jsonify(error="所有图源均取图失败,请稍后再试", code="all_sources_failed"), 500
@app.route('/health') @app.route('/health')
@@ -196,4 +211,4 @@ if __name__ == '__main__':
else: else:
print("Warning: access_token missing in tokens.json") print("Warning: access_token missing in tokens.json")
print(f"Cache directory: {IMAGE_DIR}") print(f"Cache directory: {IMAGE_DIR}")
app.run(host='0.0.0.0', port=9088, debug=False) app.run(host='127.0.0.1', port=9007, debug=False, threaded=True)
+17
View File
@@ -0,0 +1,17 @@
[Unit]
Description=Ginkgo Stu Sign-in Service (dufe)
After=network-online.target
Wants=network-online.target
[Service]
Environment=PATH=/home/openapi/env/general/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Type=simple
WorkingDirectory=/home/openapi/source/dufe/ginkgostu_sign_in
ExecStart=/home/openapi/env/general/bin/python /home/openapi/source/dufe/ginkgostu_sign_in/ginkgo_sign_in.py
Restart=always
RestartSec=5
StandardOutput=append:/var/log/ginkgo_sign_in.log
StandardError=append:/var/log/ginkgo_sign_in.log
[Install]
WantedBy=multi-user.target
+17
View File
@@ -0,0 +1,17 @@
[Unit]
Description=JM Comic Download API Service
After=network-online.target
Wants=network-online.target
[Service]
Environment=PATH=/home/openapi/env/general/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Type=simple
WorkingDirectory=/home/openapi/source/jm
ExecStart=/home/openapi/env/general/bin/python /home/openapi/source/jm/jm.py
Restart=always
RestartSec=5
StandardOutput=append:/var/log/jm.log
StandardError=append:/var/log/jm.log
[Install]
WantedBy=multi-user.target
+2 -2
View File
@@ -6,8 +6,8 @@ Wants=network-online.target
[Service] [Service]
Environment=PATH=/home/openapi/env/general/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Environment=PATH=/home/openapi/env/general/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Type=simple Type=simple
WorkingDirectory=/home/openapi/source/openapi WorkingDirectory=/home/openapi/source
ExecStart=/home/openapi/env/general/bin/python /home/openapi/source/openapi/main.py ExecStart=/home/openapi/env/general/bin/python /home/openapi/source/main.py
Restart=always Restart=always
RestartSec=5 RestartSec=5
StandardOutput=append:/var/log/openapi.log StandardOutput=append:/var/log/openapi.log
+1 -1
View File
@@ -8,7 +8,7 @@ Type=simple
User=root User=root
WorkingDirectory=/home/openapi/source/pixiv WorkingDirectory=/home/openapi/source/pixiv
Environment=PATH=/home/openapi/env/general/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Environment=PATH=/home/openapi/env/general/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ExecStart=/home/openapi/env/general/bin/python /home/openapi/source/pixiv/pixiv.py ExecStart=/home/openapi/env/general/bin/waitress-serve --host=127.0.0.1 --port=9007 --threads=8 pixiv:app
Restart=always Restart=always
RestartSec=5 RestartSec=5
StandardOutput=append:/var/log/pixiv.log StandardOutput=append:/var/log/pixiv.log