feat: 增加 systemd 定时执行与断点续跑

- backend/server.py: 信令 WebSocket 被远端关闭或视频流停滞超过阈值时自动拆除会话,
  由 finally 释放网易云端实例,避免会话空挂、后续 /start 一直拿到 503
- bin/maa-online-daily-run: 新增日常任务包装脚本,等待 API 就绪、启动云游戏、
  停滞看门狗(连续 ScreencapFailed 或 Disconnected)、失败即释放云会话并最多重试 3 次
- bin/maa-resume-tasks.py: 新增断点续跑规划器,重试时只运行未出现 Completed 的任务,
  全部完成则直接判定成功,不再开启新会话;StartUp 永不跳过
- systemd/: 新增 server/daily/update 服务与定时器模板,每天 04:00 起随机 0-2 小时执行
- bin/maa-online-install-systemd: 渲染 __ROOT__ 并安装、启用上述单元
- README.md, user_config/README.md: 补充自动定时执行、断点续跑与已知限制说明
- .gitignore: 忽略生成的 config/tasks/daily-resume.toml
This commit is contained in:
RRRosmontis
2026-09-19 16:45:48 +08:00
parent 73250c91b6
commit 0264053b08
12 changed files with 481 additions and 3 deletions
+1
View File
@@ -11,6 +11,7 @@ backend/.pending-login
runtime/
config/profiles/cloud.toml
config/tasks/daily.toml
config/tasks/daily-resume.toml
user_config/daily.toml
user_config/backups/
tests/fake-actions.jsonl
+36 -1
View File
@@ -134,6 +134,39 @@ bin/maa-online-update
该脚本更新官方 maa-cli、MaaCore、基础资源和热更新资源。适配代码位于官方管理目录之外,不会被 `maa update` 清理。
## 自动定时执行
仓库自带 systemd 单元模板(`systemd/`)。以 root 执行:
```bash
sudo bin/maa-online-install-systemd
```
脚本会把单元渲染到 `/etc/systemd/system/`(`__ROOT__` 替换为当前检出路径)并启用:
| 单元 | 作用 |
|---|---|
| `maa-online-server.service` | 云游戏 API 常驻服务(`Restart=always`,启动前清除代理环境变量) |
| `maa-daily.timer` | 每天 04:00 起随机 0–2 小时触发日常任务 |
| `maa-update.timer` | 每周日 03:20 起随机 30 分钟执行更新 |
日常任务由 `bin/maa-online-daily-run` 驱动:等待 API 就绪 → 启动云游戏 → 在停滞看门狗下执行
`config/tasks/daily.toml` → 失败则释放云会话并重试。定时器使用 `Persistent=true`,机器重启
错过触发时间后会自动补跑;同一单元不会并发执行。
重试采用**断点续跑**:`bin/maa-resume-tasks.py` 读取前几次尝试的日志,跳过已出现
`<任务类型> Completed` 的任务,只运行剩余任务;若全部完成则直接判定成功,不再开启新会话。
`StartUp` 永不跳过,因为每次 `/start` 都是全新的云实例。详见
[`user_config/README.md`](user_config/README.md#自动定时与断点续跑)。
查看运行情况:
```bash
systemctl list-timers maa-daily.timer maa-update.timer
journalctl -u maa-daily.service -n 50
journalctl -u maa-online-server.service -n 50
```
## 离线测试
启动假后端:
@@ -161,7 +194,9 @@ MaaCore -> adb-cloud -> fake backend: PASS
- 云视频压缩与网络抖动会降低 MAA 图像识别稳定性;
- 高频截图在低性能服务器上较慢,基建换班可能耗时较长;
- Home/ESC 等键值尚未完整验证;
- 断线自动恢复仍有限;
- 云端会主动断开会话(信令 WebSocket close、视频流停滞)。后端会在检测到这类情况时自动拆除
会话并释放云端实例,日常包装脚本另有停滞看门狗与最多 3 次断点续跑重试,但仍无法保证每次都跑完;
- 云游戏免费额度通常有限(常见为每天 30 分钟),一次完整日常约 24 分钟,重试会进一步消耗额度;
- 后端可能出现少量 H.264 解码丢包警告;
- 游戏自动化会真实消耗招聘许可、理智、信用等资源,请先 dry-run 并审阅配置。
+20 -2
View File
@@ -35,6 +35,7 @@ WIDTH = int(os.getenv("MAA_ONLINE_WIDTH", "1280"))
HEIGHT = int(os.getenv("MAA_ONLINE_HEIGHT", "720"))
FPS = os.getenv("MAA_ONLINE_FPS", "30")
START_WAIT_SECONDS = float(os.getenv("MAA_ONLINE_START_WAIT_SECONDS", "30"))
STALE_LIMIT = float(os.getenv("MAA_ONLINE_STALE_LIMIT_SECONDS", "120"))
# --- 颜色定义 ---
class Colors:
@@ -439,9 +440,26 @@ async def run_cloud_game():
raise RuntimeError("failed to receive the first cloud-game video frame")
# Use a cancellable wait instead of Event().wait()
# This allows the task to be properly cancelled by signals
# This allows the task to be properly cancelled by signals.
# Also watch for a dead cloud session: if the remote signaling
# websocket closes or the video stream stalls for too long, tear
# the session down (finally-block releases the NetEase session and
# closes the peer connection) so the next /start reconnects cleanly
# instead of serving 503s forever.
while True:
await asyncio.sleep(1)
await asyncio.sleep(5)
sock = app_state.sock
if sock is not None and sock.close_code is not None:
raise RuntimeError(
f"cloud signaling websocket closed (code {sock.close_code}); tearing down session"
)
snapshotter = app_state.snapshotter
if snapshotter is not None:
last_recv = getattr(snapshotter, "_last_recv_ts", 0.0)
if last_recv and (time.monotonic() - last_recv) > STALE_LIMIT:
raise RuntimeError(
f"cloud video stream stale for over {STALE_LIMIT:.0f}s; tearing down session"
)
except asyncio.CancelledError:
print(f"\n{Colors.CYAN}[*] Cloud game task cancelled.{Colors.RESET}")
+142
View File
@@ -0,0 +1,142 @@
#!/usr/bin/env bash
# MAA Online daily runner, designed for systemd oneshot execution.
#
# Flow: wait for the cloud API -> plan remaining tasks from earlier attempts ->
# start a cloud session -> run the daily task file under a stall watchdog ->
# on failure release the cloud session and retry (up to MAA_ONLINE_MAX_ATTEMPTS).
#
# Retries use resume mode: bin/maa-resume-tasks.py inspects the earlier
# attempts' logs and only the tasks that never reported "<Type> Completed" are
# run again, so a retry does not re-burn cloud-game minutes on finished work.
#
# Manual catch-up run (for example after a failed night, using the remaining
# daily quota):
# MAA_ONLINE_RESUME_FROM="artifacts/maa-daily-<stamp>.log" bin/maa-online-daily-run
set -uo pipefail
ROOT="${MAA_ONLINE_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}"
API="${MAA_ONLINE_API:-http://127.0.0.1:22888}"
LOG_DIR="${MAA_ONLINE_LOG_DIR:-$ROOT/artifacts}"
MAA_ONLINE_BIN="${MAA_ONLINE_BIN:-$ROOT/bin/maa-online}"
PYTHON="${MAA_ONLINE_PYTHON:-$ROOT/runtime/venv/bin/python}"
RESUME_HELPER="$ROOT/bin/maa-resume-tasks.py"
DAILY_SOURCE="${MAA_ONLINE_DAILY_SOURCE:-$ROOT/config/tasks/daily.toml}"
RESUME_FILE="${MAA_ONLINE_RESUME_OUT:-$ROOT/config/tasks/daily-resume.toml}"
MAX_ATTEMPTS="${MAA_ONLINE_MAX_ATTEMPTS:-3}"
STALL_FAILS="${MAA_ONLINE_STALL_FAILS:-12}" # consecutive ScreencapFailed (~24s+)
ATTEMPT_MAX_SECONDS="${MAA_ONLINE_ATTEMPT_MAX_SECONDS:-3000}" # per-attempt ceiling
mkdir -p "$LOG_DIR"
api() { # api <METHOD> <PATH>
curl --noproxy '*' -fsS --max-time 180 -X "$1" "$API$2"
}
echo "[daily-run] $(date --iso-8601=seconds) waiting for cloud API at $API ..."
ready=""
for _ in $(seq 1 90); do
if api GET /info >/dev/null 2>&1; then ready=1; break; fi
sleep 2
done
if [ -z "$ready" ]; then
echo "[daily-run] ERROR: cloud API not ready after 180s" >&2
exit 2
fi
REASON=""
run_daily_watchdog() { # arg1: log file, arg2: task file name; sets REASON on abort
local log="$1"
local task="$2"
REASON=""
"$MAA_ONLINE_BIN" run "$task" --batch -v --log-file="$log" &
local maa_pid=$!
while kill -0 "$maa_pid" 2>/dev/null; do
sleep 5
if [ $(( $(date +%s) - attempt_epoch )) -gt "$ATTEMPT_MAX_SECONDS" ]; then
REASON="attempt exceeded ${ATTEMPT_MAX_SECONDS}s ceiling"
break
fi
local streak
streak=$(awk '/ScreencapFailed/{c++; next} {c=0} END{print c+0}' "$log" 2>/dev/null)
if [ "${streak:-0}" -ge "$STALL_FAILS" ]; then
REASON="screencap stalled (${streak} consecutive failures; cloud session likely dead)"
break
fi
if grep -qE '\] Disconnected$' "$log" 2>/dev/null; then
REASON="MAA reported Disconnected (cloud session lost)"
break
fi
done
if [ -n "$REASON" ]; then
echo "[daily-run] watchdog: $REASON; aborting attempt" >&2
kill -INT "$maa_pid" 2>/dev/null
for _ in $(seq 1 6); do kill -0 "$maa_pid" 2>/dev/null || break; sleep 5; done
kill -0 "$maa_pid" 2>/dev/null && kill -9 "$maa_pid" 2>/dev/null
wait "$maa_pid" 2>/dev/null
return 1
fi
wait "$maa_pid" || { REASON="maa exited rc=$?"; return 1; }
return 0
}
attempt_logs=()
if [ -n "${MAA_ONLINE_RESUME_FROM:-}" ]; then
read -r -a seeded <<< "$MAA_ONLINE_RESUME_FROM"
attempt_logs+=("${seeded[@]}")
echo "[daily-run] seeded resume planning with: ${attempt_logs[*]}"
fi
plan_tasks() { # echoes the task file to run ("daily", "daily-resume" or "NONE")
if [ "${#attempt_logs[@]}" -eq 0 ]; then
echo daily
return
fi
local result
result=$("$PYTHON" "$RESUME_HELPER" "${attempt_logs[@]}" \
--source "$DAILY_SOURCE" --out "$RESUME_FILE" 2>&1 | tail -n1)
case "$result" in
NONE)
echo NONE
;;
RESUME_TASKS=*)
echo "[daily-run] resume plan: ${result#RESUME_TASKS=}" >&2
echo daily-resume
;;
*)
echo "[daily-run] WARN: resume planning failed (${result}); using full daily" >&2
echo daily
;;
esac
}
attempt=0
while [ "$attempt" -lt "$MAX_ATTEMPTS" ]; do
attempt=$((attempt + 1))
TASK_FILE=$(plan_tasks)
if [ "$TASK_FILE" = "NONE" ]; then
echo "[daily-run] $(date --iso-8601=seconds) every daily task already completed in an earlier attempt; nothing left to do"
exit 0
fi
echo "[daily-run] attempt $attempt/$MAX_ATTEMPTS: task file '$TASK_FILE'; starting cloud game ..."
if ! api POST /start; then
echo "[daily-run] attempt $attempt: /start failed (cloud service or token issue)" >&2
sleep 20
continue
fi
RUN_LOG="$LOG_DIR/maa-daily-$(date +%Y%m%d-%H%M%S).log"
echo "[daily-run] attempt $attempt: running '$TASK_FILE' (log: $RUN_LOG) ..."
attempt_epoch=$(date +%s)
if run_daily_watchdog "$RUN_LOG" "$TASK_FILE"; then
echo "[daily-run] $(date --iso-8601=seconds) daily completed successfully (attempt $attempt)"
exit 0
fi
attempt_logs+=("$RUN_LOG")
echo "[daily-run] attempt $attempt failed: ${REASON:-unknown}; releasing cloud session" >&2
api POST /exit >/dev/null 2>&1 || true
sleep 30
done
echo "[daily-run] ERROR: daily failed after $MAX_ATTEMPTS attempts (last: ${REASON:-unknown})" >&2
echo "[daily-run] if the failure mentions login/token, re-run: runtime/venv/bin/python backend/login.py request <phone>" >&2
exit 1
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env bash
# Install (or refresh) the systemd units that schedule the MAA Online daily
# task. Run as root. Units are rendered from systemd/*.tmpl-style sources in
# this repository with __ROOT__ replaced by the checkout path.
set -euo pipefail
ROOT="${MAA_ONLINE_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}"
SRC="$ROOT/systemd"
DST="${MAA_ONLINE_SYSTEMD_DIR:-/etc/systemd/system}"
UNITS=(maa-online-server.service maa-daily.service maa-daily.timer maa-update.service maa-update.timer)
if [ "$(id -u)" != "0" ]; then
echo "error: must run as root (writes to $DST)" >&2
exit 1
fi
if [ ! -d "$SRC" ]; then
echo "error: $SRC not found" >&2
exit 1
fi
for unit in "${UNITS[@]}"; do
sed "s|__ROOT__|$ROOT|g" "$SRC/$unit" > "$DST/$unit"
echo "installed $DST/$unit"
done
systemctl daemon-reload
systemctl enable --now maa-online-server.service
systemctl enable maa-daily.timer maa-update.timer
systemctl start maa-daily.timer maa-update.timer
echo
systemctl --no-pager list-timers maa-daily.timer maa-update.timer
cat <<EOF
MAA Online systemd units are active.
server log: journalctl -u maa-online-server -f
daily log: journalctl -u maa-daily -n 50
next runs: systemctl list-timers maa-daily.timer maa-update.timer
EOF
+145
View File
@@ -0,0 +1,145 @@
#!/usr/bin/env python3
"""Build a resume task file from partially finished MAA daily attempts.
A failed daily attempt still completes some tasks (their progress lives on the
game server), so re-running the whole daily file on a retry wastes cloud-game
minutes. This helper reads the previous attempts' MAA logs, finds which task
types already reported "<Type> Completed", and writes a task file containing
only the remaining tasks. The daily runner (bin/maa-online-daily-run) calls it
automatically before every retry.
Usage:
maa-resume-tasks.py <log> [<log> ...] [--source <daily.toml>] [--out <file>]
Output (stdout, last line):
RESUME_TASKS=<name1>,<name2> remaining tasks written to --out
NONE nothing left to do
ERROR:<reason> could not build a resume file (caller should
fall back to the full daily run)
StartUp is intentionally never skipped: every /start yields a fresh cloud
instance, and the global `startup = true` flag guarantees the game is launched
and sitting on the main screen before the remaining tasks run.
"""
from __future__ import annotations
import argparse
import os
import re
import sys
from pathlib import Path
ROOT = Path(os.environ.get("MAA_ONLINE_ROOT") or Path(__file__).resolve().parent.parent)
DEFAULT_SOURCE = Path(os.environ.get("MAA_ONLINE_DAILY_SOURCE", ROOT / "config/tasks/daily.toml"))
DEFAULT_OUT = Path(os.environ.get("MAA_ONLINE_RESUME_OUT", ROOT / "config/tasks/daily-resume.toml"))
TASK_HEADER = re.compile(r"^\[\[tasks\]\]\s*$")
TYPE_RE = re.compile(r'^type\s*=\s*"([^"]+)"\s*$', re.MULTILINE)
NAME_RE = re.compile(r'^name\s*=\s*"([^"]+)"\s*$', re.MULTILINE)
COMPLETED_RE = re.compile(r"^.*\b(\w+) Completed\s*$", re.MULTILINE)
# Task types that are never skipped even when a completion marker is present:
# a retry always reconnects to a fresh cloud instance and must relaunch/verify
# the game before the remaining tasks can run.
NEVER_SKIP = {"StartUp"}
def split_source(text: str) -> tuple[str, list[str]]:
"""Return (prelude, task blocks).
A block owns the contiguous comment lines directly above its [[tasks]]
header, so a "# 1. xxx" section comment travels with its task instead of
being left behind in the prelude. Blocks keep their exact formatting.
"""
lines = text.splitlines(keepends=True)
header_indexes = [i for i, line in enumerate(lines) if TASK_HEADER.match(line)]
if not header_indexes:
raise ValueError("no [[tasks]] blocks found in source file")
starts = []
for index in header_indexes:
start = index
while start - 1 >= 0 and lines[start - 1].lstrip().startswith("#"):
start -= 1
starts.append(start)
prelude = "".join(lines[: starts[0]])
blocks = []
for position, start in enumerate(starts):
end = starts[position + 1] if position + 1 < len(starts) else len(lines)
blocks.append("".join(lines[start:end]).strip("\n") + "\n")
return prelude, blocks
def completed_types(log_text: str) -> set[str]:
return set(COMPLETED_RE.findall(log_text))
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("logs", nargs="+", help="previous attempts' MAA log files")
parser.add_argument("--source", default=str(DEFAULT_SOURCE))
parser.add_argument("--out", default=str(DEFAULT_OUT))
args = parser.parse_args()
source = Path(args.source)
if not source.is_file():
print(f"ERROR:source task file not found: {source}")
return 0
try:
prelude, blocks = split_source(source.read_text(encoding="utf-8"))
except (OSError, ValueError) as exc:
print(f"ERROR:{exc}")
return 0
done: set[str] = set()
for log in args.logs:
log_path = Path(log)
if not log_path.is_file():
continue
try:
done |= completed_types(log_path.read_text(encoding="utf-8", errors="replace"))
except OSError as exc:
print(f"ERROR:cannot read previous log {log}: {exc}")
return 0
kept: list[str] = []
kept_names: list[str] = []
for block in blocks:
match = TYPE_RE.search(block)
if not match:
continue
task_type = match.group(1)
if task_type in done and task_type not in NEVER_SKIP:
continue
kept.append(block)
name_match = NAME_RE.search(block)
kept_names.append(name_match.group(1) if name_match else task_type)
if not kept:
print("NONE")
return 0
header = (
"# AUTO-GENERATED by bin/maa-resume-tasks.py -- do not edit.\n"
f"# Source: {source}\n"
f"# Already completed (skipped): {', '.join(sorted(done)) or 'none'}\n\n"
)
out = Path(args.out)
try:
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(
header + prelude.rstrip("\n") + "\n\n" + "\n".join(kept), encoding="utf-8"
)
except OSError as exc:
print(f"ERROR:cannot write resume file: {exc}")
return 0
print(f"RESUME_TASKS={','.join(kept_names)}")
return 0
if __name__ == "__main__":
sys.exit(main())
+14
View File
@@ -0,0 +1,14 @@
[Unit]
Description=MAA Online daily task (Arknights via NetEase cloud game)
Requires=maa-online-server.service
After=maa-online-server.service
[Service]
Type=oneshot
ExecStart=__ROOT__/bin/maa-online-daily-run
# A normal run takes about 25 minutes; the runner's own per-attempt watchdog
# aborts a hung session well before this ceiling.
TimeoutStartSec=7200
# Safety net: whatever stops the unit (success, failure, timeout), make sure
# the remote cloud session is released.
ExecStopPost=/bin/sh -c "/usr/bin/curl --noproxy '*' -fsS -X POST http://127.0.0.1:22888/exit || true"
+11
View File
@@ -0,0 +1,11 @@
[Unit]
Description=Run MAA Online daily at a random moment between 04:00 and 06:00
[Timer]
OnCalendar=*-*-* 04:00:00
RandomizedDelaySec=2h
Persistent=true
Unit=maa-daily.service
[Install]
WantedBy=timers.target
+16
View File
@@ -0,0 +1,16 @@
[Unit]
Description=MAA Online cloud game API server
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=__ROOT__/bin/maa-online-server
Restart=always
RestartSec=5
# Belt and braces: never let a proxy environment leak into the cloud gaming
# server (bin/maa-online-server already unsets these).
Environment=NO_PROXY=localhost,127.0.0.1,::1
[Install]
WantedBy=multi-user.target
+7
View File
@@ -0,0 +1,7 @@
[Unit]
Description=Weekly MAA Online maintenance (maa-cli, MaaCore and resource updates)
[Service]
Type=oneshot
ExecStart=__ROOT__/bin/maa-online-update
TimeoutStartSec=1800
+11
View File
@@ -0,0 +1,11 @@
[Unit]
Description=Weekly MAA Online maintenance, Sunday 03:20 (before the daily window)
[Timer]
OnCalendar=Sun *-*-* 03:20:00
RandomizedDelaySec=30m
Persistent=true
Unit=maa-update.service
[Install]
WantedBy=timers.target
+39
View File
@@ -195,3 +195,42 @@ curl -X POST http://127.0.0.1:22888/exit
- 领取每日/每周任务奖励;
- 领取所有邮件;
- 任务结束后退出云游戏。
## 自动定时与断点续跑
仓库提供 systemd 单元,可让日常任务每天自动执行(安装见根目录 README 的「自动定时执行」一节):
| 单元 | 作用 |
|---|---|
| `maa-online-server.service` | 云游戏 API 常驻服务,崩溃自动重启,启动前清除代理环境变量 |
| `maa-daily.timer` | 每天 04:00 起随机 0–2 小时触发 |
| `maa-update.timer` | 每周日 03:20 起随机 30 分钟,更新 maa-cli / MaaCore / 资源 |
`bin/maa-online-daily-run` 的流程:等待 API 就绪 → 规划剩余任务 → 启动云游戏 → 在停滞看门狗下
执行任务 → 失败则释放云会话并重试(默认最多 3 次)。
看门狗会在出现连续 `ScreencapFailed` 或日志中的 `Disconnected` 时中止当前尝试;后端本身也会在信令
WebSocket 被远端关闭、或视频流停滞超过 `MAA_ONLINE_STALE_LIMIT_SECONDS`(默认 120 秒)时主动拆掉
会话,避免云实例空挂。
**断点续跑**:第 2 次及之后的重试不再重跑整份 `daily.toml`。`bin/maa-resume-tasks.py` 读取前几次尝试
的日志,跳过已出现 `<任务类型> Completed` 的任务,只把剩余任务写入
`config/tasks/daily-resume.toml` 后执行。若所有任务都已完成,脚本直接判定成功,不再开新会话。
`StartUp` 永不跳过:每次连接都是新的云实例,必须先进入游戏。
手动补跑(例如某天中断后,用当天剩余额度把没做完的任务补上):
```bash
MAA_ONLINE_RESUME_FROM="artifacts/maa-daily-20260919-053237.log" bin/maa-online-daily-run
```
查看执行情况:
```bash
systemctl list-timers maa-daily.timer
journalctl -u maa-daily.service -n 50
ls -lt artifacts/maa-daily-*.log | head
```
注意:网易云游戏免费额度有限(常见为每天 30 分钟),而一次完整日常约需 24 分钟,重试次数越多越容易
额度耗尽。断点续跑正是为压缩重试开销而设计的,但仍建议关注额度消耗。