feat: 增加内存观测日志
- bin/maa-online-memlog: 每分钟采集一次服务 cgroup 匿名内存、swap 占用、进程 RSS、 以及大于 32 MB 的匿名映射数量（glibc arena 膨胀指纹），追加写入 logs/memlog.csv， 超过 5 MB 自动滚动并保留一代；同时输出一行到 journald - systemd/maa-online-memlog.service, maa-online-memlog.timer: 每分钟触发采集， Nice=10 避免影响流媒体负载 - bin/maa-online-install-systemd: 安装并启用观测定时器，安装提示补充内存日志路径 - README.md, user_config/README.md: 补充内存观测说明、字段表与查看命令 - .gitignore: 忽略运行期产生的 logs/
This commit is contained in:
RRRosmontis
@@ -16,6 +16,7 @@ user_config/daily.toml
|
||||
user_config/backups/
|
||||
tests/fake-actions.jsonl
|
||||
artifacts/
|
||||
logs/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
|
||||
@@ -149,6 +149,7 @@ sudo bin/maa-online-install-systemd
|
||||
| `maa-online-server.service` | 云游戏 API 常驻服务（`Restart=always`，启动前清除代理环境变量，限制 malloc arena 数量） |
|
||||
| `maa-daily.timer` | 每天 04:00 起随机 0–2 小时触发日常任务 |
|
||||
| `maa-online-recycle.timer` | 每天 03:59（游戏每日刷新前）重启 API 服务，回收内存 |
|
||||
| `maa-online-memlog.timer` | 每分钟把服务内存/swap/arena 指标写入 `logs/memlog.csv` |
|
||||
| `maa-update.timer` | 每周日 03:20 起随机 30 分钟执行更新 |
|
||||
|
||||
日常任务由 `bin/maa-online-daily-run` 驱动：等待 API 就绪 → 启动云游戏 → 在停滞看门狗下执行
|
||||
@@ -168,11 +169,24 @@ sudo bin/maa-online-install-systemd
|
||||
查看运行情况：
|
||||
|
||||
```bash
|
||||
systemctl list-timers maa-daily.timer maa-update.timer maa-online-recycle.timer
|
||||
systemctl list-timers 'maa-*'
|
||||
journalctl -u maa-daily.service -n 50
|
||||
journalctl -u maa-online-server.service -n 50
|
||||
```
|
||||
|
||||
**内存观测**：`maa-online-memlog.timer` 每分钟运行 `bin/maa-online-memlog`，把服务 cgroup 的
|
||||
匿名内存、swap 占用、以及大于 32 MB 的匿名映射数量（arena 膨胀指纹）追加到 `logs/memlog.csv`
|
||||
（超过 5 MB 自动滚动，保留一代）。带表头的字段见脚本内的 `FIELDS`。用法示例：
|
||||
|
||||
```bash
|
||||
tail -f logs/memlog.csv # 实时观察
|
||||
awk -F, 'NR>1 && $11>0 {print $1, $10, $11, $12"MB"}' logs/memlog.csv # 只看出现大 arena 的时刻
|
||||
```
|
||||
|
||||
CSV 表头：`timestamp, cloud_state, pid, proc_uptime_s, cg_current_mb, cg_peak_mb, cg_swap_mb,
|
||||
cg_swap_peak_mb, rss_mb, anon_regions, big_anon_regions, anon_total_mb, sys_mem_avail_mb,
|
||||
sys_swap_used_mb, sys_swap_free_mb`。
|
||||
|
||||
## 离线测试
|
||||
|
||||
启动假后端：
|
||||
|
||||
@@ -7,7 +7,7 @@ set -euo pipefail
|
||||
ROOT="${MAA_ONLINE_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}"
|
||||
SRC="$ROOT/systemd"
|
||||
DST="${MAA_ONLINE_SYSTEMD_DIR:-/etc/systemd/system}"
|
||||
UNITS=(maa-online-server.service maa-daily.service maa-daily.timer maa-update.service maa-update.timer maa-online-recycle.service maa-online-recycle.timer)
|
||||
UNITS=(maa-online-server.service maa-daily.service maa-daily.timer maa-update.service maa-update.timer maa-online-recycle.service maa-online-recycle.timer maa-online-memlog.service maa-online-memlog.timer)
|
||||
|
||||
if [ "$(id -u)" != "0" ]; then
|
||||
echo "error: must run as root (writes to $DST)" >&2
|
||||
@@ -25,15 +25,16 @@ done
|
||||
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now maa-online-server.service
|
||||
systemctl enable maa-daily.timer maa-update.timer maa-online-recycle.timer
|
||||
systemctl start maa-daily.timer maa-update.timer maa-online-recycle.timer
|
||||
systemctl enable maa-daily.timer maa-update.timer maa-online-recycle.timer maa-online-memlog.timer
|
||||
systemctl start maa-daily.timer maa-update.timer maa-online-recycle.timer maa-online-memlog.timer
|
||||
|
||||
echo
|
||||
systemctl --no-pager list-timers maa-daily.timer maa-update.timer maa-online-recycle.timer
|
||||
systemctl --no-pager list-timers maa-daily.timer maa-update.timer maa-online-recycle.timer maa-online-memlog.timer
|
||||
cat <<EOF
|
||||
|
||||
MAA Online systemd units are active.
|
||||
server log: journalctl -u maa-online-server -f
|
||||
daily log: journalctl -u maa-daily -n 50
|
||||
next runs: systemctl list-timers maa-daily.timer maa-update.timer maa-online-recycle.timer
|
||||
memory log: tail -f $ROOT/logs/memlog.csv
|
||||
next runs: systemctl list-timers 'maa-*'
|
||||
EOF
|
||||
|
||||
Executable
+209
@@ -0,0 +1,209 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Continuously record MAA Online memory/swap behaviour into a log.
|
||||
|
||||
Run once per minute by maa-online-memlog.timer. Each run appends one row to
|
||||
<ROOT>/logs/memlog.csv (auto-rotated at 5 MB, one generation kept) and prints a
|
||||
compact line to the journal.
|
||||
|
||||
The numbers tracked here are the ones that matter for the glibc malloc arena
|
||||
behaviour: the service cgroup's anonymous memory, how much of it has been
|
||||
pushed into swap, and how many large (>= 32 MB) anonymous mappings the process
|
||||
holds. An arena-bloat regression shows up as the big-region count climbing
|
||||
while the service is streaming.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(os.environ.get("MAA_ONLINE_ROOT") or Path(__file__).resolve().parent.parent)
|
||||
LOG_DIR = Path(os.environ.get("MAA_ONLINE_LOG_DIR", ROOT / "logs"))
|
||||
LOG = Path(os.environ.get("MAA_ONLINE_MEMLOG", LOG_DIR / "memlog.csv"))
|
||||
MAX_BYTES = int(os.environ.get("MAA_ONLINE_MEMLOG_MAX_BYTES", 5 * 1024 * 1024))
|
||||
|
||||
CG = Path("/sys/fs/cgroup/system.slice/maa-online-server.service")
|
||||
API = os.environ.get("MAA_ONLINE_API", "http://127.0.0.1:22888")
|
||||
|
||||
FIELDS = [
|
||||
"timestamp",
|
||||
"cloud_state",
|
||||
"pid",
|
||||
"proc_uptime_s",
|
||||
"cg_current_mb",
|
||||
"cg_peak_mb",
|
||||
"cg_swap_mb",
|
||||
"cg_swap_peak_mb",
|
||||
"rss_mb",
|
||||
"anon_regions",
|
||||
"big_anon_regions",
|
||||
"anon_total_mb",
|
||||
"sys_mem_avail_mb",
|
||||
"sys_swap_used_mb",
|
||||
"sys_swap_free_mb",
|
||||
]
|
||||
|
||||
MIB = 1048576
|
||||
|
||||
|
||||
def read_text(path: Path, default: str = "") -> str:
|
||||
try:
|
||||
return path.read_text(encoding="utf-8").strip()
|
||||
except OSError:
|
||||
return default
|
||||
|
||||
|
||||
def cg_value(name: str) -> int:
|
||||
raw = read_text(CG / name, "0")
|
||||
try:
|
||||
return int(raw)
|
||||
except ValueError:
|
||||
return 0
|
||||
|
||||
|
||||
def mb(value: int) -> float:
|
||||
return round(value / MIB, 1)
|
||||
|
||||
|
||||
def main_pid() -> int:
|
||||
try:
|
||||
out = subprocess.run(
|
||||
["systemctl", "show", "-p", "MainPID", "--value", "maa-online-server.service"],
|
||||
capture_output=True, text=True, timeout=10, check=False,
|
||||
).stdout.strip()
|
||||
return int(out or 0)
|
||||
except (OSError, ValueError, subprocess.SubprocessError):
|
||||
return 0
|
||||
|
||||
|
||||
def proc_uptime(pid: int) -> int:
|
||||
stat = read_text(Path(f"/proc/{pid}/stat"))
|
||||
if not stat:
|
||||
return 0
|
||||
try:
|
||||
start_ticks = int(stat.rsplit(")", 1)[1].split()[19])
|
||||
except (IndexError, ValueError):
|
||||
return 0
|
||||
try:
|
||||
with open("/proc/uptime") as handle:
|
||||
uptime = float(handle.read().split()[0])
|
||||
except (OSError, ValueError):
|
||||
return 0
|
||||
return int(uptime - start_ticks / os.sysconf("SC_CLK_TCK"))
|
||||
|
||||
|
||||
def proc_memory(pid: int) -> tuple[float, int, int, float]:
|
||||
"""Return (rss_mb, anon_regions, big_anon_regions, anon_total_mb)."""
|
||||
rss = anon_regions = big_anon_regions = 0
|
||||
anon_total = 0
|
||||
try:
|
||||
with open(f"/proc/{pid}/smaps") as handle:
|
||||
for line in handle:
|
||||
if line[0] in "0123456789abcdef" and "-" in line.split(" ", 1)[0]:
|
||||
parts = line.split()
|
||||
start, end = parts[0].split("-")
|
||||
size = (int(end, 16) - int(start, 16)) // 1024
|
||||
path = parts[5] if len(parts) > 5 else "[anon]"
|
||||
if path == "[anon]":
|
||||
anon_regions += 1
|
||||
anon_total += size
|
||||
if size >= 32 * 1024:
|
||||
big_anon_regions += 1
|
||||
elif line.startswith("Rss:"):
|
||||
rss += int(line.split()[1])
|
||||
except OSError:
|
||||
pass
|
||||
return round(rss / 1024, 1), anon_regions, big_anon_regions, round(anon_total / 1024, 1)
|
||||
|
||||
|
||||
def system_memory() -> tuple[float, float, float]:
|
||||
values: dict[str, int] = {}
|
||||
try:
|
||||
with open("/proc/meminfo") as handle:
|
||||
for line in handle:
|
||||
key, _, rest = line.partition(":")
|
||||
values[key] = int(rest.split()[0])
|
||||
except (OSError, ValueError, IndexError):
|
||||
return 0.0, 0.0, 0.0
|
||||
return (
|
||||
mb(values.get("MemAvailable", 0) * 1024),
|
||||
mb((values.get("SwapTotal", 0) - values.get("SwapFree", 0)) * 1024),
|
||||
mb(values.get("SwapFree", 0) * 1024),
|
||||
)
|
||||
|
||||
|
||||
def cloud_state() -> str:
|
||||
try:
|
||||
with urllib.request.urlopen(f"{API}/info", timeout=5) as response:
|
||||
body = response.read().decode("utf-8", "replace")
|
||||
except (urllib.error.URLError, OSError, ValueError):
|
||||
return "unreachable"
|
||||
match = re.search(r'"status"\s*:\s*"([^"]+)"', body)
|
||||
return match.group(1) if match else "unknown"
|
||||
|
||||
|
||||
def rotate_if_needed() -> None:
|
||||
try:
|
||||
if LOG.exists() and LOG.stat().st_size > MAX_BYTES:
|
||||
backup = LOG.with_name(LOG.name + ".1")
|
||||
backup.unlink(missing_ok=True)
|
||||
LOG.rename(backup)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def main() -> int:
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
rotate_if_needed()
|
||||
|
||||
pid = main_pid()
|
||||
state = cloud_state()
|
||||
if pid:
|
||||
rss, anon_regions, big_regions, anon_total = proc_memory(pid)
|
||||
uptime = proc_uptime(pid)
|
||||
else:
|
||||
rss = anon_total = 0.0
|
||||
anon_regions = big_regions = uptime = 0
|
||||
|
||||
mem_avail, swap_used, swap_free = system_memory()
|
||||
row = [
|
||||
time.strftime("%Y-%m-%dT%H:%M:%S%z"),
|
||||
state,
|
||||
pid,
|
||||
uptime,
|
||||
mb(cg_value("memory.current")),
|
||||
mb(cg_value("memory.peak")),
|
||||
mb(cg_value("memory.swap.current")),
|
||||
mb(cg_value("memory.swap.peak")),
|
||||
rss,
|
||||
anon_regions,
|
||||
big_regions,
|
||||
anon_total,
|
||||
mem_avail,
|
||||
swap_used,
|
||||
swap_free,
|
||||
]
|
||||
|
||||
fresh = not LOG.exists() or LOG.stat().st_size == 0
|
||||
with LOG.open("a", encoding="utf-8") as handle:
|
||||
if fresh:
|
||||
handle.write(",".join(FIELDS) + "\n")
|
||||
handle.write(",".join(str(value) for value in row) + "\n")
|
||||
|
||||
print(
|
||||
f"{row[0]} state={state} pid={pid} "
|
||||
f"cg={row[4]}MB peak={row[5]}MB cg_swap={row[6]}MB "
|
||||
f"rss={rss}MB anon_regions={anon_regions} big={big_regions} anon={anon_total}MB "
|
||||
f"mem_avail={mem_avail}MB swap_used={swap_used}MB"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,8 @@
|
||||
[Unit]
|
||||
Description=Sample MAA Online server memory/swap into a CSV log
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=__ROOT__/bin/maa-online-memlog
|
||||
# Pure observation: keep it from competing with the streaming workload.
|
||||
Nice=10
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Sample MAA Online server memory every minute
|
||||
|
||||
[Timer]
|
||||
OnBootSec=1min
|
||||
OnUnitActiveSec=1min
|
||||
AccuracySec=5s
|
||||
Persistent=false
|
||||
Unit=maa-online-memlog.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -205,6 +205,7 @@ curl -X POST http://127.0.0.1:22888/exit
|
||||
| `maa-online-server.service` | 云游戏 API 常驻服务，崩溃自动重启，启动前清除代理环境变量，限制 malloc arena 数量 |
|
||||
| `maa-daily.timer` | 每天 04:00 起随机 0–2 小时触发 |
|
||||
| `maa-online-recycle.timer` | 每天 03:59（游戏每日刷新前）重启 API 服务以回收内存 |
|
||||
| `maa-online-memlog.timer` | 每分钟记录服务内存/swap/arena 指标到 `logs/memlog.csv` |
|
||||
| `maa-update.timer` | 每周日 03:20 起随机 30 分钟，更新 maa-cli / MaaCore / 资源 |
|
||||
|
||||
`bin/maa-online-daily-run` 的流程：等待 API 就绪 → 规划剩余任务 → 启动云游戏 → 在停滞看门狗下
|
||||
|
||||
Reference in New Issue
Block a user