72602

Scope

This section is the single source of truth for 72602 cluster operations.

Topology

Public ECS: 47.110.67.161 (2C4G, Aliyun Hongkong)
Active ingress domain: 72602.space; legacy .72602.online routes are retired
ArgoCD host: argocd.72602.space
k3s node: 72602-minipc (192.168.0.25, MiniPC N100 28G+1TB NVMe)
SSH reverse tunnel: :10021 (main), :10022 (backup, also carries 80/443)
Ingress NodePort: 32080 (HTTP), 32443 (HTTPS)
Ingress class: nginx
Ingress namespace: basic-components
cert-manager issuer: lets-encrypt
Storage class: local-path (default, RWO)
OS: Ubuntu 26.04 LTS (minipc)
k3s version: v1.34.6+k3s1 (installed via install.sh)

Traffic Path

Internet -> ECS(sshd reverse tunnel) -> minipc:k3s ingress-nginx

ECS Port Forwarding

All traffic reaches ECS via SSH reverse tunnel established from minipc. ECS side ports:

10021 -> minipc:22 (72602 main SSH)
10022 -> minipc:22 (72602 backup SSH, also carries 80→32080, 443→32443)
80 -> minipc:32080 (HTTP via SSH reverse tunnel on 10022)
443 -> minipc:32443 (HTTPS via SSH reverse tunnel on 10022)

DNS Setup

Active service records use 72602.space and point to 47.110.67.161:

Host	Type	Value	Service
`argocd.72602.space`	A	`47.110.67.161`	ArgoCD UI
`ops.docs.72602.space`	A	`47.110.67.161`	Hugo Docs
`sub2api.72602.space`	A	`47.110.67.161`	AI API proxy
`port.72602.space`	A	`47.110.67.161`	Homepage dashboard
`n8n.72602.space`	A	`47.110.67.161`	N8N workflow
`webhook.n8n.72602.space`	A	`47.110.67.161`	N8N webhook receiver
`ops.agent.72602.space`	A	`47.110.67.161`	OpenCode operations agent
`uptime.72602.space`	A	`47.110.67.161`	Uptime Kuma
`clash.72602.space`	A	`47.110.67.161`	Clash/mihomo panel
`api.minio.72602.space`	A	`47.110.67.161`	MinIO S3 API
`console.minio.72602.space`	A	`47.110.67.161`	MinIO Console

txt2img.agent.72602.online is retired. Its DNS record, certificate, TLS Secret, and unreferenced ai data claims have been removed.

DNS is managed via Cloudflare / Aliyun DNS (add A record → ECS IP).

Deployed ArgoCD Apps

App	Namespace	Type	Source	Ingress
argocd	argocd	Helm (arco-cd)	argo-cd 9.5.4	argocd.72602.space
cert-manager	basic-components	Helm (Jetstack)	cert-manager 1.20.2	internal
ingress-nginx	basic-components	Helm	ingress-nginx 4.15.1	shared ingress controller
ops-docs	application	manifests (Git)	docs.git/main	ops.docs.72602.space
homepage	monitor	manifests (Git)	docs.git/main	port.72602.space
uptime-kuma	monitor	manifests (Git)	docs.git/main	uptime.72602.space
sub2api	application	Helm (ghcr)	sub2api 0.1.1	sub2api.72602.space
postgresql	database	Helm (Bitnami)	postgresql 18.1.8	internal
redis-shared	storage	Helm (Bitnami)	redis 18.16.0	internal
minio	storage	Helm	minio 16.0.10	console.minio.72602.space, api.minio.72602.space
n8n	n8n	Helm (community)	n8n 1.16.36	n8n.72602.space, webhook.n8n.72602.space

Non-ArgoCD (手动部署)

Deployment	Namespace	Image	Ingress
ops-agent	application	ay-dev/ops-agent:0.2.2	ops.agent.72602.space

Network Proxy

Egress Proxy Architecture

k8s Pod (10.42.x.x) --HTTP_PROXY--> 192.168.0.25:17890 (socat) --forward--> 127.0.0.1:7890 (mihomo/clash) --tunnel--> upstream proxies

mihomo (clash): listens on 127.0.0.1:7890 (HTTP), 127.0.0.1:7891 (SOCKS5)
- Config: /home/aaron/clashctl/resources/runtime.yaml
- Key setting: allow-lan: false (只监听 localhost)
socat bridge: 0.0.0.0:17890 → 127.0.0.1:7890 (桥接使 k8s Pod 可达)
- 进程: socat -d -d TCP-LISTEN:17890,fork,reuseaddr,bind=0.0.0.0 TCP:127.0.0.1:7890
k8s Service: argocd-egress-proxy.argocd.svc.cluster.local (ClusterIP: 10.43.42.223:17890) → Host 192.168.0.25:17890
App proxy env: 应统一使用 http://192.168.0.25:17890（不是 192.168.0.25:7890，因为 mihomo 仅绑定 127.0.0.1）

关键约束

mihomo allow-lan: false 意味着不能直接用 192.168.0.25:7890 作为代理地址
必须通过 socat 桥接 (192.168.0.25:17890) 或 argocd-egress-proxy Service 访问
GitHub acceleration: ghfast.top URL rewrite + NO_PROXY bypass
Image mirror: m.daocloud.io/docker.io, m.daocloud.io/ghcr.io

Known Incident Pattern

Symptom: HTTPS handshake fails for argocd.72602.online (tls alert internal error).
Root cause: ECS Docker/derper occupies public 443, traffic never reaches k3s ingress.
Fix baseline: derper must expose 8443:443, keep public 443 for ingress NodePort 32443.
Symptom: n8n 所有 workflow 报 connect ECONNREFUSED 192.168.0.25:7890。
Root cause: HTTP_PROXY 指向 192.168.0.25:7890，但 mihomo 只监听 127.0.0.1:7890（allow-lan: false）。Pod 无法直连 mihomo 的 LAN IP。
Fix baseline: HTTP_PROXY/HTTPS_PROXY 必须使用 socat 桥接端口 192.168.0.25:17890（或 k8s Service 10.43.19.4:17890），该端口由 socat 转发至 127.0.0.1:7890。

Host-Level Services

Service	Port	Bind	Description
mihomo (clash) HTTP proxy	`7890`	`127.0.0.1`	Egress proxy, `allow-lan: false`
mihomo (clash) SOCKS5	`7891`	`127.0.0.1`	SOCKS5 proxy
mihomo external controller	`9090`	`0.0.0.0`	Clash API/UI, exposed via `clash.72602.space`
socat bridge	`17890`	`0.0.0.0`	Forwards to `127.0.0.1:7890`, k8s pod accessible
autossh tunnel (main)	`10021`→ECS	-	Reverse tunnel to ECS
autossh tunnel (backup)	`10022→ECS`	-	Reverse tunnel + HTTP/HTTPS forwarding
k3s ingress HTTP	`32080`	`0.0.0.0`	NodePort for ingress HTTP
k3s ingress HTTPS	`32443`	`0.0.0.0`	NodePort for ingress HTTPS

Notes

Keep derper away from public 443 (use 8443).
Keep app ingress aligned with ArgoCD ingress pattern:
- ingressClassName: nginx
- cert-manager.io/cluster-issuer: lets-encrypt
- TLS secret per host.
argocd-egress-proxy 由 ops-docs ArgoCD Application 管理，并为 repo-server 提供 Git/Helm 出站代理。
mihomo allow-lan: false 意味着 Pod 代理地址必须是 socat 桥接端口 17890，不能用 7890。
SSH 隧道依赖 loginctl enable-linger 保持用户级 systemd 服务运行。

Recent Operations

2026-07-16: reset `csst` and update N8N webhook host

Deleted and recreated the csst namespace. Only the namespace default ServiceAccount and kube-root-ca.crt ConfigMap remain.
Changed N8N WEBHOOK_URL, webhook worker URL, Ingress rule, and TLS DNS name from webhook.72602.online to webhook.n8n.72602.online.
Synced ArgoCD application argocd/n8n; main, webhook, MCP webhook, and worker rollouts completed.
cert-manager completed HTTP-01 validation and issued the updated certificate.

2026-07-16: migrate OpenCode web to k3s

Replaced the host systemd process and static EndpointSlice with the application/ops-agent workload.
The Pod mounts /home/aaron/Ops/docs at /workspace, loads the project .opencode/opencode.json, and persists sessions in the opencode-data PVC.
Image ay-dev/ops-agent:0.2.2 uses glibc and contains OpenCode 1.18.2, kubectl 1.34.6, Argo CD CLI 3.3.8, VibeGuard, DCP, and Goal Mode.
OpenCode native Basic Auth protects both Ingress and cluster-internal access. Anonymous HTTPS returns 401; authenticated HTTPS returns 200.
An Nginx sidecar publishes the Ops Agent browser title and proxies Terminal WebSocket and event streams.

2026-07-16: remove Langfuse and refresh Homepage

Permanently removed the seven unused Langfuse PVCs (56 GiB) and six residual Secrets from monitor.
Removed Langfuse and pgAdmin from Homepage and added the OpenCode operations agent.
Restored the ArgoCD Homepage widget by binding the readonly API account to role:readonly.

2026-07-17: align Ops resource names

Renamed the Hugo workload and its Service, ConfigMap, build Job, and Ingress resources to ops-docs; retained hugo-docs-pvc to preserve generated content.
Renamed the OpenCode-based workload, Service, Ingress, proxy ConfigMap, manifest directory, and Dockerfile to ops-agent; retained existing opencode-* PVC and Secrets to preserve sessions and credentials.
Replaced the manual local-proxy-bridge with the GitOps-managed argocd-egress-proxy; ArgoCD repo-server uses its cluster Service while applications can continue using host port 17890.

Default Verification Commands

# 公网入口
curl -vI http://argocd.72602.space
curl -vkI https://argocd.72602.space

# k8s 资源
kubectl get ingress -A -o wide
kubectl get svc -A -o wide
kubectl get pods -A -o wide
kubectl get certificate,certificaterequest,order,challenge -A

# 主机端口
sudo ss -lntp | grep -E ':80|:443|:8443|:32080|:32443|:7890|:17890|:9090'

# ECS 转发
sudo iptables -t nat -L PREROUTING -n -v --line-numbers
sudo iptables -t nat -L DOCKER -n -v --line-numbers

# Egress proxy 完整性
curl -s --connect-timeout 3 -x http://127.0.0.1:17890 http://httpbin.org/ip
kubectl exec -n n8n deploy/n8n -- env | grep -E 'HTTP_PROXY|HTTPS_PROXY'

# SSH 隧道
journalctl --user -u reverse-tunnel-ecs-10021.service --since "1 hour ago" --no-pager
journalctl --user -u reverse-tunnel-ecs-10022.service --since "1 hour ago" --no-pager

ECS Security Group

安全组 IP 自动更新

背景

72602-minipc 的 ISP 不定期更换公网 IP，而阿里云 ECS (ecs-99) 安全组限制了 SSH 端口只能从特定 IP 访问。

当公网 IP 变化时：

SSH 反向隧道断开
无法通过 ssh aaron@47.110.67.161 -p 10022 访问
无法直接 ssh root@47.110.67.161

解决方案

定时检测公网 IP，变化时自动更新阿里云安全组规则。

所有阿里云安全组和 AliDNS 操作统一从 72602-minipc 执行。

工作原理

每 5 分钟 ──> 获取公网 IP (ifconfig.me → ip.sb → icanhazip.com)
                 │
                 ├── 全部失败 ──> 钉钉告警，退出
                 │
                 └── 获取成功 ──> 与上次对比
                                 ├── 未变化 ──> 退出
                                 └── 已变化 ──> 更新安全组规则 → 钉钉通知

受影响的端口

端口	用途
22	ECS SSH 直接连接、反向隧道
10021	预留
10022	SSH 反向隧道（72602-minipc 入口）

文件位置

文件	说明
`/home/aaron/bin/update-sg-ip.sh`	主脚本
`/home/aaron/.aliyun-keys`	阿里云 AccessKey（chmod 600）
`/etc/systemd/system/update-sg-ip.service`	systemd 服务
`/etc/systemd/system/update-sg-ip.timer`	systemd 定时器（每 5 分钟，开机 30 秒后首次触发）
`/tmp/last_public_ip`	上次公网 IP 缓存

可恢复备份

脱敏后的可恢复备份位于私有仓库 /home/aaron/Ops/ops-private，对应提交 8abb81d，包含：

scripts/update-sg-ip.sh
systemd service、timer 和 env.example
runbooks/72602/security-group-updater.md
.gitignore

该提交未包含 AccessKey、钉钉 token 或真实安全组 ID。备份操作未修改现网脚本、systemd units 或 timer。

AliDNS 环境

官方 AliDNS SDK 使用独立虚拟环境 /home/aaron/.local/venvs/alidns。
SDK 或依赖需要下载时，使用 HTTP 代理 http://192.168.0.25:17890。
当前凭证具备 72602.online 区域的 AliDNS 记录管理能力，也具备 ECS 安全组变更能力。
DNS 变更前应限定目标区域和记录，并在变更后分别执行权威 DNS 与公共 DNS 验证。

常用命令

# 查看定时器状态
systemctl status update-sg-ip.timer

# 手动触发一次更新
sudo systemctl start update-sg-ip.service

# 查看执行日志
journalctl -u update-sg-ip.service -f

# 查看脚本输出
cat /tmp/update-sg-ip.log

# 手动运行脚本
~/bin/update-sg-ip.sh

钉钉通知

脚本支持钉钉机器人通知。需要设置 DING_TOKEN 环境变量或修改脚本中的 DING_TOKEN 变量：

# 在 ~/.aliyun-keys 中添加：
DING_TOKEN=你的钉钉机器人access_token

凭证安全

阿里云 AccessKey 存储在 /home/aaron/.aliyun-keys，权限 600。当前同一 AccessKey 同时具备 ECS 安全组和 AliDNS 变更权限；后续应拆分为两个最小权限 RAM 身份。 /home/aaron/bin/update-sg-ip.sh 当前权限为 0775，仍可由组写入；应收紧为 0755。此项为待执行的加固建议，并非已完成状态。定期轮换 AccessKey。AccessKey 获取：阿里云控制台 → 头像 → AccessKey 管理。

SSH Tunnel

72602 隧道统一采用双入口反向隧道：

主入口：10021
备入口：10022
目标 ECS：47.110.67.161 (ecs-99)

快速连接命令：

ssh -p 10021 aaron@47.110.67.161
ssh -p 10022 aaron@47.110.67.161

上线顺序建议：

先在 72602-minipc 创建并启动 10022（备入口）
验证 ECS 已监听 10022
再创建并启动 10021（主入口）
最后做外网双端口连通性验证

完整步骤、故障恢复与运维命令见子页面。

72602-minipc → ecs-99

72602-minipc → ecs-99

SSH 反向隧道：72602-minipc → ecs-99（双入口）

本文档是 72602-minipc 的新标准方案，目标是避免单端口掉线导致完全失联。

主入口：10021
备入口：10022
两个端口由两个独立 service 维护

一、架构

外网任意机器                          ecs-99 (47.110.67.161)                     72602-minipc (192.168.0.25)
ssh -p 10021 aaron@47.110.67.161  ->   0.0.0.0:10021 (sshd) --SSH reverse-->      localhost:22
ssh -p 10022 aaron@47.110.67.161  ->   0.0.0.0:10022 (sshd) --SSH reverse-->      localhost:22

说明：反向隧道必须由 72602-minipc 主动发起。ECS 上看到端口监听，才表示隧道在线。

二、上线前检查

2.1 在 72602-minipc 检查基础条件

# 1) 本机 SSH 服务
sudo systemctl is-active ssh

# 2) autossh 是否安装
autossh -V

# 3) 本机到 ECS 网络与认证
ssh -o ConnectTimeout=5 root@47.110.67.161 hostname
# 期望输出: ecs-99

2.2 在 ECS 检查前置配置

/etc/ssh/sshd_config 至少包含：

GatewayPorts clientspecified

重载：

sudo systemctl reload sshd

安全组放行：10021/tcp、10022/tcp（授权范围按你自己的安全策略）。

同时确认 ECS 本机防火墙（UFW）放行这两个端口：

sudo ufw status numbered
# 至少应包含 10021/tcp 和 10022/tcp 的 ALLOW 规则

三、创建双 service（72602-minipc 上执行）

下面步骤全部在 72602-minipc 上执行。

3.1 统一 SSH 客户端配置（可选但推荐）

编辑 ~/.ssh/config：

Host ecs-99
    HostName 47.110.67.161
    User root
    ServerAliveInterval 60
    ServerAliveCountMax 3
    ExitOnForwardFailure yes
    TCPKeepAlive yes
    ConnectTimeout 10

3.2 创建 systemd 用户服务目录

mkdir -p ~/.config/systemd/user

3.3 新建 service（10021 主）

文件：~/.config/systemd/user/reverse-tunnel-ecs-10021.service

[Unit]
Description=Reverse SSH tunnel to ecs-99 (port 10021 -> local SSH)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment="AUTOSSH_GATETIME=0"
Environment="AUTOSSH_POLL=60"
Environment="AUTOSSH_FIRST_POLL=30"
ExecStart=/usr/bin/autossh -M 0 -N -R 0.0.0.0:10021:localhost:22 ecs-99
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=default.target

3.4 新建 service（10022 备，含 HTTP/HTTPS）

文件：~/.config/systemd/user/reverse-tunnel-ecs-10022.service

[Unit]
Description=Reverse SSH tunnel to ecs-99 (port 10022 -> local SSH)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment="AUTOSSH_GATETIME=0"
Environment="AUTOSSH_POLL=60"
Environment="AUTOSSH_FIRST_POLL=30"
ExecStart=/usr/bin/autossh -M 0 -N \
  -R 0.0.0.0:10022:localhost:22 \
  -R 0.0.0.0:80:localhost:32080 \
  -R 0.0.0.0:443:localhost:32443 \
  ecs-99
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=default.target

说明：10022 的 service 额外承载 72602-minipc 的 HTTP（32080→80）和 HTTPS（32443→443）服务。如果不需要公网 Web 服务，可只保留 SSH 转发。

3.5 启用并启动

export XDG_RUNTIME_DIR=/run/user/$(id -u)

systemctl --user daemon-reload
systemctl --user enable --now reverse-tunnel-ecs-10021.service
systemctl --user enable --now reverse-tunnel-ecs-10022.service

systemctl --user status reverse-tunnel-ecs-10021.service --no-pager
systemctl --user status reverse-tunnel-ecs-10022.service --no-pager

3.6 防登出失效（强烈建议）

sudo loginctl enable-linger aaron
loginctl show-user aaron | grep Linger
# 期望: Linger=yes

四、连通性验证

4.1 在 ECS 上看监听

ssh root@47.110.67.161 "ss -tlnp | grep -E '10021|10022'"

期望看到：

0.0.0.0:10021
0.0.0.0:10022

4.2 在外网验证

nc -zv 47.110.67.161 10021
nc -zv 47.110.67.161 10022

ssh -p 10021 aaron@47.110.67.161
ssh -p 10022 aaron@47.110.67.161

若 10021/10022 外网超时，但 ECS 上已监听，优先检查：

云安全组来源网段是否覆盖当前出口 IP
ECS UFW 是否放行对应端口

五、故障恢复（按顺序）

场景 A：ECS 本机 `ssh localhost -p 10022` 失败

这说明 ECS 上没有监听 10022，问题几乎总在源机器（72602-minipc）侧。

在 72602-minipc 执行：

export XDG_RUNTIME_DIR=/run/user/$(id -u)

# 1) 看 service
systemctl --user status reverse-tunnel-ecs-10021.service --no-pager
systemctl --user status reverse-tunnel-ecs-10022.service --no-pager

# 2) 重启 service
systemctl --user restart reverse-tunnel-ecs-10021.service
systemctl --user restart reverse-tunnel-ecs-10022.service

# 3) 看日志
journalctl --user -u reverse-tunnel-ecs-10021.service --since "10 min ago" --no-pager
journalctl --user -u reverse-tunnel-ecs-10022.service --since "10 min ago" --no-pager

# 4) 验证到 ECS 的基础连通
ssh -o ConnectTimeout=5 root@47.110.67.161 echo ok

场景 B：外网超时但 ECS 本机可通

问题在安全组或 ECS 防火墙，不在隧道本身。

ssh root@47.110.67.161 "ss -tlnp | grep -E '10021|10022'"
ssh root@47.110.67.161 "iptables -L INPUT -n | grep -E '10021|10022' || true"

场景 C：重启后隧道没起来

export XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user is-enabled reverse-tunnel-ecs-10021.service
systemctl --user is-enabled reverse-tunnel-ecs-10022.service
loginctl show-user aaron | grep Linger

六、常用运维命令

export XDG_RUNTIME_DIR=/run/user/$(id -u)

# 重载并重启
systemctl --user daemon-reload
systemctl --user restart reverse-tunnel-ecs-10021.service
systemctl --user restart reverse-tunnel-ecs-10022.service

# 停止
systemctl --user stop reverse-tunnel-ecs-10021.service
systemctl --user stop reverse-tunnel-ecs-10022.service

# 日志实时跟踪
journalctl --user -u reverse-tunnel-ecs-10021.service -f
journalctl --user -u reverse-tunnel-ecs-10022.service -f

七、和现网监控的关系

ECS 侧巡检仅监控 72602-minipc 的两个公开入口。

当 72602-minipc 双入口上线后，ECS 无需改架构，只要确认：

安全组已放行 10021/10022
/etc/tunnel-healthcheck-ports.conf 包含 10021、10022
/etc/tunnel-healthcheck.env 企业应用参数可用（DINGTALK_CLIENT_ID/SECRET/AGENT_ID/USER_IDS）

当前线上实现：

告警通道为钉钉企业应用 API（非 webhook）
告警消息为 Markdown 格式（标题：🚨 ECS Tunnel Alert）
连续失败 3 次才发送告警（防抖）