72602-minipc → ecs-99

SSH 反向隧道：72602-minipc → ecs-99（双入口）

本文档是 72602-minipc 的当前参考方案；执行前应核对 live unit、ECS 安全组、 SSH banner 和监控状态，目标是避免单端口掉线导致完全失联。

• 主入口：10021
• 备入口：10022
• 两个端口由两个独立 service 维护

一、架构

已登记且获安全组授权的来源客户端        ecs-99 (47.110.67.161)                     72602-minipc (192.168.0.25)
ssh -p 10021 aaron@47.110.67.161  ->   0.0.0.0:10021 (sshd) --SSH reverse-->      localhost:22
ssh -p 10022 aaron@47.110.67.161  ->   0.0.0.0:10022 (sshd) --SSH reverse-->      localhost:22
ECS HAProxy :25/:465/:587/:993 -> 127.0.0.1:10225/:10465/:10587/:10993 (sshd)
                                      --SSH reverse--> minipc hostPort :25/:465/:587/:993

说明：反向隧道必须由 72602-minipc 主动发起。0.0.0.0 是 ECS 上 sshd 的 reverse-bind，并不等于对任意公网来源开放；安全组来源限制、SSH key 认证、已建立的 SSH child/session、banner 和监控恢复都必须分别验证。

二、上线前检查

2.1 在 72602-minipc 检查基础条件

# 1) 本机 SSH 服务
sudo systemctl is-active ssh

# 2) autossh 是否安装
autossh -V

# 3) 本机到 ECS 网络与认证
ssh -o ConnectTimeout=5 root@47.110.67.161 hostname
# 期望输出: ecs-99

2.2 在 ECS 检查前置配置

/etc/ssh/sshd_config 至少包含：

GatewayPorts clientspecified

重载：

sudo systemctl reload sshd

安全组仅应按当前 72602 出口地址来源受限地放行 10021/tcp、10022/tcp。 不要为 ZJLAB 的 10023/tcp、10024/tcp 添加公网规则；那两条 listener 属于 ECS loopback-only 的 ProxyJump 路径。

同时确认 ECS 本机防火墙（UFW）放行这两个端口：

sudo ufw status numbered
# 应包含 10021/tcp 和 10022/tcp 的 ALLOW 规则；公网来源边界由云安全组控制

三、创建双 service（72602-minipc 上执行）

下面步骤全部在 72602-minipc 上执行。

3.1 统一 SSH 客户端配置（可选但推荐）

编辑 ~/.ssh/config：

Host ecs-99
    HostName 47.110.67.161
    User root
    ServerAliveInterval 60
    ServerAliveCountMax 3
    ExitOnForwardFailure yes
    TCPKeepAlive yes
    ConnectTimeout 10

3.2 创建 systemd 用户服务目录

mkdir -p ~/.config/systemd/user

3.3 新建 service（10021 主）

文件：~/.config/systemd/user/reverse-tunnel-ecs-10021.service

[Unit]
Description=Reverse SSH tunnel to ecs-99 (port 10021 -> local SSH)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment="AUTOSSH_GATETIME=0"
Environment="AUTOSSH_POLL=60"
Environment="AUTOSSH_FIRST_POLL=30"
ExecStart=/usr/bin/autossh -M 0 -N -R 0.0.0.0:10021:localhost:22 ecs-99
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=default.target

3.4 10022 备入口（含 Mailu）

文件：~/.config/systemd/user/reverse-tunnel-ecs-10022.service

[Unit]
Description=Reverse SSH tunnel to ecs-99 (port 10022 -> local SSH)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment="AUTOSSH_GATETIME=0"
Environment="AUTOSSH_POLL=60"
Environment="AUTOSSH_FIRST_POLL=30"
ExecStart=/usr/bin/autossh -M 0 -N \
   -R 0.0.0.0:10022:localhost:22 \
  -R 127.0.0.1:10225:127.0.0.1:25 \
  -R 127.0.0.1:10465:127.0.0.1:465 \
  -R 127.0.0.1:10587:127.0.0.1:587 \
  -R 127.0.0.1:10993:127.0.0.1:993 \
  ecs-99
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=default.target

说明：10022 只承载备用 SSH 和 Mailu 四个 hostPort。Web 80/443 已迁移 到 HAProxy + WireGuard，不得重新加入该 service，除非按 WireGuard 回滚步骤 临时恢复旧路径。ECS 上的四个 Mailu入口必须是 loopback-only；公网绑定由 HAProxy 完成。

3.5 启用并启动

export XDG_RUNTIME_DIR=/run/user/$(id -u)

systemctl --user daemon-reload
systemctl --user enable --now reverse-tunnel-ecs-10021.service
systemctl --user enable --now reverse-tunnel-ecs-10022.service

systemctl --user status reverse-tunnel-ecs-10021.service --no-pager
systemctl --user status reverse-tunnel-ecs-10022.service --no-pager

3.6 防登出失效（强烈建议）

sudo loginctl enable-linger aaron
loginctl show-user aaron | grep Linger
# 期望: Linger=yes

四、连通性验证

4.1 在 ECS 上看监听

ssh root@47.110.67.161 "ss -tlnp | grep -E '10021|10022'"

期望看到（监听归属示例；地址、来源限制和健康状态需按 2026-08-13 之后的 live 状态复核）：

• 0.0.0.0:10021
• 0.0.0.0:10022
• 127.0.0.1:10225, 127.0.0.1:10465, 127.0.0.1:10587, 127.0.0.1:10993 (sshd)
• 0.0.0.0:80, 0.0.0.0:443 (HAProxy)
• 0.0.0.0:51820/udp (WireGuard)

4.2 在外网验证

nc -zv 47.110.67.161 10021
nc -zv 47.110.67.161 10022

ssh -p 10021 aaron@47.110.67.161
ssh -p 10022 aaron@47.110.67.161

若 10021/10022 外网超时，但 ECS 上已监听，优先检查：

• 云安全组来源网段是否覆盖当前出口 IP
• ECS UFW 是否放行对应端口

4.3 Mailu 入口验证

在 ECS 上确认公网 Web/Mail 端口归 HAProxy；10021/10022 SSH listener 和 10225/10465/10587/10993 Mailu loopback backend 归 sshd：

sudo systemctl is-active haproxy
haproxy -c -f /etc/haproxy/haproxy.cfg
sudo ss -ltnp

从不发送邮件数据的测试客户端验证 banner、TLS 和 STARTTLS。relay 检查最多 发送 EHLO、MAIL FROM、RCPT TO、QUIT，不要发送 DATA，不要认证。 同时检查 ArgoCD 与 front rollout：

argocd app get ops-docs --refresh --insecure --grpc-web
argocd app get mailu --refresh --insecure --grpc-web
kubectl -n mailu rollout status deployment/mailu-front
kubectl -n mailu get endpoints mailu-front mailu-front-ext

若 Dovecot 报 Client not trusted，先检查 Mailu realIpFrom 与 hostPort/CNI 实际传输源是否一致。修正必须提交到 Git 并等待 ArgoCD 自动同步，不能手动 patch、apply、sync 或重启 Mailu。

五、故障恢复（按顺序）

场景 A：ECS 本机 ssh localhost -p 10022 失败

这说明 ECS 上没有监听 10022，问题几乎总在源机器（72602-minipc）侧。

在 72602-minipc 执行：

export XDG_RUNTIME_DIR=/run/user/$(id -u)

# 1) 看 service
systemctl --user status reverse-tunnel-ecs-10021.service --no-pager
systemctl --user status reverse-tunnel-ecs-10022.service --no-pager

# 2) 一次只处理一条；先处理 backup，验证 listener/banner/监控恢复后再决定是否处理 primary
TUNNEL_SERVICE=reverse-tunnel-ecs-10022.service
systemctl --user restart "$TUNNEL_SERVICE"

# 3) 看日志
journalctl --user -u "$TUNNEL_SERVICE" --since "10 min ago" --no-pager

# 4) 验证到 ECS 的基础连通
ssh -o ConnectTimeout=5 root@47.110.67.161 echo ok

场景 B：外网超时但 ECS 本机可通

问题在安全组或 ECS 防火墙，不在隧道本身。

ssh root@47.110.67.161 "ss -tlnp | grep -E '10021|10022'"
ssh root@47.110.67.161 "iptables -L INPUT -n | grep -E '10021|10022' || true"

场景 C：重启后隧道没起来

export XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user is-enabled reverse-tunnel-ecs-10021.service
systemctl --user is-enabled reverse-tunnel-ecs-10022.service
loginctl show-user aaron | grep Linger

六、常用运维命令

export XDG_RUNTIME_DIR=/run/user/$(id -u)

# 重载；重启时一次只选择一个入口，验证 listener/banner/监控后再处理另一个
systemctl --user daemon-reload
TUNNEL_SERVICE=reverse-tunnel-ecs-10021.service
systemctl --user restart "$TUNNEL_SERVICE"

# 停止（仅在明确授权的维护窗口内；一次只停止一个入口）
TUNNEL_SERVICE=reverse-tunnel-ecs-10021.service
systemctl --user stop "$TUNNEL_SERVICE"

# 日志实时跟踪
journalctl --user -u "$TUNNEL_SERVICE" -f

七、和现网监控的关系（72602；2026-08-13 审计快照，动态状态需 live verify）

ECS 上原有巡检继续只监控 72602-minipc 的两个公开入口；其端口列表、脚本、 unit、timer 和状态文件保持不变。

当 72602-minipc 双入口上线后，只需确认：

• 安全组已放行 10021/10022
• /etc/tunnel-healthcheck-ports.conf 包含 10021、10022
• /etc/tunnel-healthcheck.env 企业应用参数可用（DINGTALK_CLIENT_ID/SECRET/AGENT_ID/USER_IDS）

按该审计快照记录的 72602 线上实现：

• 告警通道为钉钉企业应用 API（非 webhook）
• 告警消息为 Markdown 格式（标题：🚨 ECS Tunnel Alert）
• 连续失败 3 次才发送告警（防抖）

ZJLAB 的两个私有 loopback listener 不加入上述端口列表，而由独立的 check-only 巡检按 primary、backup 标签检查。该巡检要求每个标签恰有一个 loopback-only listener、唯一 owner 为 sshd、两个 owner 相互独立、短超时 SSH banner 正常且 owner 签名稳定；连续失败 3 次才告警。日志和告警不包含 endpoint、端口、账户、内网拓扑或签名值。巡检不会 restart/kill 隧道，也不会 修改 sshd、防火墙、安全组、DNS 或 endpoint。真实配置和恢复步骤只保存在 私有 SOPS inventory 与 root-only host 配置中。

八、Mailu 代理回滚

按以下顺序回滚，不恢复旧 HAProxy 配置，也不卸载 HAProxy：

# ECS
sudo systemctl stop haproxy

# 72602-minipc（使用实际保存的备份路径）
cp /home/aaron/Ops/ops-private/backups/reverse-tunnel-ecs-10022.service.<UTC>.before-haproxy \
  ~/.config/systemd/user/reverse-tunnel-ecs-10022.service
export XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user daemon-reload
systemctl --user restart reverse-tunnel-ecs-10022.service

若还原 ECS 的 GatewayPorts 修正，恢复对应的 /var/backups/sshd_config.<UTC>.before-haproxy，执行 sshd -t 后 reload sshd，再按上面步骤恢复并重启 10022。10021 不在本次回滚范围内。

WireGuard Web Tunnel

WireGuard Web Tunnel

公网 Web 入口固定由 ECS HAProxy 持有，HAProxy 通过 WireGuard 访问 72602-minipc 的 ingress-nginx NodePort。TLS 仍由 ingress-nginx 和 cert-manager 管理；ECS 不复制证书，不启用 PROXY protocol，也不终止 TLS。

Internet TCP 80/443
  -> ECS HAProxy
       -> primary: WireGuard 10.77.0.1 <-> 10.77.0.2 over UDP 51820
                    -> minipc TCP 32080/32443 -> ingress-nginx
       -> backup:  ECS-loopback SSH Web path 127.0.0.1:18080/18443
                    -> minipc TCP 32080/32443 -> ingress-nginx

The WireGuard and SSH Web paths are alternative HAProxy backends, not a serial chain. The SSH Web fallback is an independent ECS-loopback service; its host-local unit and credentials are kept in private host state/ ops-private, not reproduced in this repository.

Reference Configuration (verify live state; audit snapshot 2026-08-13)

Both WireGuard configs and private keys are root-only mode 0600. Never print, copy, commit, or place private keys in a ticket. Public keys are identifiers but do not need to be published in this handbook.

Firewall

• UDP 51820 在公网路径上有两层入口：阿里云安全组的 /32 规则（云端边界） 和 ECS 本机 UFW 的 /32 规则（实例边界）。两者都由 72602-minipc 上同一个 5 分钟 systemd timer （update-sg-ip.timer / update-sg-ip.service）协调：安全组由协调器 通过官方 Aliyun ECS / VPC SDK 写入；ECS UFW 规则（comment wg 72602-minipc）由协调器通过专用受限 SSH key 调用 ECS 上 root-only 的 forced-command 助手 /usr/local/sbin/72602-wireguard-ufw-reconcile 调整。两条规则在协调器「两端都已核实」之前都会被保留。
• 协调器只在 Aliyun 安全组与 ECS UFW 这两个 consumer 都验证生效后才清理旧 的 updater-owned 规则并落盘持久状态；任意一侧失败都会让新规则保留、旧 managed 规则保留到下一次重试，重试本身幂等。协调器在 /home/aaron/.local/state/update-sg-ip/ 写入的最近已知 IP 与上一次 「两端都已核实」时间戳只描述成功的协调结果，不描述 IP 探测成功本身。
• minipc UFW 允许 WireGuard 子网访问 TCP 32080 和 32443。
• Do not expose 32080/32443 through the Aliyun security group.

Health Checks

Run these checks without displaying key material:

# Both hosts
systemctl is-active wg-quick@wg0
systemctl is-enabled wg-quick@wg0
sudo wg show wg0

# ECS listener ownership
sudo ss -ltnup | grep -E ':(80|443|51820) '
sudo haproxy -c -f /etc/haproxy/haproxy.cfg

# ECS -> ingress over WireGuard
curl --resolve port.72602.space:32443:10.77.0.2 \
  https://port.72602.space:32443/

# Public strict TLS
curl -fsS -o /dev/null https://port.72602.space/
curl -fsS -o /dev/null https://ops.docs.72602.space/

Expected listener ownership on ECS:

• 80/443: HAProxy
• 51820/udp: WireGuard kernel interface
• 10021/10022: sshd reverse listeners

The WireGuard handshake alone is not sufficient. A valid handshake with a failed NodePort or HAProxy backend still breaks Web traffic, so monitor both the public HTTPS URL and the direct ECS-to-NodePort path.

The ECS host-local Web/tunnel monitor (private runtime configuration; verify it live before relying on its path or unit name) checks the HAProxy frontend, WireGuard primary, SSH Web backup, 72602 SSH listeners, and Mail public/loopback ports. DingTalk receives a message only when the state changes. The ZJLAB 10023/10024 check-only monitor is independent and is not managed by this Web monitor:

• PRIMARY: WireGuard serves Web and the SSH backup is ready.
• BACKUP: WireGuard failed and HAProxy automatically uses SSH.
• PRIMARY_BACKUP_FAILED: Web remains healthy through WireGuard but redundancy is unavailable.
• DOWN: no usable Web backend remains or the HAProxy frontend failed.
• _SSH_PORT_FAILURE / _MAIL_FAILURE: the corresponding critical listener checks failed.

Notifications include the diagnosed layer, active path, whether automatic service recovery succeeded, timestamp, and host. They use DingTalk plain-text messages with real line breaks, not escaped \n text.

Runtime Recovery

ECS uses wg-quick@wg0 and /etc/wireguard/wg0.conf as the sole owner of the WireGuard interface address and peer configuration. Do not add a competing /etc/systemd/network/10-wg0.network file.

The ECS runtime repair timer 72602-wireguard-runtime-repair.timer runs every 30 seconds after boot. Its root-owned check restarts wg-quick@wg0 only when the service is inactive, wg0 lacks 10.77.0.1/30, or the route to 10.77.0.2 does not use wg0. A healthy interface is not restarted. Verify it with:

systemctl list-timers 72602-wireguard-runtime-repair.timer --all
sudo journalctl -t 72602-wireguard-repair --since "-15 min"

systemd-networkd also retries failures with a five-second delay and a 60-second, 12-start limit. The previous outage was not automatically repaired because networkd’s watchdog restart loop hit its default five-start limit after repeated 203/EXEC failures, while wg-quick@wg0 is a successful Type=oneshot unit with RemainAfterExit=yes and no restart policy. That unit therefore remained active (exited) after its address was lost and had no runtime address/route check.

Failure Handling

If public Web fails, inspect in this order:

1. HAProxy owns 80/443 and its configuration validates.
2. Both wg-quick@wg0 services are active and have a recent handshake.
3. ECS can reach 10.77.0.2:32080 and 10.77.0.2:32443.
4. ingress-nginx Service, Pod, Endpoint, Host routing, and certificate are healthy.
5. The Aliyun security group and ECS UFW still allow UDP 51820 from the current home public IP.

Do not restart reverse-tunnel-ecs-10021.service during Web troubleshooting; it is the independent 72602 SSH access/rescue path. Keep the 72602 primary and backup SSH services operationally separate, and do not touch the ZJLAB system-level 10023/10024 path from this runbook. Do not enable PROXY protocol on HAProxy unless ingress-nginx is changed in the same reviewed operation.

Automatic Failover

HAProxy marks WireGuard as the primary backend and the independent SSH loopback path as backup. Checks run every two seconds with fall 2 and rise 2. The dated controlled test record measured automatic failover in approximately 8-9 seconds and automatic return to WireGuard after recovery. Treat that as a test observation, not an SLO; existing connections may fail and must reconnect, while new connections use the healthy path.

The approved SSH Web backup service is referenced as reverse-tunnel-ecs-web-backup.service in private host state. It must remain independent from 10021 (72602 SSH) and 10022 (SSH + Mailu). Never bind its 18080/18443 listeners publicly; they are ECS loopback-only.

Emergency Web Rollback (historical pre-migration path)

Use this only when WireGuard cannot be restored promptly and the independent 10021 72602 SSH access path has been authenticated first. This is not the ZJLAB 10023/10024 ProxyJump path.

1. Keep 10021 authenticated and do not stop Mail HAProxy frontends.
2. Restore the saved HAProxy configuration or the independent SSH Web backup service from ops-private; validate with haproxy -c before reload.
3. If HAProxy itself cannot be restored, only then remove the Web frontends and restore the pre-migration 10022 unit containing -R 80 and -R 443.
4. Confirm listener ownership, both 72602 SSH entries, Mail loopbacks, and strict public TLS.

Do not delete WireGuard keys, uninstall packages, change DNS, or alter ingress/cert-manager during an emergency Web rollback. The host-local migration backup is root/private state and must not be copied into Git.