72602

Scope

This section is the single source of truth for 72602 cluster operations.

Topology

  • Public ECS: 47.110.67.161 (2C4G, Aliyun Hongkong)
  • Active ingress domain: 72602.space; legacy .72602.online routes are retired
  • ArgoCD host: argocd.72602.space
  • k3s node: 72602-minipc (192.168.0.25, MiniPC N100 28G+1TB NVMe)
  • SSH reverse tunnel: :10021 (main), :10022 (backup, also carries 80/443)
  • Ingress NodePort: 32080 (HTTP), 32443 (HTTPS)
  • Ingress class: nginx
  • Ingress namespace: basic-components
  • cert-manager issuer: lets-encrypt
  • Storage class: local-path (default, RWO)
  • OS: Ubuntu 26.04 LTS (minipc)
  • k3s version: v1.34.6+k3s1 (installed via install.sh)

Traffic Path

Internet -> ECS(sshd reverse tunnel) -> minipc:k3s ingress-nginx

ECS Port Forwarding

All traffic reaches ECS via SSH reverse tunnel established from minipc. ECS side ports:

  • 10021 -> minipc:22 (72602 main SSH)
  • 10022 -> minipc:22 (72602 backup SSH, also carries 80→32080, 443→32443)
  • 80 -> minipc:32080 (HTTP via SSH reverse tunnel on 10022)
  • 443 -> minipc:32443 (HTTPS via SSH reverse tunnel on 10022)

DNS Setup

Active service records use 72602.space and point to 47.110.67.161:

HostTypeValueService
argocd.72602.spaceA47.110.67.161ArgoCD UI
ops.docs.72602.spaceA47.110.67.161Hugo Docs
sub2api.72602.spaceA47.110.67.161AI API proxy
port.72602.spaceA47.110.67.161Homepage dashboard
n8n.72602.spaceA47.110.67.161N8N workflow
webhook.n8n.72602.spaceA47.110.67.161N8N webhook receiver
ops.agent.72602.spaceA47.110.67.161OpenCode operations agent
uptime.72602.spaceA47.110.67.161Uptime Kuma
clash.72602.spaceA47.110.67.161Clash/mihomo panel
api.minio.72602.spaceA47.110.67.161MinIO S3 API
console.minio.72602.spaceA47.110.67.161MinIO Console

txt2img.agent.72602.online is retired. Its DNS record, certificate, TLS Secret, and unreferenced ai data claims have been removed.

DNS is managed via Cloudflare / Aliyun DNS (add A record → ECS IP).

Deployed ArgoCD Apps

AppNamespaceTypeSourceIngress
argocdargocdHelm (arco-cd)argo-cd 9.5.4argocd.72602.space
cert-managerbasic-componentsHelm (Jetstack)cert-manager 1.20.2internal
ingress-nginxbasic-componentsHelmingress-nginx 4.15.1shared ingress controller
ops-docsapplicationmanifests (Git)docs.git/mainops.docs.72602.space
homepagemonitormanifests (Git)docs.git/mainport.72602.space
uptime-kumamonitormanifests (Git)docs.git/mainuptime.72602.space
sub2apiapplicationHelm (ghcr)sub2api 0.1.1sub2api.72602.space
postgresqldatabaseHelm (Bitnami)postgresql 18.1.8internal
redis-sharedstorageHelm (Bitnami)redis 18.16.0internal
miniostorageHelmminio 16.0.10console.minio.72602.space, api.minio.72602.space
n8nn8nHelm (community)n8n 1.16.36n8n.72602.space, webhook.n8n.72602.space

Non-ArgoCD (手动部署)

DeploymentNamespaceImageIngress
ops-agentapplicationay-dev/ops-agent:0.2.2ops.agent.72602.space

Network Proxy

Egress Proxy Architecture

k8s Pod (10.42.x.x) --HTTP_PROXY--> 192.168.0.25:17890 (socat) --forward--> 127.0.0.1:7890 (mihomo/clash) --tunnel--> upstream proxies
  • mihomo (clash): listens on 127.0.0.1:7890 (HTTP), 127.0.0.1:7891 (SOCKS5)
    • Config: /home/aaron/clashctl/resources/runtime.yaml
    • Key setting: allow-lan: false (只监听 localhost)
  • socat bridge: 0.0.0.0:17890 → 127.0.0.1:7890 (桥接使 k8s Pod 可达)
    • 进程: socat -d -d TCP-LISTEN:17890,fork,reuseaddr,bind=0.0.0.0 TCP:127.0.0.1:7890
  • k8s Service: argocd-egress-proxy.argocd.svc.cluster.local (ClusterIP: 10.43.42.223:17890) → Host 192.168.0.25:17890
  • App proxy env: 应统一使用 http://192.168.0.25:17890（不是 192.168.0.25:7890，因为 mihomo 仅绑定 127.0.0.1）

关键约束

  • mihomo allow-lan: false 意味着 不能 直接用 192.168.0.25:7890 作为代理地址
  • 必须通过 socat 桥接 (192.168.0.25:17890) 或 argocd-egress-proxy Service 访问
  • GitHub acceleration: ghfast.top URL rewrite + NO_PROXY bypass
  • Image mirror: m.daocloud.io/docker.io, m.daocloud.io/ghcr.io

Known Incident Pattern

  • Symptom: HTTPS handshake fails for argocd.72602.online (tls alert internal error).

  • Root cause: ECS Docker/derper occupies public 443, traffic never reaches k3s ingress.

  • Fix baseline: derper must expose 8443:443, keep public 443 for ingress NodePort 32443.

  • Symptom: n8n 所有 workflow 报 connect ECONNREFUSED 192.168.0.25:7890。

  • Root cause: HTTP_PROXY 指向 192.168.0.25:7890，但 mihomo 只监听 127.0.0.1:7890（allow-lan: false）。Pod 无法直连 mihomo 的 LAN IP。

  • Fix baseline: HTTP_PROXY/HTTPS_PROXY 必须使用 socat 桥接端口 192.168.0.25:17890（或 k8s Service 10.43.19.4:17890），该端口由 socat 转发至 127.0.0.1:7890。

Host-Level Services

ServicePortBindDescription
mihomo (clash) HTTP proxy7890127.0.0.1Egress proxy, allow-lan: false
mihomo (clash) SOCKS57891127.0.0.1SOCKS5 proxy
mihomo external controller90900.0.0.0Clash API/UI, exposed via clash.72602.space
socat bridge178900.0.0.0Forwards to 127.0.0.1:7890, k8s pod accessible
autossh tunnel (main)10021→ECS-Reverse tunnel to ECS
autossh tunnel (backup)10022→ECS-Reverse tunnel + HTTP/HTTPS forwarding
k3s ingress HTTP320800.0.0.0NodePort for ingress HTTP
k3s ingress HTTPS324430.0.0.0NodePort for ingress HTTPS

Notes

  • Keep derper away from public 443 (use 8443).
  • Keep app ingress aligned with ArgoCD ingress pattern:
    • ingressClassName: nginx
    • cert-manager.io/cluster-issuer: lets-encrypt
    • TLS secret per host.
  • argocd-egress-proxy 由 ops-docs ArgoCD Application 管理，并为 repo-server 提供 Git/Helm 出站代理。
  • mihomo allow-lan: false 意味着 Pod 代理地址必须是 socat 桥接端口 17890，不能用 7890。
  • SSH 隧道依赖 loginctl enable-linger 保持用户级 systemd 服务运行。

Recent Operations

2026-07-16: reset csst and update N8N webhook host

  • Deleted and recreated the csst namespace. Only the namespace default ServiceAccount and kube-root-ca.crt ConfigMap remain.
  • Changed N8N WEBHOOK_URL, webhook worker URL, Ingress rule, and TLS DNS name from webhook.72602.online to webhook.n8n.72602.online.
  • Synced ArgoCD application argocd/n8n; main, webhook, MCP webhook, and worker rollouts completed.
  • cert-manager completed HTTP-01 validation and issued the updated certificate.

2026-07-16: migrate OpenCode web to k3s

  • Replaced the host systemd process and static EndpointSlice with the application/ops-agent workload.
  • The Pod mounts /home/aaron/Ops/docs at /workspace, loads the project .opencode/opencode.json, and persists sessions in the opencode-data PVC.
  • Image ay-dev/ops-agent:0.2.2 uses glibc and contains OpenCode 1.18.2, kubectl 1.34.6, Argo CD CLI 3.3.8, VibeGuard, DCP, and Goal Mode.
  • OpenCode native Basic Auth protects both Ingress and cluster-internal access. Anonymous HTTPS returns 401; authenticated HTTPS returns 200.
  • An Nginx sidecar publishes the Ops Agent browser title and proxies Terminal WebSocket and event streams.

2026-07-16: remove Langfuse and refresh Homepage

  • Permanently removed the seven unused Langfuse PVCs (56 GiB) and six residual Secrets from monitor.
  • Removed Langfuse and pgAdmin from Homepage and added the OpenCode operations agent.
  • Restored the ArgoCD Homepage widget by binding the readonly API account to role:readonly.

2026-07-17: align Ops resource names

  • Renamed the Hugo workload and its Service, ConfigMap, build Job, and Ingress resources to ops-docs; retained hugo-docs-pvc to preserve generated content.
  • Renamed the OpenCode-based workload, Service, Ingress, proxy ConfigMap, manifest directory, and Dockerfile to ops-agent; retained existing opencode-* PVC and Secrets to preserve sessions and credentials.
  • Replaced the manual local-proxy-bridge with the GitOps-managed argocd-egress-proxy; ArgoCD repo-server uses its cluster Service while applications can continue using host port 17890.

Default Verification Commands

# 公网入口
curl -vI http://argocd.72602.space
curl -vkI https://argocd.72602.space

# k8s 资源
kubectl get ingress -A -o wide
kubectl get svc -A -o wide
kubectl get pods -A -o wide
kubectl get certificate,certificaterequest,order,challenge -A

# 主机端口
sudo ss -lntp | grep -E ':80|:443|:8443|:32080|:32443|:7890|:17890|:9090'

# ECS 转发
sudo iptables -t nat -L PREROUTING -n -v --line-numbers
sudo iptables -t nat -L DOCKER -n -v --line-numbers

# Egress proxy 完整性
curl -s --connect-timeout 3 -x http://127.0.0.1:17890 http://httpbin.org/ip
kubectl exec -n n8n deploy/n8n -- env | grep -E 'HTTP_PROXY|HTTPS_PROXY'

# SSH 隧道
journalctl --user -u reverse-tunnel-ecs-10021.service --since "1 hour ago" --no-pager
journalctl --user -u reverse-tunnel-ecs-10022.service --since "1 hour ago" --no-pager