72602

Scope

This section is the single source of truth for 72602 cluster operations.

Topology

• Public ECS: 47.110.67.161 (2C4G, cn-hangzhou, zone cn-hangzhou-i)
• Active ingress domain: 72602.space; legacy .72602.online routes are retired
• ArgoCD host: argocd.72602.space
• k3s node: 72602-minipc (192.168.0.25, MiniPC N100 28G+1TB NVMe)
• SSH reverse tunnel: :10021 (main), :10022 (backup and Mailu loopback)
• Web tunnel: WireGuard UDP 51820 between ECS and minipc
• Ingress NodePort: 32080 (HTTP), 32443 (HTTPS)
• Ingress class: nginx
• Ingress namespace: basic-components
• cert-manager issuer: lets-encrypt
• Storage class: local-path (default, RWO)
• OS: Ubuntu 26.04 LTS (minipc)
• k3s version: v1.34.6+k3s1 (installed via install.sh)

Traffic Path

Web:

Internet -> ECS HAProxy TCP passthrough -> WireGuard -> 72602-minipc ingress-nginx NodePort

Mail:

Internet -> ECS HAProxy TCP passthrough -> ECS loopback sshd forwards -> 72602-minipc Mailu front

ECS Port Forwarding

Web traffic crosses WireGuard; SSH and mail continue to use independent SSH reverse tunnels. ECS side ports:

• 10021 -> minipc:22 (72602 main SSH)
• 10022 -> minipc:22 (72602 backup SSH and Mailu loopback forwards)
• 80 -> 10.77.0.2:32080 (HAProxy TCP passthrough over WireGuard)
• 443 -> 10.77.0.2:32443 (HAProxy TCP passthrough over WireGuard)
• 127.0.0.1:10225 -> minipc:25 (SMTP via SSH reverse tunnel on 10022)
• 127.0.0.1:10465 -> minipc:465 (SMTPS via SSH reverse tunnel on 10022)
• 127.0.0.1:10587 -> minipc:587 (submission via SSH reverse tunnel on 10022)
• 127.0.0.1:10993 -> minipc:993 (IMAPS via SSH reverse tunnel on 10022)

ZJLAB NewAPI public route

The isolated ZJLAB NewAPI-compatible service is published at https://newapi.zjlab.72602.space. Its relay, credentials, tunnel, and rollback procedure are maintained in the private operations repository. The existing MaaS route at https://llm.72602.space remains an independent service.

ECS HAProxy listens on public IPv4 and IPv6 25, 465, 587, and 993, and on public IPv4 80 and 443. Web is plain TCP passthrough without PROXY protocol or TLS termination. Mail passes TCP with PROXY v2 to the four ECS loopback backends. TLS remains terminated by ingress-nginx for Web and Mailu for the implicit-TLS mail protocols.

DNS Setup

Active service records use 72602.space and point to 47.110.67.161:

ACME CAA policy

The AliDNS zone 72602.space is delegated to dns15.hichina.com and dns16.hichina.com. AliDNS reports DNSSEC OFF; the parent delegation has no DS record, and both authoritative servers match the delegated nameservers. The zone has one enabled apex CAA record:

This record was created after confirming that no enabled equivalent or conflicting CAA existed. It is scoped to ACME issuance and did not change any address, alias, delegation, DNSSEC, or unrelated record. Verify the record with the AliDNS API, both authoritative servers, and public resolvers 1.1.1.1 and 8.8.8.8; each must return NOERROR and the exact value. Roll back only this change with the official AliDNS SDK DeleteDomainRecord call for 2084917630338801664, then repeat the same resolver checks for an empty CAA answer. Do not delete other records.

The Clash route is basic-components/clash-ui-ingress using class nginx, ClusterIssuer lets-encrypt, and Secret clash.72602.space-tls. After the CAA addition, cert-manager retained the failed Order and automatically retried at its scheduled backoff time. The new Order became valid and the Certificate became Ready. Keep failed Orders and Challenges until normal cert-manager cleanup; do not repeatedly delete them.

The existing Prometheus receiver also accepts ZJLAB Kubernetes metrics through a dedicated HTTPS write-only Ingress at prometheus-write.72602.space. The Ingress exposes only the exact /api/v1/write path and requires a runtime Basic Auth Secret; it does not expose Prometheus query, status, or admin APIs. The TLS certificate is issued by lets-encrypt. ZJLAB sends metrics with the external label cluster=zjlab, so the existing Grafana Prometheus datasource can query ZJLAB without a second Grafana datasource. Credentials remain in runtime/private secret stores and must not be added to public manifests.

The Git-provisioned Grafana dashboard Kubernetes Resources (UID kubernetes-resources-multicluster) is in the Kubernetes folder and is defined by manifests/grafana-kubernetes-dashboard.yaml. It uses the existing Prometheus datasource UID prometheus; its Cluster selector maps 72602 to the empty-label matcher (^$), ZJLAB to cluster="zjlab", and All to both. Node and namespace selectors are dependent query variables. Node resource panels use the kubernetes-service-endpoints scrape job to avoid counting the ZJLAB node-exporter series twice. Verification returned one, two, and three unique nodes for 72602, ZJLAB, and All respectively; Pod CPU, Deployment, and scrape-target queries were non-empty. The previous runtime dashboard is not part of the operating path.

On 2026-08-14, the user-authorized reset deleted the 30Gi Prometheus TSDB PVCs monitor/prometheus-server in 72602 and monitoring/zjlab-prometheus-server in ZJLAB. ArgoCD recreated both PVCs and the existing scrape and remote-write configuration was retained. No backup was made; all prior metrics history is intentionally unrecoverable. Loki, Tempo, Grafana users, and Grafana datasource configuration were not cleared.

Mail records are managed in the 72602.space zone with TTL 600:

The mail records were verified through AliDNS, both authoritative nameservers, and public resolvers 1.1.1.1 and 8.8.8.8. Mailu is configured with DKIM selector dkim for 72602.space, using /dkim/{domain}.{selector}.key. The current admin /dkim file is 72602.space.dkim.key; the front /dkim directory is not used for this admin-owned key.

Mailu DKIM and PTR readiness

• The domain key was generated idempotently in the admin Pod with the official update import path. The input contained only the domain name and dkim_key: -generate-; -u -q was used and the default replace mode was not used:

  kubectl -n mailu exec deploy/mailu-admin -- sh -c \
    'printf "%s\\n" "domain:" "  - name: 72602.space" \
      "    dkim_key: -generate-" | flask mailu config-import -u -q -'

• The admin image does not contain openssl. A temporary mailu-dkim-openssl Pod using the already deployed Mailu front image mounted only the mailu-storage PVC dkim subPath and ran container openssl to derive RSA public DER base64. It mounted no Secret, was deleted after derivation, and emitted no private-key content. The derived public value is intentionally not stored in this handbook.

• AliDNS was queried first from aaron@72602-minipc with the official SDK v4.6.0 in /home/aaron/.local/venvs/alidns, sourcing only the mode-0600 /home/aaron/.aliyun-keys credential file inside that SDK process. The exact dkim._domainkey / TXT record was absent, so one record was created with TTL 600 and value shape v=DKIM1; k=rsa; p=<derived-public-key>. Its RecordId is 2082130099188750336.

• The authoritative servers dns15.hichina.com and dns16.hichina.com, and public resolvers 1.1.1.1 and 8.8.8.8, all returned the complete matching TXT value with TTL 600 (410 characters). SPF, DMARC, and MX retained their existing values. Mailu admin/front were Ready, all Mailu Pod restart counts were zero, and no Warning events were present after the operation.

• Roll back this newly created DNS record only with the official AliDNS SDK DeleteDomainRecord call for RecordId 2082130099188750336. Do not delete or regenerate the Mailu key during DNS rollback. If a future operation updates an existing record instead of creating one, restore that same RecordId’s pre-change value and TTL rather than deleting it.

• Reverse DNS for 47.110.67.161 currently returns NXDOMAIN. A fresh read-only ECS API check found the address as the primary public IP of running instance i-bp1caavgd1twh7wb3n63 in cn-hangzhou (zone cn-hangzhou-i); its EipAddress.AllocationId and EipAddress.IpAddress are empty. The installed official aliyun-python-sdk-ecs is 4.24.83: DescribeEipAddresses found no EIP allocation for this address in the 32 discovered regions. The installed DescribeNatGateways model returned zero gateways in 31 regions; sa-east-1 returned 503 ServiceUnavailable twice and remains unconfirmed. The SDK contains ModifyEipAddressAttributeRequest, but its request model exposes AllocationId and Bandwidth (plus common owner parameters), not ResourceId, RegionId, or ReverseDnsName; it contains no ModifyReverseDns, ReverseDnsName, DescribeNatGatewayEipAddresses, or DescribeNatGatewayAttribute model. RegionId is supplied to AcsClient, not as a PTR parameter in that request. Do not treat ModifyEipAddressAttribute as a PTR operation or change PTR automatically; identify the owning Alibaba product/resource and use its documented console reverse-DNS action or Alibaba support path to request mail.72602.space if that product permits it. This verification made no PTR or DNS change.

Safe checks:

kubectl -n mailu exec deploy/mailu-admin -- sh -c 'ls -1 /dkim'
kubectl -n mailu exec deploy/mailu-front -- sh -c 'ls -1 /dkim'
dig @dns15.hichina.com dkim._domainkey.72602.space TXT +noall +answer +authority
dig @dns16.hichina.com dkim._domainkey.72602.space TXT +noall +answer +authority
dig -x 47.110.67.161 +noall +answer +authority

The AliDNS credential file must be sourced only inside the official SDK process; never print, log, copy, or commit its values. Never print Mailu Secret data, passwords, or DKIM private-key contents during these checks.

txt2img.agent.72602.online is retired. Its DNS record, certificate, TLS Secret, and unreferenced ai data claims have been removed.

DNS for 72602.space is managed in AliDNS. Active service records should point to the ECS public address only when the corresponding Ingress and certificate are Ready; do not describe this zone as dual-managed by Cloudflare.

Deployed ArgoCD Apps

filing-site is uninstalled. Commit 0c250db869ae45c6c6a5a850876728783f1b08dd removed its manifest from the ops-docs source. Detailed filing-site deployment checks are intentionally omitted from this current-state page; use Git history when an older incident record is required.

argocd/ops-docs reconciles the repository’s manifests path and owns the application workloads and their child Applications, including sub2api and mailu. The same source also defines the observability Applications listed above. Sub2API uses the application namespace, nginx Ingress, a Ready TLS certificate, a 10Gi local-path RWO application PVC, and an 8Gi local-path RWO Redis PVC with AOF enabled.

The alloy Application is also live in monitor (Grafana Alloy chart 1.10.1) and receives OTLP traffic at otlp.72602.space; it forwards traces, metrics, and logs to Tempo, Prometheus, and Loki. Confirm the Application and its endpoints before changing the observability pipeline.

Ops Docs Publishing

argocd/ops-docs compares only the repository’s manifests path. A commit that changes only content can advance .status.sync.revision while remaining Synced; it does not create a sync operation, so the ops-docs-build Sync hook does not run.

Publish a reviewed content commit by setting its full SHA in manifests/configmap.yaml as PUBLISH_REVISION, then commit and push that single manifest change. Automatic sync configures ops-docs-config and runs the hook. The hook fetches and verifies that exact SHA, builds Hugo into hugo-docs-pvc, and writes the SHA to /usr/share/nginx/html/.ops-docs-revision. Its fixed Job name is safe because the delete policy is BeforeHookCreation,HookSucceeded; the successful Job is normally absent after the operation.

Verify the source, operation, build marker, rollout, and public route:

git -C /home/aaron/Ops/docs ls-remote origin refs/heads/main

kubectl -n application exec deployment/ops-agent -c ops-agent -- \
  argocd app get ops-docs --hard-refresh --insecure --grpc-web
kubectl -n application exec deployment/ops-agent -c ops-agent -- \
  argocd app history ops-docs --insecure --grpc-web

kubectl -n application rollout status deployment/ops-docs --timeout=300s
kubectl -n application exec deployment/ops-docs -- sh -c \
  'tr -d "\n" < /usr/share/nginx/html/.ops-docs-revision; printf "\n"'
curl -fsS -o /dev/null -w '%{http_code}\n' https://ops.docs.72602.space/

The Application revision and published content revision can differ by the manifest-only trigger commit; both must match their reviewed Git commits. To roll back the generated site, set PUBLISH_REVISION to the previous reviewed content SHA in Git and push a new trigger commit. Let automatic sync rebuild the PVC. Do not copy HTML directly or patch the Deployment, ConfigMap, or PVC.

Non-ArgoCD (手动部署)

2026-08-14: ops-agent 0.2.3 deployed

• Live checks confirmed hostname=72602-minipc, kubeconfig context default, and node 72602-minipc Ready at 192.168.0.25 on v1.34.6+k3s1.
• The live owner is argocd/ops-agent, sourced from https://github.com/AaronYang0628/docs.git at manifests/ops-agent, target main.
• The local 0.2.3 image built successfully with OpenCode 1.18.16 and image ID 2e8de824615d5cfc8b3cb887bff21bad974985fe88ad24beda1c9082760c0f7c. The OCI manifest digest was sha256:f1e5e79d570cdec0ce375e264f3c83291912180591f43308d7fe2b66dc326bff. The first attempt hit a transient direct dl.k8s.io curl timeout; retrying completed the build.
• The image was pushed to the Aliyun personal registry and imported into local k3s containerd under tag 0.2.3. Registry credentials were used through a temporary authfile and were not written to Git or the host’s persistent container configuration.
• The GitOps manifest was committed and pushed to main; ArgoCD reconciled the updated Deployment. The replacement Pod became Ready with the 0.2.3 image, and the rollout completed successfully.
• Authenticated /global/health returned healthy, DNS resolved ops.agent.72602.space to 47.110.67.161, its nginx Ingress and Ready TLS remained healthy, and the anonymous public health request returned the expected 401.
• Roll back by restoring the prior 0.2.2 source revision and reconciling; do not delete Secrets or PVCs.

Network Proxy

For host-level command loading and the standard read-only preflight, use the shared Clash/Mihomo runbook. In particular, clashctl is a shell function and must be sourced explicitly in non-interactive agent shells; do not probe guessed ports when clashctl status and the runtime configuration provide the answer directly.

Egress Proxy Architecture

k8s Pod (10.42.x.x) --HTTP_PROXY--> 192.168.0.25:17890 (socat) --forward--> 127.0.0.1:7890 (mihomo/clash) --tunnel--> upstream proxies

• mihomo (clash): listens on 127.0.0.1:7890 (HTTP), 127.0.0.1:7891 (SOCKS5)
  • Config: /home/aaron/clashctl/resources/runtime.yaml
  • Key setting: allow-lan: false (只监听 localhost)
• socat bridge: 0.0.0.0:17890 → 127.0.0.1:7890 (桥接使 k8s Pod 可达)
  • 进程: socat -d -d TCP-LISTEN:17890,fork,reuseaddr,bind=0.0.0.0 TCP:127.0.0.1:7890
• k8s Service: argocd-egress-proxy.argocd.svc.cluster.local:17890 → Host 192.168.0.25:17890; use the Service DNS name rather than a mutable ClusterIP.
• App proxy env: 应统一使用 http://192.168.0.25:17890（不是 192.168.0.25:7890，因为 mihomo 仅绑定 127.0.0.1）

关键约束

• mihomo allow-lan: false 意味着 不能 直接用 192.168.0.25:7890 作为代理地址
• 必须通过 socat 桥接 (192.168.0.25:17890) 或 argocd-egress-proxy Service 访问
• GitHub acceleration: ghfast.top URL rewrite + NO_PROXY bypass
• Image mirror: m.daocloud.io/docker.io, m.daocloud.io/ghcr.io

Known Incident Pattern

• Historical pattern (legacy .72602.online hostname): HTTPS handshake fails with tls alert internal error.

• Root cause: ECS Docker/derper occupies public 443, traffic never reaches k3s ingress.

• Fix baseline: derper must expose 8443:443, keep public 443 for ingress NodePort 32443.

• Symptom: n8n 所有 workflow 报 connect ECONNREFUSED 192.168.0.25:7890。

• Root cause: HTTP_PROXY 指向 192.168.0.25:7890，但 mihomo 只监听 127.0.0.1:7890（allow-lan: false）。Pod 无法直连 mihomo 的 LAN IP。

• Fix baseline: HTTP_PROXY/HTTPS_PROXY 必须使用 socat 桥接端口 192.168.0.25:17890（或 Service DNS argocd-egress-proxy.argocd.svc.cluster.local:17890），该端口由 socat 转发至 127.0.0.1:7890。

Host-Level Services

Notes

• Keep derper away from public 443 (use 8443).
• Keep app ingress aligned with ArgoCD ingress pattern:
  • ingressClassName: nginx
  • cert-manager.io/cluster-issuer: lets-encrypt
  • TLS secret per host.
• argocd-egress-proxy 由 ops-docs ArgoCD Application 管理，并为 repo-server 提供 Git/Helm 出站代理。
• mihomo allow-lan: false 意味着 Pod 代理地址必须是 socat 桥接端口 17890，不能用 7890。
• 72602 的 SSH 隧道当前由两个独立的用户级 systemd 服务维护，并依赖 loginctl enable-linger 在登出和重启后继续运行；ZJLAB 的对应 loopback 隧道采用独立的系统级服务，不能把两套服务模型混写。
• ECS must allow inbound TCP 25,465,587,993 from 0.0.0.0/0 and UFW must allow the same ports before testing public mail delivery. Preserve the existing default firewall policies and unrelated rules.

Mailu Public Mail Path

• Inbound mail remains Internet -> ECS 47.110.67.161:25/465/587/993 -> HAProxy TCP passthrough with PROXY v2 -> ECS loopback backends 127.0.0.1:10225/10465/10587/10993 -> independent 10022 reverse tunnel -> 72602-minipc Mailu front hostPort.
• ECS public mail ports are owned by HAProxy; the four loopback backends are owned by sshd. The 10021 SSH service is independent and must not be restarted during mail changes.
• The current Mailu source trusts the k3s CNI gateway 10.42.0.1/32 for PROXY traffic and exposes PROXY ports 993, 25, 465, and 587. Read-only verification on 2026-08-13 confirmed the Mailu workloads were Ready and the 465/993 TLS and 587 STARTTLS handshakes completed. The old 127.0.0.1/32 trust mismatch and EOF observations are historical incident records, not the current baseline.

Mailu outbound delivery and SPF

• Mailu outbound delivery is direct from Postfix through the 72602 home egress. It does not traverse ECS 10022 or ECS HAProxy; those are inbound mail paths. The egress address is dynamic, so update-mailu-spf.timer refreshes the AliDNS SPF record hourly. Query the current record and current egress IP during an incident rather than treating an observed historical IP as a permanent value.
• The 2026-08-13 audit observed home egress and authoritative/public SPF as 36.24.58.213; the single SPF record is maintained by update-mailu-spf.timer. Treat that address, and the 36.24.59.216 and 125.121.102.50 values in the dated 2026-07-31 delivery record, as dated observations rather than permanent allowlist values. A successful SMTP queue response still does not prove final inbox placement or reputation.
• Rspamd logged DKIM_SIGNED for 72602.space with selector dkim. The corresponding DNS record is 2082130099188750336; DMARC is record 2082063800085560320 with p=none. PTR lookups for both the dynamic egress IP and ECS 47.110.67.161 returned NXDOMAIN. A fixed outbound SMTP relay remains the reliable solution; no relay was configured.
• SPF automation still has a one-hour polling interval and cannot guarantee delivery immediately after a home IP change. Preserve the timer and inspect its last run before making any manual DNS change; do not hard-code a transient egress address in this current-state section.

Useful checks:

sudo ss -lntp | grep -E ':(25|465|587|993|10022)$'
kubectl -n mailu get deploy,pod,svc,certificate,order -o wide
kubectl -n mailu get endpoints mailu-front -o wide

Rollback for this mail proxy change is: systemctl stop haproxy; restore the saved reverse-tunnel-ecs-10022.service backup; run systemctl --user daemon-reload; restart only reverse-tunnel-ecs-10022.service. If reverting the required ECS sshd binding change, restore /var/backups/sshd_config.20260728T142924Z.before-haproxy, run sshd -t, reload sshd, then restore/restart the 10022 tunnel as needed. Do not restore an old HAProxy configuration or uninstall the package as part of this rollback, and do not delete Mailu Secrets or PVCs.

Recent Operations

2026-08-02: application namespace prune incident

• Trigger and time: commit 0c250db869ae45c6c6a5a850876728783f1b08dd deleted manifests/filing-site.yaml. The file’s first object was the shared Namespace/application. At 2026-08-02 08:25:53 +08, the ops-docs Application’s automated prune deleted that Namespace along with the intended filing-site resources.
• Root cause: the lifecycle of a shared Namespace was coupled to one removable workload manifest while automated prune was enabled. Once the Namespace disappeared from Git, Argo CD treated it as stale. Kubernetes then cascade-deleted namespaced resources; sync options on an individual PVC or Secret cannot protect it from deletion through its parent Namespace.
• Impact: runtime Secrets and the application local-path PVCs were deleted, so Ops Agent, Sub2API, and its Redis could not start. Argo CD recreated declarative resources, but not runtime Secret values or deleted volume contents. PostgreSQL remained intact because it runs in the separate database namespace.
• Unrecovered data: the pre-incident local-path directories for opencode-data, Sub2API Redis AOF, sub2api-data, and filing-site photos were deleted and had no snapshot. Those contents were not recovered; replacement PVCs do not contain the former data.
• Secret recovery: the 2026-08-02 00:00 +08 etcd snapshot was restored only into an isolated same-version temporary k3s. The approved whitelist was the aliyun-registry Secret, five opencode-* Secrets, and three sub2api-* Secrets. All nine were still absent immediately before create-only restoration. No Secret value was printed, logged, committed, or allowed to overwrite a newer object. The production server never ran cluster-reset; the procedure is documented in Runtime Secret Recovery.
• Result: kubelet recovered the existing post-prune Pods without manual deletion or restart. Ops Agent reached 2/2 Ready; Sub2API and its Redis reached 1/1; application health checks passed; and ops-docs, ops-agent, and sub2api were Synced/Healthy. filing-site-upload-auth, 72602.space-tls, filing-site, and deleted local-path data were not restored.
• Immediate guard: the live Namespace/application was merge-patched only with argocd.argoproj.io/sync-options=Prune=false,Delete=false. This protects an Argo CD-tracked Namespace from prune and Application deletion, but it is not an admission policy and cannot block direct deletion by another actor.
• Durable guard: during the incident response, manifests/application-namespace.yaml was prepared as a dedicated Namespace manifest but was not included in a commit or pushed at that response checkpoint; remote main and ops-docs were still at 0c250db8. This is a historical checkpoint, not a claim about the current repository state. Before relying on the guard, verify that the reviewed file is in remote Git, Argo CD tracks the Namespace with both sync options, no removable workload manifest defines the shared Namespace, and recoverable off-volume backups exist for required local-path data. Do not delete or recreate the Namespace as rollback.

2026-07-30: fix MinIO S3 upload HTTP 413

• Symptom: Sub2API backup uploads to api.minio.72602.space failed with S3 PutObject HTTP 413 for an approximately 2.45 MB request. The default ingress-nginx client_max_body_size 1m rejected the request before MinIO.

• Fix: commit 2338bc6 added this annotation under the MinIO apiIngress.annotations in Git:

  nginx.ingress.kubernetes.io/proxy-body-size: "0"

  This change is scoped only to the MinIO API Ingress; it does not change the MinIO console or the shared/global ingress configuration. ArgoCD and MinIO became Synced/Healthy; generated nginx reported client_max_body_size 0. An authenticated S3 PutObject/Stat/Delete smoke test passed and its temporary object was removed. Sub2API backup upload was then manually confirmed successful.

• Roll back by reverting 2338bc6 in Git and allowing ArgoCD to reconcile. If an emergency live reversal is required first, remove only the annotation from storage/minio-api, then verify ArgoCD convergence. Do not delete MinIO Secrets or PVCs. Do not globally disable request body limits without explicit scope and security review.

2026-07-29: rotate Ops Agent provider credentials

• Confirmed the operation on 72602-minipc with context default; the only node was Ready at 192.168.0.25 on v1.34.6+k3s1. Before the change, application/opencode-model contained only the api-key key and the live Deployment injected only OPENAI_API_KEY. No Secret value was read or printed.
• Commit ddb69f1 on main routes both the OpenAI and Grok providers to the then-current endpoint https://sub2api.72602.space/v1. The current public endpoint is https://token.72602.space/v1; this dated record is retained as history, not as a current endpoint instruction. The Secret was merge-patched through stdin to update api-key and add grok-api-key, preserving its other fields, and manifests/ops-agent/deployment.yaml was applied. No credential was written to Git or a temporary file, and no sub2api or unrelated resource was changed.
• The live Deployment now injects OPENAI_API_KEY from opencode-model/api-key and GROK_API_KEY from opencode-model/grok-api-key. Rollout completed with Deployment generation and observed generation 16; Pod ops-agent-68556dc7f5-6jd84 was 2/2 Ready with zero restarts, and the Service endpoint was 10.42.0.207:8080.
• The authenticated internal /global/health check returned healthy=true. Separate, read-only /v1/models requests from the Pod returned HTTP 200 with the OpenAI credential and HTTP 200 with the Grok credential. Filtered live merged configuration at that time showed both provider base URLs as https://sub2api.72602.space/v1; new-Pod logs contained no Invalid API key message.
• DNS for ops.agent.72602.space resolved to 47.110.67.161; Ingress ops-agent used class nginx and that host, and Certificate ops.agent.72602.space-tls was Ready. The unauthenticated public health URL returned the expected HTTP 401. An Argo CD hard refresh from the Ops Agent Pod with --insecure --grpc-web reported ops-agent Synced to ddb69f1 and Healthy.
• Roll back by restoring the prior approved model credential with the same non-output stdin merge-patch method. Remove grok-api-key only when reverting to the previous single-provider state, and restore/sync source revision a7e434b for the prior Deployment. Do not delete the Secret, expose its values, or modify sub2api during rollback.

2026-07-28: read-only Mailu verification after c5d1e0a

• Checks ran from the Ops Agent Pod (hostname=ops-agent-5d6878f6c-xwdb). kubectl config current-context was unset, but in-cluster credentials reached the only node 72602-minipc, Ready at 192.168.0.25 on v1.34.6+k3s1.
• Automatic sync completed at 2026-07-28T12:50:54Z. Read-only ArgoCD checks reported ops-docs Synced/Healthy at full revision c5d1e0adddaace9ad0ba2d3c57c3ef27eb0e6802 (c5d1e0a, history ID 24), and child Application mailu targeting chart 2.7.3 as Synced/Healthy. No manual apply, sync, delete, rollback, or restart was run.
• Read-only observations from 12:53:05Z through approximately 13:00Z kept all eight Mailu Deployments and both StatefulSets at 1/1 Ready. The mailu-front rollout status completed. mailu-front (10.43.76.102) and mailu-front-ext (10.43.108.81, ClusterIP) both had endpoint 10.42.0.198. The mailu-envvars ConfigMap reported PORTS=80,443,4190,995,993,25,465,587,4190.
• Certificate mail.72602.space-tls was Ready=True; its Order was valid, with validity 2026-07-28T10:22:57Z through 2026-10-26T10:22:56Z, and no active Challenge. Public DNS returned mail.72602.space A 47.110.67.161, 72602.space MX 10 mail.72602.space., SPF v=spf1 mx -all, and DMARC v=DMARC1; p=none; rua=mailto:admin@72602.space, all with TTL 600. The three PVCs were Bound on local-path with RWO: 2Gi ClamAV, 100Gi Mailu storage, and 8Gi Redis.
• Following the HTTPS redirect, https://mail.72602.space/ returned final HTTP 200 at /sso/login?url=/webmail/?homepage. Port 25 returned 220 mail.72602.space ESMTP ready; ports 465 and 993 completed verified TLS 1.3 handshakes for mail.72602.space.
• Port 587 did not provide STARTTLS: openssl s_client reported no STARTTLS and an unexpected EOF. Connections to port 587 on both front ClusterIP Services and the postfix ClusterIP were refused, and the front container listener list had no 587. The ConfigMap update was not followed by a new front Pod template rollout: the running Pod started at 2026-07-28T12:32:36Z, before the automatic sync, while the Deployment remained 1/1 with generation and observed generation 3. Fix the rollout through Git and automatic ArgoCD reconciliation before treating submission as ready.
• A single non-authenticated relay probe sent only EHLO, MAIL FROM, RCPT TO:<nobody@example.net>, and QUIT; it sent no DATA, credentials, or message. The external RCPT received 250 2.1.5 Ok instead of a rejection. This does not prove actual delivery or an open relay, but it fails the required black-box rejection check and requires immediate relay-policy review.
• No Secret data, password, or DKIM private key was read. Remaining manual actions are to fix and reverify the front submission rollout, review relay policy, obtain the initial admin password through an approved secure process, publish the generated DKIM public record, and confirm ECS/UFW/cloud policy for outbound TCP 25 before any real delivery test. Do not delete Mailu Secrets or PVCs during correction or rollback.

2026-07-28: read-only Mailu deployment verification

• Read-only checks ran from the Ops Agent Pod (hostname=ops-agent-5d6878f6c-xwdb). kubectl config current-context was unset, but the in-cluster credentials reached the only node 72602-minipc, which was Ready at 192.168.0.25 on v1.34.6+k3s1.
• Ran argocd app get ops-docs --hard-refresh --insecure --grpc-web: revision 2773ef5, Synced, Healthy. Ran argocd app get mailu --refresh --insecure --grpc-web: target 2.7.3, Synced, Degraded; only the mailu-front Deployment was Degraded.
• Over approximately eight minutes, 17 read-only observations at 30-second intervals showed mailu-admin, mailu-dovecot, mailu-oletools, mailu-postfix, mailu-rspamd, mailu-tika, and mailu-webmail at 1/1 Ready, with mailu-clamav and mailu-redis-master at 1/1; all three PVCs were Bound on local-path (2Gi, 100Gi, and 8Gi). mailu-front remained 0/1 Ready with ProgressDeadlineExceeded, and mailu-front had no Endpoints. The old Pod mailu-front-85d9b6d7d4-6bd9s used m.daocloud.io/ghcr.io/mailu/nginx:2024.06.57 and remained in ImagePullBackOff after the mirror returned 403 Forbidden. The replacement Pod mailu-front-5cbbf9bc99-2qc7v used the desired ghcr.nju.edu.cn/mailu/nginx:2024.06.57 but remained Pending because the single node had no free requested host ports. Both ReplicaSets requested host ports 110,995,143,993,25,465,587.
• Ingress mailu uses class nginx for mail.72602.space. Certificate mail.72602.space-tls is Ready=True, its Order is valid, and the certificate is valid from 2026-07-28T10:22:57Z through 2026-10-26T10:22:56Z; there is no active Challenge.
• getent ahostsv4 mail.72602.space resolved the host to 47.110.67.161. Direct https://mail.72602.space/ returned 503. ECS TCP ports 25, 465, 587, and 993 accepted connections, but SMTP ports closed before returning a banner and SMTPS/IMAPS TLS handshakes were reset. No authentication or real mail delivery was attempted.
• The exact blocker is the mailu-front image-pull 403 Forbidden in the old ReplicaSet combined with host-port contention during the single-node rolling update. DNS, the issued certificate, and public ECS port reachability are not the blocker. Webmail/Admin and the mail protocols are therefore not usable yet. No delete, rollback, manual apply, or ArgoCD sync was run, and no Secret values were read. A future fix must correct the image/rollout through the Git source, then verify the front Endpoints and protocol handshakes; preserve all Mailu Secrets and PVCs. Any rollback should restore the reviewed source revision through ArgoCD and must not delete Mailu Secrets or PVCs.

2026-07-28: diagnose MinIO Console slowness and port TLS fix

• Read-only identity checks from the Ops Agent Pod reported hostname=ops-agent-5d6878f6c-xwdb; the kubeconfig context was unset, but in-cluster credentials reached the only node 72602-minipc, Ready at 192.168.0.25 on v1.34.6+k3s1. No Secret values, credentials, object names, or object data were read.
• The pending manifests/ingress-port.yaml change has ClusterIssuer lets-encrypt, nginx.ingress.kubernetes.io/ssl-redirect: "true", and TLS host port.72602.space, but no matching port.72602.space rule. It belongs to the application/ops-docs source, while the live owner is monitor/homepage from manifests/homepage/ingress.yaml. Therefore it was invalid for the requested route and was not committed, pushed, applied, synced, or rolled back. Before correction, http://port.72602.space/ returned 200 without redirect; HTTPS returned 200 only with the Kubernetes Ingress Controller fake certificate. The correct future fix must update the owning Homepage ingress source, then sync ops-docs, wait for Certificate Ready, and verify strict HTTPS.
• storage/minio was 1/1 Ready (minio-7f776484df-sslgs), with four restarts 58 days ago, 2m CPU and 662Mi memory observed, requests 250m/512Mi, limits 512m/1Gi, Service endpoints 10.42.0.21:9000,9001, and an 8Gi Bound local-path PVC. The mounted filesystem reported 937G total, 269G used, and 621G free. Node MemoryPressure, DiskPressure, and PIDPressure were False; node usage was 389m CPU (2%) and 11737Mi memory (40%).
• From 72602, public Console root/API timings were 0.125s/0.241s; from ecs-99, 0.772s/0.783s. Direct Service/Pod representative requests were approximately 0.0003–0.0006s; direct ingress was approximately 0.107–0.179s. The 3,259,800-byte main JavaScript asset took 16.802s locally and 13.893s from ECS; the 3,835,591-byte login video took 21.337s and 17.413s. The 663,820-byte background SVG took 2.144s and 5.437s. API responses remained fast.
• Ingress-nginx was at 3m CPU and 253Mi memory with five restarts 58 days ago, 16 workers, and 16384 worker connections. In the last 24 hours, 165 Console access requests had no observed 499/502/504 or upstream errors; MinIO logs had no warning/error lines. /ws/objectManager returned HTTP 101 and several websocket connections ended at 60.000–60.002s, matching the generated proxy-read-timeout 60s; this is a long-lived websocket timeout/reconnect concern, not a slow initial API response.
• Root cause assessment: initial Console slowness is dominated by large static/media transfers through the ECS reverse tunnel and its network variability; MinIO, ingress CPU/concurrency, storage capacity, readiness, and API/backend response time were not limiting. No performance mutation was justified, so no resource, timeout, keepalive, scaling, DNS, or broad configuration change was made. No rollback is needed. A future websocket timeout change must be committed through manifests/minio-argocd.yaml and synced through ArgoCD; any port TLS correction must first update the owning Homepage ingress source.

2026-07-28: observe automatic sync for d639a46

• Ran the read-only checks from the Ops Agent Pod (hostname=ops-agent-5d6878f6c-xwdb). kubectl config current-context was unset, but in-cluster credentials reached the only node 72602-minipc, which was Ready at 192.168.0.25 on v1.34.6+k3s1. The expected Git manifest route is mail.72602.space, with Ingress class nginx, issuer lets-encrypt, local-path storage, Mailu chart 2.7.3, and global.security.allowInsecureImages: true.
• Ran argocd app get ops-docs --hard-refresh --insecure --grpc-web. The first observation was OutOfSync from main (d639a46) with Healthy; automatic sync then completed without a manual sync, and the final state was Synced to main (d639a46) and Healthy. The output showed child argocd/mailu configured. No argocd app sync, permission bypass, Kubernetes apply, create, patch, or delete was run.
• Ran argocd app get mailu --refresh --insecure --grpc-web. Automatic sync removed the previous ComparisonError; Mailu was Synced to 2.7.3 but Progressing. Twenty read-only observations over approximately five minutes (10:05:17–10:10:07 UTC) remained Synced / Progressing with no ComparisonError condition.
• The required kubectl -n mailu get deploy,statefulset,pod,pvc,svc,ingress,certificate,order,challenge -o wide check found all eight Mailu Deployments at 0/1; mailu-clamav StatefulSet at 0/1 and mailu-redis-master at 1/1. The Redis Pod and ACME solver Pod were Running and ready; admin, dovecot, and postfix were ImagePullBackOff; oletools, rspamd, and webmail were ErrImagePull; front, tika, and clamav were ContainerCreating. The three PVCs were Bound (2Gi, 100Gi, and 8Gi, all local-path/RWO). Ingress mailu used nginx for mail.72602.space at 10.43.13.156 on ports 80,443. Certificate mail.72602.space-tls was not Ready, its Order was pending, and its Challenge was pending.
• The redacted ACME error was: Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://mail.72602.space/.well-known/acme-challenge/...': Get "http://mail.72602.space/.well-known/acme-challenge/...": dial tcp: lookup mail.72602.space on 10.43.0.10:53: no such host. Kubelet also reported, for Mailu components including admin, webmail, oletools, rspamd, postfix, and dovecot: failed to pull and unpack image "m.daocloud.io/ghcr.io/mailu/<component>:2024.06.57": failed to resolve reference ...: unexpected status from HEAD request to https://m.daocloud.io/v2/ghcr.io/mailu/<component>/manifests/2024.06.57: 403 Forbidden. The front Pod additionally reported MountVolume.SetUp failed for volume "certs" : secret "mail.72602.space-tls" not found. No Secret data was read.
• The Helm comparison blocker was cleared by the committed configuration, and ArgoCD automatically created the Mailu resources. The rollout remained blocked by the image-mirror 403 Forbidden responses and missing DNS record, which prevented cert-manager HTTP-01 validation and TLS Secret issuance. No resource deletion or rollback was performed. Do not delete the Mailu Secret or PVCs; any Git revert, ArgoCD sync, DNS change, or resource mutation requires explicit authorization and should first correct the image source and DNS.

2026-07-28: Mailu deployment blocked before resource sync

• Ran the checks from the Ops Agent Pod (hostname=ops-agent-5d6878f6c-xwdb). kubectl config current-context was unset, but in-cluster credentials reached 72602-minipc, which is Ready at 192.168.0.25 on v1.34.6+k3s1. The existing mailu namespace and Secret metadata were left unchanged; Secret values were not read.
• GitHub main contains 91986de with manifests/mailu-argocd.yaml. The manifest declares the Mailu route mail.72602.space, using the active 72602.space domain, nginx, lets-encrypt, and local-path.
• Ran argocd app get ops-docs --hard-refresh --insecure --grpc-web; argocd/ops-docs reported Synced, Healthy, and revision 91986de, and its output confirmed that argocd/mailu was created. The required argocd app sync ops-docs --revision main --assumeYes --insecure --grpc-web then failed before applying changes with PermissionDenied: applications, sync, default/ops-docs, sub: readonly.
• No mailu sync was attempted, and no Helm install, deletion, or automatic rollback was performed. argocd app get mailu --insecure --grpc-web reported Sync Unknown, Health Healthy, and a ComparisonError: Helm rejected the substituted Daocloud Bitnami Redis images because global.security.allowInsecureImages=true is not enabled.
• Read-only checks found no Mailu Deployment, Pod, PVC, Service, Ingress, Certificate, CertificateRequest, Order, Challenge, or namespace events. mail.72602.space had no DNS result, and the HTTPS probe timed out because no Mailu endpoint was deployed.
• Rollback boundary: stop further sync; after explicit authorization, restore or revert the Git source to the pre-change known-good revision and sync through ArgoCD. Do not delete the mailu Secret or any PVC, and do not auto-rollback.
• Next actions: obtain an ArgoCD identity permitted to sync ops-docs, review and update the Git manifest with the chart-supported insecure-image setting if the image mirror is retained, then sync ops-docs and mailu through ArgoCD. Add mail.72602.space DNS A record to 47.110.67.161 before validating the public endpoint.

2026-07-28: create Mailu namespace and bootstrap Secret

• Ran the checks from the Ops Agent Pod (hostname=ops-agent-5d6878f6c-xwdb). kubectl config current-context was unset, but the in-cluster Kubernetes credentials reached the live node 72602-minipc, which is Ready at 192.168.0.25 on v1.34.6+k3s1.
• The mailu namespace did not exist, so kubectl create namespace mailu created it. A second check confirmed that mailu/mailu-secrets did not exist before creation.
• Generated secret-key with openssl rand -hex 32 and initial-account-password with openssl rand -base64 24 in shell memory, then ran kubectl create secret generic mailu-secrets -n mailu --from-literal=secret-key="$secret_key" --from-literal=initial-account-password="$initial_account_password". No credential value was written to a file, command output, logs, Git, or this documentation.
• Safe verification confirmed namespace mailu is Active; Secret metadata is name=mailu-secrets, namespace=mailu, type=Opaque, and keys initial-account-password and secret-key. Secret data values were not read or output.
• No ArgoCD sync was run, and no DNS, security-group, or reverse-tunnel changes were made. Rollback, only with explicit authorization: kubectl -n mailu delete secret mailu-secrets. The Secret was not deleted.

2026-07-28: black-box verification of filing-site upload policy

• Ran the checks with curl on 72602-minipc (hostname=72602-minipc, context default). The live route is https://72602.space/; GET http://72602.space/ returned 308 with Location: https://72602.space, and the HTTPS home returned 200 text/html (19,881 bytes) containing data-upload="aaron", data-upload="licorice", and data-upload="yakult".
• Anonymous GET https://72602.space/photos/{aaron,licorice,yakult}/ each returned 200 application/json with []; the corresponding HEAD requests each returned 200 application/json.
• The one temporary test object was https://72602.space/photos/aaron/verification-1785228022571821377.png, a valid 1x1 PNG of 68 bytes. Anonymous PUT and wrong-credential PUT each returned 401 text/html; PUT with the temporary uploader credential held in shell memory since Secret creation returned 201. The follow-up Aaron listing returned 200 application/json and showed the file as type=file, size=68; the image GET returned 200 image/png with 68 bytes.
• Authenticated DELETE on the temporary object and authenticated MKCOL, MOVE, COPY, and POST on the Aaron directory were all rejected with 403 text/html; no authenticated DELETE succeeded. Cleanup used kubectl -n application exec filing-site-55cff975bf-z67xw -- rm -f -- /data/files/aaron/verification-1785228022571821377.png. The post-cleanup Aaron listing returned 200 application/json with [], and the exact path was absent in the Pod.
• Live resource checks passed: Deployment filing-site is 1/1 available, Pod filing-site-55cff975bf-z67xw is Running and ready with zero restarts, Ingress filing-site uses class nginx for 72602.space, and 72602.space-tls is Ready. PVC filing-site-photos is Bound to a 5Gi local-path PV.
• In the nginx container (uid=101, gid=101), /data/files/aaron, /data/files/licorice, and /data/files/yakult exist and are writable with mode 775; /data/.tmp exists and is writable with mode 770. No Kubernetes Secret value was read, no manifest was changed, no ArgoCD sync was run, and no test object was retained. Recheck with the same curl method matrix and kubectl exec path test; rollback is limited to deleting the exact temporary path if a test object remains. Do not delete the PVC or Secret.

2026-07-28: sync filing-site photo albums from ops-docs

• Confirmed 72602-minipc, context default, node 72602-minipc Ready at 192.168.0.25 (v1.34.6+k3s1); live route is https://72602.space/ through the nginx ingress class.
• Hard-refreshed argocd/ops-docs with argocd app get ops-docs --hard-refresh --insecure --grpc-web (the installed CLI is v3.3.8 and does not support argocd app refresh --hard), then ran argocd app sync ops-docs --revision main --assumeYes --insecure --grpc-web.
• The requested baseline was 98dbe94, but origin/main advanced during the operation to 07f0e515feb7379ca79516a6c31f0e41be5a04b4 (fix: increase sub2api ingress body timeout); the final sync used that remote main revision. ArgoCD finished Succeeded, Synced, and Healthy from 16:30:08 to 16:30:47 (+0800), with message successfully synced (no more tasks).
• ops-docs-build briefly remained in Init:0/1 while clone-repo fetched the repository. Both clone-repo and hugo exited 0; the hook Job reached the expected succeeded pods. No manifest fix was necessary.
• application/filing-site rollout completed. Pod filing-site-55cff975bf-z67xw is 1/1 Running; init-albums completed with exit 0, and nginx is ready with zero restarts. Deployment conditions Available=True and Progressing=True are present.
• PVC filing-site-photos is Bound to a 5Gi local-path PV. Ingress annotations remain issuer lets-encrypt, SSL redirect enabled, body size 25m, request buffering off, and read/send timeouts 120; TLS Secret is 72602.space-tls.
• Read-only verification: nginx -t reported syntax ok and test successful inside the Pod; HTTPS GET https://72602.space/ returned HTTP 200 with text/html from 47.110.67.161. Recent events show successful local-path provisioning, old ReplicaSet scale-down/new ReplicaSet scale-up, and Ingress scheduled for sync.
• No Kubernetes Secret value was read, and no PUT/upload request or resource deletion was performed. Rollback requires explicit authorization and review of the current main: syncing the captured pre-sync revision 98dbe94 would also roll back later commits such as 07f0e51; do not delete the PVC or Secret.

2026-07-28: create filing-site upload authentication Secret

• Confirmed 72602-minipc as the active node and found no existing application/filing-site-upload-auth Secret.
• Generated the uploader password in shell memory with openssl rand -hex 18, generated an nginx-compatible apr1 hash with openssl passwd -apr1, and applied application/filing-site-upload-auth with key htpasswd. No credential material was written to disk or Git.
• manifests/filing-site.yaml references this Secret for the filing-site Deployment, but does not define the Secret; no ArgoCD sync was required.
• Safe verification confirmed metadata name=filing-site-upload-auth, namespace=application, type=Opaque, and key htpasswd without reading its value. The Deployment rollout succeeded with its Pod 1/1 Running.
• Ingress remains nginx at https://72602.space/; certificate 72602.space-tls is Ready under lets-encrypt, and the public HTTPS check returned HTTP 200.
• Rollback, only with explicit authorization: kubectl -n application delete secret filing-site-upload-auth.

2026-07-16: reset csst and update N8N webhook host

• Deleted and recreated the csst namespace. Only the namespace default ServiceAccount and kube-root-ca.crt ConfigMap remain.
• Changed N8N WEBHOOK_URL, webhook worker URL, Ingress rule, and TLS DNS name from webhook.72602.online to webhook.n8n.72602.online.
• Synced ArgoCD application argocd/n8n; main, webhook, MCP webhook, and worker rollouts completed.
• cert-manager completed HTTP-01 validation and issued the updated certificate.

2026-07-16: migrate OpenCode web to k3s

• Replaced the host systemd process and static EndpointSlice with the application/ops-agent workload.
• The Pod mounts /home/aaron/Ops/docs at /workspace, loads the project .opencode/opencode.json, and persists sessions in the opencode-data PVC.
• Image ay-dev/ops-agent:0.2.3 uses glibc and contains OpenCode 1.18.16, kubectl 1.34.6, Argo CD CLI 3.3.8, VibeGuard, DCP, and Goal Mode.
• OpenCode native Basic Auth protects both Ingress and cluster-internal access. Anonymous HTTPS returns 401; authenticated HTTPS returns 200.
• An Nginx sidecar publishes the Ops Agent browser title and proxies Terminal WebSocket and event streams.

2026-07-16: remove Langfuse and refresh Homepage

• Permanently removed the seven unused Langfuse PVCs (56 GiB) and six residual Secrets from monitor.
• Removed Langfuse and pgAdmin from Homepage and added the OpenCode operations agent.
• Restored the ArgoCD Homepage widget by binding the readonly API account to role:readonly.

2026-07-17: align Ops resource names

• Renamed the Hugo workload and its Service, ConfigMap, build Job, and Ingress resources to ops-docs; retained hugo-docs-pvc to preserve generated content.
• Renamed the OpenCode-based workload, Service, Ingress, proxy ConfigMap, manifest directory, and Dockerfile to ops-agent; retained existing opencode-* PVC and Secrets to preserve sessions and credentials.
• Replaced the manual local-proxy-bridge with the GitOps-managed argocd-egress-proxy; ArgoCD repo-server uses its cluster Service while applications can continue using host port 17890.

Default Verification Commands

# 公网入口
curl -vI http://argocd.72602.space
curl -vkI https://argocd.72602.space

# k8s 资源
kubectl get ingress -A -o wide
kubectl get svc -A -o wide
kubectl get pods -A -o wide
kubectl get certificate,certificaterequest,order,challenge -A

# 主机端口
sudo ss -lntp | grep -E ':80|:443|:8443|:32080|:32443|:7890|:17890|:9090'

# ECS 转发
sudo iptables -t nat -L PREROUTING -n -v --line-numbers
sudo iptables -t nat -L DOCKER -n -v --line-numbers

# Egress proxy 完整性：先读实际端口，再做一次已知 204 检查
CLASH_HOME=/home/aaron/clashctl
. "$CLASH_HOME/scripts/cmd/clashctl.sh"
clashctl status
proxy_port="$("$CLASH_HOME/bin/yq" '."mixed-port" // .port // 7890' \
  "$CLASH_HOME/resources/runtime.yaml")"
curl --proxy "http://127.0.0.1:${proxy_port}" \
  --connect-timeout 5 --max-time 12 --silent --show-error \
  --output /dev/null --write-out 'proxy_http_code=%{http_code}\n' \
  https://www.gstatic.com/generate_204
kubectl exec -n n8n deploy/n8n -- sh -c \
  'test -n "$HTTP_PROXY" && test -n "$HTTPS_PROXY"'

# SSH 隧道
journalctl --user -u reverse-tunnel-ecs-10021.service --since "1 hour ago" --no-pager
journalctl --user -u reverse-tunnel-ecs-10022.service --since "1 hour ago" --no-pager

Aliyun

• OSSutil
• ECS DNS

Shared Clash/Mihomo Operations

zjlab-ubuntu and 72602-minipc both use nelvko/clash-for-linux-install from /home/aaron/clashctl. This page is the shared host-level proxy runbook; cluster-specific bridges and application proxy variables remain documented on their respective cluster pages.

Command Loading

clashctl is a Bash function loaded by /home/aaron/.bashrc, not an executable installed in PATH. Non-interactive SSH and agent shells do not automatically load it, so command -v clashctl or a direct clashctl status can incorrectly report that the command is missing.

Prefer explicit loading in automation:

CLASH_HOME=/home/aaron/clashctl
. "$CLASH_HOME/scripts/cmd/clashctl.sh"
clashctl --help
"$CLASH_HOME/bin/mihomo" -v
clashctl status

bash -ic 'clashctl status' is acceptable for a quick remote check, but it can emit harmless job-control warnings when SSH has no TTY. Explicit sourcing gives cleaner, more predictable automation output.

Read-Only Preflight

Run this once on the target host before trying alternate ports, editing proxy variables, or restarting anything:

CLASH_HOME=/home/aaron/clashctl
. "$CLASH_HOME/scripts/cmd/clashctl.sh"

hostname
"$CLASH_HOME/bin/mihomo" -v
clashctl status

proxy_port="$("$CLASH_HOME/bin/yq" \
  '."mixed-port" // .port // 7890' \
  "$CLASH_HOME/resources/runtime.yaml")"
printf 'proxy_port=%s\n' "$proxy_port"

curl --proxy "http://127.0.0.1:${proxy_port}" \
  --connect-timeout 5 --max-time 12 \
  --silent --show-error --output /dev/null \
  --write-out 'proxy_http_code=%{http_code}\n' \
  https://www.gstatic.com/generate_204

The expected result is one active mihomo process and HTTP 204. This test uses the configured local port and proves an end-to-end HTTPS request through the proxy; it is more useful than probing a sequence of guessed ports.

If it fails, inspect the existing log without opening an interactive pager:

tail -n 100 /home/aaron/clashctl/resources/mihomo.log

Do not use clashctl log in unattended work because it invokes less.

Three Independent States

Do not collapse these into one “proxy is enabled” judgment:

1. clashctl status checks whether the mihomo core process exists.
2. ._custom.system-proxy.enable is only the saved proxy preference in mixin.yaml.
3. HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY are environment variables inherited by the current shell or process.

Inspect the latter two without printing proxy URLs or authentication:

CLASH_HOME=/home/aaron/clashctl

"$CLASH_HOME/bin/yq" -r \
  '"saved_proxy_flag=" + ((._custom.system-proxy.enable // false) | tostring)' \
  "$CLASH_HOME/resources/mixin.yaml"

env | awk -F= \
  'BEGIN { IGNORECASE=1 } /^(http|https|all|no)_proxy=/{ print $1 }' | \
  sort -fu

It is valid for the core and explicit curl --proxy request to be healthy while a fresh non-interactive shell has no proxy variables. Do not call clashctl proxy on merely to make a diagnostic shell look enabled; that command changes saved state. Use an explicit per-command proxy unless the requested operation requires persistent shell proxy configuration.

Systemd Boundary

clashctl has no version subcommand. Query the installed core directly:

/home/aaron/clashctl/bin/mihomo -v

Do not use a systemd unit state as a substitute for clashctl status and the end-to-end request. ZJLAB currently has no clashctl-on.service. On 72602, clashctl-on.service is an enabled user-level Type=oneshot bootstrap; active (exited) means only that the bootstrap command completed. It does not supervise or prove the health of the current mihomo process.

Path Boundaries

• Host-local tools on both machines use 127.0.0.1:<runtime-port>.
• On 72602, Pods use http://192.168.0.25:17890 or http://argocd-egress-proxy.argocd.svc.cluster.local:17890. The socat bridge forwards to host-local mihomo.
• Do not configure a 72602 Pod with 192.168.0.25:7890 while allow-lan is false.
• The 72602 bridge is not evidence that the same Pod path exists in ZJLAB. Verify the target cluster’s live proxy path separately.

Command Safety

Do not stop or restart the proxy merely because an unrelated external endpoint is slow. Confirm core status, the configured port, one known generate_204 request, and the recent log first.

2026-08-14 Audit Snapshot

Read-only checks on both zjlab-ubuntu and 72602-minipc found:

• the installed mihomo process active;
• Mihomo Meta v1.19.17 on linux amd64;
• HTTP/mixed port 7890, SOCKS port 7891, and controller port 9090;
• allow-lan: false and TUN disabled;
• the dynamic local-port preflight above returned HTTP 204 on each host.

ZJLAB had no clashctl-on.service. The 72602 user unit was enabled and active (exited), but was a Type=oneshot bootstrap rather than the supervisor of the live core. Fresh non-interactive shells did not inherit proxy environment variables, which did not prevent the explicit local proxy checks from passing.

Treat these as dated health observations. The runtime configuration remains authoritative for current ports and modes.

Huawei

Tencent

Zhejianglab

• 👨‍💻Schedmd Slurm
  • Build & Install
    • Install On Debian
    • Install On Ubuntu
    • Install From Binary
    • Install From Helm Chart
    • Install From K8s Operator
    • Try OpenSCOW
  • CheatSheet
    • Common Environment Variables
    • File Operations
    • Submit Jobs
  • Configuration Files
    • Slurm Configuration File
    • Slurmdbd Configuration File
  • MPI Libs
    • Test Intel MPI Jobs
    • Test Open MPI Jobs
• Astro Code
• Data Warehouse

Scope

ZJLAB operational inventory and network details are private. Public pages contain reusable application guidance only; verify dynamic state against the live cluster before applying a runbook.

Access

ssh zjlab-ubuntu-local hostname       # when running on ZJLAB
ssh zjlab-ubuntu-local 'kubectl get nodes'
ssh zjlab-ubuntu-proxy hostname       # when running on 72602

Use zjlab-ubuntu-local for direct access from ZJLAB and zjlab-ubuntu-proxy for the forwarded path from 72602. These are SSH config aliases, not DNS names: validate configuration with ssh -G and reachability with SSH, not with getent hosts. The aliases are provisioned from private inventory and use an ECS ProxyJump to loopback-only reverse SSH listeners. Do not publish their resolved endpoints, ports, users, internal topology, or service names.

Detailed inventory and tunnel recovery procedures are maintained in the private ops-private repository with SOPS-encrypted values.

Tunnel Health

ECS runs an independent check-and-alert-only monitor for the approved primary and backup loopback listeners. For each label it requires exactly one listener, loopback-only binding, sole sshd ownership, an independent owner, a short-timeout SSH banner, and a stable hashed owner signature. Alerts start only after ten consecutive failures.

The monitor stores root-only state and emits labels and fixed reason codes only. It never restarts or kills a tunnel and never changes sshd, firewall, cloud network policy, DNS, keys, or endpoints. Real configuration and rollback details remain in the private SOPS inventory; public pages must not reproduce them.

The ZJLAB tunnel initiators use enabled system-level services so they recover after a host reboot. User-level legacy tunnel services must remain stopped and disabled; Linger=yes starts the user manager but does not start disabled services. If both labels fail together, inspect host boot and service-manager state first, then recover backup before primary. Verify each alias and its loopback listener before proceeding to the next label.

For external availability history, report the existing primary and backup check results to separate Uptime Kuma Push monitors. Do not expose the relay’s loopback listeners just to make them reachable by Kuma. The Push URLs and the ECS monitor configuration belong in the private SOPS inventory; see the Uptime Kuma setup for the public setup requirements.

The ECS-side integration is active: ECS listener 10023 reports the primary check and listener 10024 reports the backup check. This is separate from the 72602 public tunnel listeners on 10021 and 10022, which are not Kuma Push sources for this monitor.

The stable access map is:

• 10023 is the ZJLAB primary SSH reverse listener.
• 10024 is the backup listener.
• 10025 is a restricted break-glass maintenance listener.

All three are loopback-only on the ECS host and are reached through the ECS ProxyJump path. They must never be added to the ECS security group or exposed publicly. The maintenance listener is provided by zjlab-loopback-maintenance.service and uses a separate restricted SSH account and key; use the local zjlab-maintenance alias only for recovery. Client configurations should use HostName 127.0.0.1 for these ECS loopback targets rather than relying on localhost address-family resolution.

ECS applies ClientAliveInterval 30 and ClientAliveCountMax 10 to the zjlab-tunnel and zjlab-maintenance accounts. This releases stale remote forward listeners quickly after a broken connection without changing the global SSH policy.

The ZJLAB tunnel initiators are the system-level services located at /etc/systemd/system/zjlab-loopback-reverse-primary.service and /etc/systemd/system/zjlab-loopback-reverse-backup.service. These units invoke /usr/bin/ssh directly (not autossh) with Restart=on-failure, RestartSec=30s for the primary, RestartSec=45s for the backup, and KillMode=control-group. Legacy user-level services reverse-tunnel-ecs.service and reverse-tunnel-ecs-10024.service must remain stopped and disabled; running them alongside the system units causes listener ownership conflicts and reconnect loops.

When both monitors report ssh_banner_failed, first verify the current ZJLAB egress IPv4 and the ECS security-group allowlist for TCP 22, then check ECS sshd and the established SSH child/session. A systemd active state alone only proves that the supervisor is running; it does not prove that the reverse listener has been established.

The private checker keeps the Push request as a best-effort reporting path. Healthy checks send up; failed checks send down with a fixed reason. Push HTTP failures do not change listener judgment, failure counters, DingTalk debounce, or tunnel lifecycle. The checker merges existing query parameters when constructing the request and does not log the Push value. A normal tunnel reconnect changes the sshd PID and socket identity; those changes are not treated as failures when the listener, owner, bind address, and SSH banner are healthy. The checker completes an SSH key exchange with ssh-keyscan before closing its probe connection; do not replace it with a raw TCP banner read, which can leave CLOSE-WAIT channels on the reverse listener.

DingTalk sends one failure notification after ten consecutive failures for a label. The message includes a fixed, redacted recovery action. After a notified failure becomes healthy, the checker sends one recovery notification; if that send fails, the alert state is retained and the next healthy check retries it. Short failures that never cross the alert threshold do not generate a recovery message.

Deployment verification on 2026-09-05 confirmed the primary, backup, and maintenance aliases returned zjlab-ubuntu; the ECS listeners were unique, loopback-only, and returned valid SSH banners. The checker dry-run and live service run were healthy for both labels across more than two complete 60-second timer cycles. Rollback restores the root-only ECS backup and encrypted private inventory backup, then restarts only zjlab-tunnel-healthcheck.service; tunnel units are not restarted as part of monitoring rollback.

Preflight

For host-level proxy checks, follow the shared Clash/Mihomo runbook before trying ports or changing proxy variables. The installed clashctl is a shell function, so non-interactive sessions must source it explicitly.

# On ZJLAB, use zjlab-ubuntu-local; on 72602, use zjlab-ubuntu-proxy.
ZJLAB_SSH_ALIAS=zjlab-ubuntu-local
ssh "$ZJLAB_SSH_ALIAS" 'kubectl config current-context'
ssh "$ZJLAB_SSH_ALIAS" 'kubectl get nodes'
ssh "$ZJLAB_SSH_ALIAS" 'kubectl get namespace'
ssh "$ZJLAB_SSH_ALIAS" 'kubectl get applications.argoproj.io -A'
ssh "$ZJLAB_SSH_ALIAS" 'kubectl get ingress,certificate -A'

Independent Prometheus Deployment Attempt

An independent Prometheus deployment was prepared for ZJLAB with the public Prometheus Community prometheus chart version 29.18.0, ArgoCD Application zjlab-prometheus, and destination namespace monitoring. Helm rendering and Kubernetes server-side dry-run succeeded, including the password-file based remote-write mount and explicit kube-state-metrics and node-exporter scrape jobs.

The Application reached Synced but remained Healthy: Progressing. The 30Gi local-path PVC stayed Pending: the local-path provisioner timed out while creating its helper Pod and did not bind the volume. Consequently the Prometheus server, kube-state-metrics, and node-exporter workloads did not reach the verification gate, so readiness, targets, remote-write delivery, and the receiving-cluster queries were not claimed as successful.

The attempt was rolled back by deleting only the newly created zjlab-prometheus Application, its monitoring/zjlab-prometheus-server PVC, the runtime remote-write Secret, and the empty monitoring namespace. Existing applications, tunnels, network policies, and metrics-server were not changed. Before retrying, verify local-path provisioning on the selected node and repeat the full readiness, target, remote-write, and receiving-cluster query checks.

Prometheus Retry (2026-08-05)

The retry used a private node selector for the previously diagnosed healthy local-path node. The 30Gi PVC bound immediately, and the Prometheus server was scheduled on the same node. Kube-state-metrics and both node-exporter Pods reached Running.

The Prometheus server then exited with code 2 and entered CrashLoopBackOff. This is a stop condition, so readiness, targets, remote-write queue health, and receiving-cluster queries were not claimed as successful. The Application, PVC, runtime Secret, and monitoring namespace created for this retry were rolled back. Existing applications, tunnels, network policies, and metrics-server were not changed. The next retry must capture the Prometheus startup error before cleanup and correct the rendered configuration without publishing credentials.

Prometheus Deployment (2026-08-05)

The deployment was recreated as ArgoCD Application zjlab-prometheus using Prometheus Community chart 29.18.0. The rendered server arguments contain the chart-default web.enable-lifecycle flag exactly once; no custom server.extraArgs was configured. The runtime remote-write Secret contains only the password file and is mounted read-only; credentials are not part of the Application values.

ArgoCD is Synced and Healthy at revision 29.18.0. The 30Gi local-path PVC is Bound, the Prometheus server is co-located with that volume on the private selected node, the Prometheus and kube-state-metrics Deployments are 1/1 Ready, and both node-exporter DaemonSet Pods are Ready with zero restarts. The server Service and the KSM/node-exporter Services are ClusterIP; no Ingress, NodePort, or admin API was enabled.

Prometheus /-/ready returned HTTP 200. The targets API showed the Prometheus, kube-state-metrics, node-exporter, Kubernetes API server, nodes, cadvisor, and Kubernetes pod/service discovery jobs up with no scrape errors. The rendered configuration contains external_labels.cluster=zjlab, the remote-write password-file path, and no password value. Across two complete 30-second cycles, remote-write samples increased while failed and retried samples stayed at zero.

The receiving 72602 Prometheus queries were completed after deployment: up{cluster="zjlab"} returned 18 samples, kube_node_status_condition{cluster="zjlab"} returned 54 samples, and kube_pod_info{cluster="zjlab"} returned 72 samples. The newest ZJLAB sample was approximately five seconds old at verification time, and all three queries were served by the existing 72602 Prometheus datasource used by Grafana. No Grafana datasource change was required.

The receiving Prometheus reported zero failed and retried remote-write samples. After the user-authorized 2026-08-14 metrics reset, the local monitoring/zjlab-prometheus-server TSDB PVC was recreated while this remote-write configuration remained unchanged. The old local history was not backed up and is intentionally unrecoverable. The sender queue drained and the 72602 receiver again exposed fresh cluster="zjlab" samples; continue watching the pending queue if its delay grows.

Prometheus Storage Diagnosis (2026-08-05)

The read-only follow-up found two Ready nodes with no taints and with MemoryPressure, DiskPressure, and PIDPressure all false. The default local-path StorageClass uses rancher.io/local-path, WaitForFirstConsumer, and Delete reclaim policy. Its Rancher v0.0.35 provisioner is 1/1 Ready. The provisioner configuration has only the default root /var/lib/rancher/k3s/storage; its helper image is BusyBox 1.37.0, and the configured setup creates the volume directory and restricts its parent.

Existing Bound local-path volumes cover both nodes, and the node filesystem and inode checks showed substantial headroom. Existing local-path mounts were also writable from their consuming workloads. This does not support a root capacity or permission failure as the cause of the Prometheus incident.

The retained helper events use the helper-pod-create-pvc-<PVC-UID> naming pattern. The image was already available, but helper container creation/startup and retries exceeded the provisioner’s 120-second create-process timeout. Later retries included ContainerCreating and failed log-stream reads before the Prometheus resources were removed. No helper Pod, PVC, PV, or Prometheus Application is currently left behind. The evidence points first to helper scheduling/runtime latency or a local-path/K3s compatibility issue on the selected worker; the retained evidence does not prove a filesystem fault.

nfs-data is currently 1/1 Ready with the v4.0.2 provisioner (Helm chart 4.0.18), an active leader endpoint, Immediate binding, Retain reclaim policy, and archiveOnDelete=true. Multiple existing nfs-data PVCs are Bound, and the NFS mount is currently read-write with healthy capacity and inode headroom. It remains a single-backend/single-node dependency, so NFS is an acceptable fallback for Prometheus only after confirming its latency and failure policy; it is not automatic high availability.

The minimum reliable retry is to repair and verify helper startup on a deliberately selected, healthy local-path node, and to make the Prometheus workload’s node selection explicit so the local volume and workload remain co-located. Rollback is to remove only the new ArgoCD Application and its new resources; do not remove existing local-path data. NFS is the secondary option, and ephemeral storage is not recommended for Prometheus. Direct worker-host journal and host-permission inspection remains a verification gap because this diagnostic session was restricted to the approved canonical ZJLAB SSH entry point and did not create a debug Pod.

Prometheus Storage Follow-up (2026-08-05)

The approved node proxy exposed host system.journal and container-log directories for both nodes. The historical helper had already been removed, and no matching kubelet, containerd, CRI, mount, image, or sandbox error was retained for the helper timeout window. The control-plane journal did contain repeated PartialObjectMetadata watch errors, but no evidence connected them to the local-path helper failure. The Kubernetes node image inventory no longer advertises the helper image; the historical kubelet event is the available evidence that it was already cached during the incident.

The strongest retry candidate is the control-plane node that already hosts the local-path provisioner and several successfully Bound local-path volumes. It has the standard Linux and control-plane/etcd labels, no taint, and healthy Ready/pressure conditions. The trade-off is that Prometheus would share the node with control-plane workloads, and loss of that node makes its local data unavailable. Use the private value of its kubernetes.io/hostname label in the deployment; do not publish that value.

For a retry, the server values should include the following shape, with <selected-local-path-node> replaced only in the private manifest:

server:
  nodeSelector:
    kubernetes.io/hostname: <selected-local-path-node>
  persistentVolume:
    enabled: true
    storageClass: local-path
    accessModes:
      - ReadWriteOnce
    size: 30Gi

The retry gate is: the PVC must reach Bound and receive node affinity for the selected node before 120 seconds; the helper must reach container started/completed without ContainerCreating, mount, image, sandbox, or CRI errors; and the Prometheus server must schedule on the same node and become Ready. Any repeated runtime error, a helper still in ContainerCreating at 60 seconds, or another ProvisioningFailed at 120 seconds is a stop condition. Roll back the new Application and its new resources, then use nfs-data only after accepting its single-backend failure risk. No runtime or storage change was made during this follow-up.