Subsections of Proxy

Open WebUI

Archived

The 72602 OpenWebUI deployment is retired and has no current installation route. argocd/open-webui does not exist, and the ai namespace has no OpenWebUI resources. txt2img.agent.72602.online is retired and has no working endpoint; its DNS record, certificate, TLS Secret, and unreferenced data claims were removed.

The repository file manifests/application/open-webui.yaml is a historical, non-deployed manifest. It is not consumed by the current GitOps setup and still points to the retired route. Do not use it as an installation source.

Historical Record (2026-05)

The service was previously deployed in the 72602 cluster through an ArgoCD Helm Application in namespace ai:

  • Route: txt2img.agent.72602.online (now retired)
  • Helm chart: open-webui, version 14.5.0
  • Image: m.daocloud.io/ghcr.io/open-webui/open-webui:0.9.5
  • AI backend: sub2api.application.svc:8080/v1 (OpenAI-compatible)
  • WebSocket manager: shared Redis in namespace storage
  • Persistence: SQLite on a 2Gi local-path PVC

Any commands or verification steps from the former deployment procedure are historical and non-runnable. They are intentionally not reproduced here.

May 12, 2026

Ollama Moderation Gateway

Overview

The Ollama Moderation Gateway provides an OpenAI-compatible moderation endpoint backed by Ollama models. It is deployed via the 72602 ArgoCD GitOps pipeline using the Helm chart ollama-moderation-gateway (version 0.1.0).

  • ArgoCD Application: argocd/ollama-moderation-gateway (child of argocd/ops-docs)
  • Namespace: moderation
  • Service: ollama-moderation-gateway (port 8000)
  • Ingress: moderation.llm.72602.space (TLS via cert-manager)

Current Operational State: Deployment ollama-moderation-gateway desired replicas: 3, available replicas: 3. Service reports three Ready endpoints. Single-node resource metrics are healthy. The image, runtime configuration and Ingress remain unchanged; only replicaCount was changed.

Subsections of Ollama Moderation Gateway

Install (ArgoCD)

🚀Installation

Environment
Install By

Preliminary

1. Kubernetes is installed; if not, check 🔗link


2. Helm is installed; if not, check 🔗link


3. ArgoCD is installed; if not, check 🔗link


1.verify source and create namespace

Details
git -C /home/aaron/Ops/docs fetch origin main
git -C /home/aaron/Ops/docs \
  show origin/main:manifests/ollama-moderation-gateway-argocd.yaml >/dev/null

argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --timeout 300

kubectl wait --for=jsonpath='{.status.phase}'=Active \
  namespace/moderation --timeout=120s

The parent ops-docs Application reads the manifests path and creates the child Application. The child creates namespace moderation through CreateNamespace=true.

2.prepare runtime secret

Details
test -s /home/aaron/Ops/docs/.moderation.env
test "$(stat -c %a /home/aaron/Ops/docs/.moderation.env)" = 600

kubectl -n moderation create secret generic ollama-moderation-gateway \
  --from-env-file=/home/aaron/Ops/docs/.moderation.env \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n moderation get secret ollama-moderation-gateway --output=name

The ignored .moderation.env file must contain OLLAMA_API_KEYS and MODERATION_API_KEYS. Secret values remain outside Git and terminal output.

3.verify GitOps source and sync child

Details
# confirm parent application source
kubectl -n argocd get application ops-docs \
  -o jsonpath='{.spec.source.repoURL}{"\n"}{.spec.source.path}'

# refresh and sync the child (ollama-moderation-gateway)
argocd app get ollama-moderation-gateway --hard-refresh
argocd app sync ollama-moderation-gateway
argocd app wait ollama-moderation-gateway --sync --health --timeout 600

4.verify deployment, service, ingress and certificate

Details
# chart source details
kubectl -n argocd get application ollama-moderation-gateway \
  -o jsonpath='{.spec.source.repoURL}{"\n"}{.spec.source.path}{"\n"}{.spec.source.targetRevision}{"\n"}'

# image and release verification
kubectl -n moderation get deployment ollama-moderation-gateway \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"="}{.image}{"\n"}{end}'

# rollout status
kubectl -n moderation rollout status deployment/ollama-moderation-gateway --timeout=600s
kubectl -n moderation get pods,svc,ingress
kubectl -n moderation get certificate,certificaterequest,order,challenge
kubectl -n moderation get endpointslice -l kubernetes.io/service-name=ollama-moderation-gateway

# health checks
curl -fsS https://moderation.llm.72602.space/health
curl -fsS https://moderation.llm.72602.space/readyz

# authenticated moderation request; do not echo the key or response body
set +x
read -rsp 'Gateway API key: ' GATEWAY_KEY; printf '\n'
curl -fsS -X POST https://moderation.llm.72602.space/v1/moderations \
  -H "Authorization: Bearer ${GATEWAY_KEY}" \
  -H "Content-Type: application/json" \
  --json '{"model":"moderation-fast","input":"hello"}' \
  | jq -e '.results | length == 1' >/dev/null
unset GATEWAY_KEY
Details

Current Operational State: Deployment ollama-moderation-gateway desired replicas: 3, available replicas: 3. Service reports three Ready endpoints. Single-node resource metrics are healthy. The image, runtime configuration and Ingress remain unchanged; only replicaCount was changed.

Expected release values: chart ollama-moderation-gateway version 0.1.0 and image ghcr.io/aaronyang0628/ollama-moderation-gateway@sha256:6957f8a32ad93577500c6952d7dd0b9ae970672b82570144852c5fe5766773c5. The TLS certificate moderation.llm.72602.space-tls should be Ready with expiry 2026-12-13T00:55:24Z.

📦Rollback Guidance

If a deployment must be reverted, create a reviewed Git revert for the deployment source commit, push it, and sync the parent application. Runtime secrets and TLS certificates are external resources and are not modified by the rollback.

# create a reviewed revert in the docs repository
cd /home/aaron/Ops/docs
git fetch origin main
git revert --no-edit 39c48b3
git push origin main
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300

After rollback, re-run the verification steps above to ensure the service is healthy.

Sub2API

Web Page

sub2api web page (https://token.72602.space)

Current State

  • Git owner: argocd/ops-docs, source https://github.com/AaronYang0628/docs.git, path manifests
  • ArgoCD Application: argocd/sub2api
  • Namespace: application
  • OCI chart: ghcr.io/aaronyang0628/helm-chart-mirror/sub2api (0.1.18)
  • Application image: ghcr.io/wei-shaw/sub2api@sha256:996a0ea43500550f233a2f653de58dcd77f42d3e244805894fcccdd32b612147
  • Public API and OAuth host: token.72602.space
  • Ingress class: nginx
  • TLS certificate: token.72602.space-tls (Ready)
  • External PostgreSQL: postgresql.database.svc.cluster.local:5432 (database/user: sub2api)
  • Application PVC: sub2api-data, 10Gi, local-path, RWO
  • Redis PVC: 8Gi, local-path, RWO; AOF is enabled
  • RollingUpdate: maxUnavailable: 0, maxSurge: 1

Google OAuth Route

  • Use the 72602 public URL: https://token.72602.space.
  • Google OAuth is enabled in the live application. The start endpoint is /api/v1/auth/oauth/google/start and the callback URI generated by the running application is: https://token.72602.space/api/v1/auth/oauth/google/callback.
  • The application sets the OAuth state, redirect, and provider cookies with HttpOnly, Secure, SameSite=Lax, and Path=/api/v1/auth/oauth. The frontend callback page is /auth/oauth/callback.
  • The Pod requires the cluster egress proxy for Google token exchange. The GitOps source sets HTTP_PROXY and HTTPS_PROXY to http://192.168.0.25:17890 and bypasses that proxy for cluster-local ranges.
  • sub2api.home.geekcity.tech:32443 is a separate external endpoint, not the 72602 ingress. It returns 404 for the 72602 OAuth start path and must not be used for this login flow.
  • If Google reports a redirect URI mismatch, register only the exact callback URI above in the Google OAuth client. Google Cloud Console changes are not managed by this Kubernetes repository.

Docs

  • Backup & Restore

    Sub2API pre-upgrade backup and recovery runbook

    • Install (ArgoCD)

      Deploy Sub2API through the 72602 GitOps parent and OCI Helm child Application

      • Ops

        GitOps upgrades and daily operations for Sub2API

        Subsections of Sub2API

        Backup & Restore

        Scope

        • Database: PostgreSQL (database/postgresql-0), DB/user sub2api
        • Git source: manifests/sub2api-argocd.yaml, owned by argocd/ops-docs
        • Current release: OCI chart 0.1.22, application 0.2.16
        • Application image: ghcr.io/wei-shaw/sub2api@sha256:4a0d0196d3450c71da159137e3419fc144df0c038bbe3b9e87dbbccaef10cab7 (Linux amd64)
        • Runtime data: application/sub2api-data, 10Gi, local-path, RWO
        • Redis data: 8Gi, local-path, RWO; AOF is enabled
        • Runtime Secrets: application/sub2api-auth, application/sub2api-external-postgresql, and application/sub2api-redis

        Sub2API executes PostgreSQL migrations automatically on startup. Migrations are forward-only, so every chart or image upgrade requires a verified pg_dump before the Git version change.

        Pre-Upgrade Backup

        1.create a protected operation directory

        BACKUP_ROOT=/home/aaron/Ops/backups/sub2api
BACKUP_DIR="${BACKUP_ROOT}/upgrade-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$BACKUP_DIR"
chmod 700 "$BACKUP_DIR"
printf '%s\n' "$BACKUP_DIR"

        Use the generated UTC directory for the entire operation. Do not hard-code a previous operation timestamp into future commands.

        2.dump PostgreSQL without printing its password

        set +x
PG_PASSWORD="$(kubectl -n application get secret \
  sub2api-external-postgresql \
  -o jsonpath='{.data.postgres-password}' | base64 -d)"
test -n "$PG_PASSWORD"

printf '%s\n' "$PG_PASSWORD" | \
  kubectl -n database exec -i postgresql-0 -- \
  sh -c 'IFS= read -r PGPASSWORD; export PGPASSWORD; exec pg_dump -U sub2api -d sub2api -Fc' \
  > "$BACKUP_DIR/sub2api.dump"

unset PG_PASSWORD

        The password is passed on stdin and is not written into the backup directory.

        3.capture application data and non-secret metadata

        kubectl -n application exec deployment/sub2api -- \
  tar -C /app/data -czf - . > "$BACKUP_DIR/sub2api-data.tgz"

git -C /home/aaron/Ops/docs fetch origin main
git -C /home/aaron/Ops/docs rev-parse origin/main \
  > "$BACKUP_DIR/git-revision.txt"
git -C /home/aaron/Ops/docs show origin/main:manifests/sub2api-argocd.yaml \
  > "$BACKUP_DIR/sub2api-argocd.yaml"

kubectl -n application get pvc \
  -l app.kubernetes.io/instance=sub2api -o yaml \
  > "$BACKUP_DIR/pvc-metadata.yaml"

        Do not export Kubernetes Secret objects into this directory. Back up Secret values only through the approved secret-management process. Redis AOF supports restart recovery on its PVC, but it is not a substitute for the PostgreSQL dump. Do not copy live AOF files as if they were a consistent database backup.

        4.verify artifacts before upgrading

        test -s "$BACKUP_DIR/sub2api.dump"
test -s "$BACKUP_DIR/sub2api-data.tgz"

kubectl -n database exec -i postgresql-0 -- pg_restore --list \
  < "$BACKUP_DIR/sub2api.dump" \
  > "$BACKUP_DIR/sub2api.dump.list"
test -s "$BACKUP_DIR/sub2api.dump.list"

sha256sum \
  "$BACKUP_DIR/sub2api.dump" \
  "$BACKUP_DIR/sub2api-data.tgz" \
  "$BACKUP_DIR/sub2api-argocd.yaml" \
  "$BACKUP_DIR/pvc-metadata.yaml" \
  "$BACKUP_DIR/git-revision.txt" \
  > "$BACKUP_DIR/SHA256SUMS"
sha256sum -c "$BACKUP_DIR/SHA256SUMS"

        Verified backup: 2026-08-14

        • Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260814T063611Z (directory mode 700, files mode 600). Only the shared PostgreSQL sub2api database was dumped; n8n and other databases were not touched.
        • The host had no pg_restore; PostgreSQL Pod database/postgresql-0 had pg_restore 18.3 and GNU tar 1.34. The dump was copied temporarily with kubectl cp, checked with Pod-local pg_restore --list, and the temporary Pod file was removed. The previous kubectl exec -i streaming validation was not reused.
        • Non-empty artifacts: sub2api.dump (19,640,824 bytes), sub2api-data.tgz (10,616,251 bytes), and sub2api.dump.list (77,853 bytes). Git revision, manifest, and PVC metadata were also captured.
        • sha256sum -c SHA256SUMS and pg_restore --list both succeeded.
        • The captured origin/main revision was ce838b424f10316fc604a4a21517c90c7b6b97ae. No failure occurred; no upgrade, Argo CD sync, or GitOps/cluster change was performed.

        Verified upgrade: 2026-09-05 (application 0.2.0)

        • Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260905T000035Z (upgrade commit 5230546610c437a02ee45f5b80cd804779c9170c). The scoped sub2api.dump (47,520,105 bytes) and sub2api-data.tgz (10,192,905 bytes) were non-empty; pg_restore --list and SHA-256 checks passed.
        • schema_migrations advanced from 268 to 277 records; the latest migration is 233_group_free_openai_fast.sql.
        • ArgoCD reported Synced/Healthy; Deployment was 1/1 on the pinned ghcr.io/wei-shaw/sub2api@sha256:271bb3b34661803681cabf54e99811ab8e248b0dd4c88b09ea1226e22dea5751 image.
        • Existing application and Redis PVCs remained Bound, Service endpoints were Ready, and public health/settings checks succeeded.
        • No rollback was required.

        Verified upgrade: 2026-09-05 (application 0.2.1)

        • Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260905T115349Z-178860 (upgrade commit 21d02345de47d75454841e464756f6cf9349cc3c). The scoped sub2api.dump and sub2api-data.tgz were non-empty; pg_restore --list and SHA-256 checks passed.
        • ArgoCD reported Synced/Healthy at 2026-09-05T13:03:41Z; Deployment was 1/1, Pod ready with 0 restarts, internal /health returned {"status":"ok"}, Service endpoint ready, and Redis StatefulSet 1/1.
        • Existing application and Redis PVCs remained Bound; health and settings checks succeeded.
        • No rollback was required.

        Verified upgrade: 2026-09-10

        • Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260910T061248Z; PostgreSQL dump 46,520,009 bytes, /app/data archive 10,214,969 bytes; backup artifacts and pg_restore --list, archive listing, SHA-256 all passed; GitOps commit 7bd44adb5ed1b61976b507e1f32bc990738782b2; sub2api ArgoCD target/observed 0.1.14 Synced/Healthy operation Succeeded; runtime verification: Deployment 1/1, Pod ready 0 restarts, chart sub2api-0.1.14/app 0.2.4, actual image digest matches, Service endpoint ready/serving, internal /health HTTP 200, Redis StatefulSet 1/1.
        • No rollback required.

        Verified upgrade: 2026-09-16

        • Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260916T053309Z (directory mode 700, files mode 600). PostgreSQL dump 27,593,406 bytes, /app/data archive 4,579,406 bytes; backup artifacts and pg_restore --list, archive listing, SHA-256 all passed; GitOps commit e2c0e4576276f6a299bbd27e80f40af883f8e877 (chore(sub2api): upgrade to chart 0.1.15); sub2api ArgoCD target/observed 0.1.15 Synced/Healthy operation Succeeded; runtime verification: Deployment 1/1, Pod sub2api-7c9c4b95bf-r6cpg Running/Ready, restarts 0, chart sub2api-0.1.15/app 0.2.5, actual image digest matches sha256:4c5dffab6e5ba4d3bd5382f19aad9654847b4e23de1a3d48e190146a3e6eb977, Service endpoint ready at 10.42.0.198:8080, internal /health HTTP 200 body {"status":"ok"}, Redis StatefulSet ready at 10.42.0.12:6379 with 1 historical restart.
        • No rollback required.

        Verified upgrade: 2026-09-20 (application 0.2.7)

        • Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260920T051850Z
        • Backup artifacts: sub2api.dump (29,365,379 bytes, SHA-256 d9a0fb1870cde3405d47765291190dbc0bf17bd724708dc1fd7f448a8027eace), sub2api-data.tgz (4,598,099 bytes, SHA-256 f6b1691f6ba771f415e78caebedb0da225ef665a7af22c3b9d9854c6f4e097f7), pg_restore --list (1,210 entries), archive listing (10 entries), checksums passed.
        • GitOps commit: 67d8fef5859ec579cee101da22cc4988ca789b84 (chore: upgrade sub2api)
        • ArgoCD status: Synced/Healthy, operation Succeeded, observed chart revision 0.1.16
        • Deployment: 1/1 ready; Pod running/ready with 0 restarts
        • Service endpoint: 10.42.0.224:8080
        • Internal /health HTTP 200 (body discarded)
        • Dedicated Redis StatefulSet: 1/1 ready; Redis Pod has 1 historical restart
        • No rollback required.

        Verified upgrade: 2026-09-23 (chart 0.1.17 / application 0.2.8)

        • Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260923T222508Z (directory mode 700, files mode 600). The scoped PostgreSQL dump was 32,663,617 bytes and the /app/data archive was 4,614,791 bytes. pg_restore --list returned 1,210 entries, the archive listing contained 10 entries, and all SHA-256 checks passed. The initial streamed validation timed out; validation using a controlled temporary file inside the PostgreSQL container passed.
        • GitOps commit: 646d50010cf39090567098bc819e74b06976a5ef; only manifests/sub2api-argocd.yaml changed, from chart 0.1.16 to 0.1.17. Mirror chart PR #8 merged as 87ae24c. The image digest is sha256:11b2dc8d9ea297c676581daf1322d71200eeed58ebfb5fecf2a991fa2741fb86.
        • ArgoCD parent and child reconciled successfully; Sub2API was Synced/Healthy on chart 0.1.17. Deployment was 1/1, the Pod was Ready with 0 restarts, and its image ID matched the target digest. The Service endpoint 10.42.0.226:8080 was ready; internal and public /health checks returned HTTP 200. Dedicated Redis was 1/1 with a ready endpoint; both PVCs remained Bound.
        • Migrations completed without errors. Preflight found 27 non-numeric legacy reasoning-effort keys, which migration 239 removed without conversion; one existing affiliate-ledger row received a NULL operation ID under migration 240, with no unique-index conflict.
        • Image pull took 10m37s and briefly caused ProgressDeadlineExceeded; the rollout recovered. Existing pricing-repository timeout warnings remained. No rollback was required. Deployment rollback is a new Git revert; migrations are forward-only, so database recovery requires an isolated restore and a deliberate configuration switch.

        Verified upgrade: 2026-09-28 (chart 0.1.18 / application 0.2.9)

        • Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260928T135349Z (directory mode 700, files mode 600). The scoped PostgreSQL dump was 37,222,935 bytes with 1,214 pg_restore --list entries; the /app/data archive was 4,631,639 bytes with 10 entries. SHA-256 checks passed, and the captured origin/main revision was 622ea9171286d9e218bd15a9569ce8c1b565d48b.
        • GitOps commit: b37fe3549cc8c65bba6d268c45ffdda97f86d9ca (chore(sub2api): upgrade to chart 0.1.18); only manifests/sub2api-argocd.yaml changed, from chart 0.1.17 to 0.1.18. Mirror chart PR #9 merged as a748960; the published chart 0.1.18 was anonymously pulled back with package SHA-256 196406aa97b89ff747cc0317ad71d2eaa3b22adbc0057c7d4aec2ec126ad35e4 matching Git. The image digest is sha256:996a0ea43500550f233a2f653de58dcd77f42d3e244805894fcccdd32b612147.
        • ArgoCD synced Sub2API to chart 0.1.18 at 2026-09-28T13:58Z (operation Succeeded) and reported Synced/Healthy by 14:00Z. Deployment was 1/1, Pod sub2api-8c7fbd445-bhhnp Ready with 0 restarts, and its image ID matched the target digest exactly. The Service endpoint 10.42.0.239:8080 was ready; internal /health returned {"status":"ok"} and public https://token.72602.space/health returned HTTP 200. Dedicated Redis was 1/1 and both PVCs remained Bound.
        • Application 0.2.9 introduced no new database migrations: the latest applied migration remains 240_affiliate_ledger_operation_id.sql (applied 2026-09-23) and schema_migrations holds 289 records. No migration or startup errors appeared in the logs; the rollout completed in about two minutes without ProgressDeadlineExceeded.
        • No rollback was required. Deployment rollback is a new Git revert; migrations are forward-only, so database recovery requires an isolated restore and a deliberate configuration switch.

        Verified upgrade: 2026-10-01 (chart 0.1.19 / application 0.2.11)

        • Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20261001T001943Z (outside the repository). Only the sub2api PostgreSQL database and application/sub2api-data PVC (/app/data) were backed up. sub2api.dump is 38,717,665 bytes (SHA-256 bd4b8663b645e24058a034342b8b1b1fa58f2475fbc5b84a94f79cc509c3ca83); sub2api-data.tgz is 4,638,794 bytes (SHA-256 f2917628f296013f8062891c18172d8116e2f488a5a9466deda463f19989a0d9). Both artifacts are non-empty and mode 600; pg_restore --list returned 1,214 entries and the tar listing returned 10 entries.
        • Published OCI chart 0.1.19 is anonymously pullable; package SHA-256 3bd25eb5b5cdf69c6980e688db9730db86d72f6129315af9eb085c94a8c9dc89 matched Git. GitOps targetRevision is 0.1.19, from commit b805c440d53daa4f90706b028a5457de7e0790ae.
        • ArgoCD reported Synced/Healthy. Deployment rollout was 1/1 ready and available; the Pod was Ready with 0 restarts, and the running image digest matched Linux amd64 image ghcr.io/wei-shaw/sub2api@sha256:9a71717a6f82efe663b613c730069ad427a2beee38bd16663346ae1c27a6cbc1.
        • Service ClusterIP is 10.43.88.246:8080, with endpoint 10.42.0.70:8080. Internal /health returned HTTP 200 with {"status":"ok"}. Dedicated Redis was 1/1 ready at 10.42.0.12:6379; its one historical restart is not a current health issue.
        • No rollback was required. Roll back deployment configuration with a new Git revert to chart 0.1.18; do not use kubectl rollout undo. Migrations are forward-only. Database rollback requires restoring into an isolated database and making a deliberate configuration switch.

        Verified upgrade: 2026-10-02 (chart 0.1.20 / application 0.2.13)

        • Fresh backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20261002T154620Z (outside the repository; directory mode 700, files mode 600). Only the sub2api database and PVC application/sub2api-data mounted at /app/data were backed up; exactly two artifacts were created. sub2api.dump is 39,995,006 bytes (SHA-256 b749ec86ec5aa6244780edeb29b1a7e82fd7cd87881b9d971a5cf6aaf408d2ef), with 1,214 pg_restore --list entries. sub2api-data.tgz is 4,644,880 bytes (SHA-256 c312329d676c8eab04f87e419ab00a68d6636460bcbe80c650d4028951396160), with 10 tar listing entries.
        • GitOps commit 18c5010062fbb9a1b22450ff3fb8b055f91fa2b1 changed only targetRevision from 0.1.19 to 0.1.20. Published chart 0.1.20 is application 0.2.13; package SHA-256 b681c230a96ad93ef24a17946fdcb867cad4dae4ac319b70258258bb01b19b6b and OCI manifest digest sha256:60d44de5e929d2c4f3fe0a1d6d3b02d5e8597acd7fdaf99e5a02cf08e27a8e34. An independent anonymous pull matched the Git archive.
        • ArgoCD operation Succeeded, Synced/Healthy; sync completed at 2026-10-02T15:57:26Z and Healthy was reported at 16:01:36Z. Deployment was 1/1 ready and available; Pod sub2api-c98d8b48b-dmzwg was Ready/Running with 0 restarts, and the running image digest matched Linux amd64 image ghcr.io/wei-shaw/sub2api@sha256:834172e90a5e51ae29afc9f93ec13235768eaad77497bc8c83274c141cd55759.
        • Service ClusterIP 10.43.88.246:8080 had endpoint 10.42.0.74:8080; internal /health returned HTTP 200 with {"status":"ok"}. Dedicated Redis StatefulSet was 1/1 with ready endpoint 10.42.0.12:6379; its Pod has one historical restart.
        • No rollback was required. Deployment rollback is a new Git revert to chart 0.1.19; do not use kubectl rollout undo. Database migrations are forward-only, so database rollback requires restoring into an isolated database and making a deliberate configuration switch.

        Verified upgrade: 2026-10-14 (chart 0.1.22 / application 0.2.16)

        • Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20261014T123853Z (outside the repository; directory mode 700). Exactly two files, both mode 600: sub2api.dump (52,013,200 bytes, SHA-256 7e3bf6693e21a9ea644cccda36f85f03f506bd1a4dac7bd589118551b7272d8f) and sub2api-data.tgz (4,674,990 bytes, SHA-256 1b68a4c9af54fe032c582e51eee5f039d8b9594a9592441e31e7d2893b164f75). The dump covered only database/user sub2api on database/postgresql-0; the archive covered only application/sub2api-data mounted at /app/data. pg_restore --list passed with 1,214 entries, the archive listing passed with 10 entries, and SHA-256 recomputation passed. Initial streamed restore listing timed out; the pod-local temporary-file fallback passed and the temporary file was removed.
        • Chart mirror PR #13 merged as 440034baaaac6efc56eba1ae8c906a475f5f1518. Package SHA-256 was 718ff8e75d4e803d351da82e2d02161a86ac85178b8e569ce6b98f241672dccb; an independent anonymous Helm pull matched. GHCR OCI manifest digest was sha256:4f3fdb5942b218717099821597b29324f98400256adc560e11cc0b7922372fc8. Chart CI, Mirror consistency CI, Publish Sub2API chart, and Pages workflows succeeded. GitOps commit ff542ee8a14d2615233281cf6886ddc5ab8a59e9 changed only targetRevision from 0.1.21 to 0.1.22.
        • Parent ops-docs was Synced/Healthy at that commit, operation Succeeded. Child sub2api target and observed chart were 0.1.22, Synced/Healthy, operation Succeeded. Deployment rollout finished with desired/updated/ready/available 1/1/1/1, unavailable 0; Pod sub2api-c8d4cd46d-5lb4z was Running/Ready with 0 restarts, and its live image ID matched Linux amd64 image ghcr.io/wei-shaw/sub2api@sha256:4a0d0196d3450c71da159137e3419fc144df0c038bbe3b9e87dbbccaef10cab7.
        • Ready/serving Service endpoint was 10.42.0.55:8080; internal /health returned HTTP 200 with {"status":"ok"}. Dedicated Redis StatefulSet and Pod were 1/1 ready. Redis had 2 historical restarts; last termination was 2026-10-07 with exit 255, with no current Redis errors.
        • Non-fatal observations: one startup timeout fetching the remote model-price hash, security allowlist warnings, and a historical readiness failure from the replaced Pod. No migration failure was observed; no specific migration is asserted. No rollback was required. Roll back the GitOps release with a new Git revert to chart 0.1.21; a true database rollback requires restoring the dump into an isolated database and making a deliberate configuration switch. Never use kubectl rollout undo as a database rollback.

        Restore PostgreSQL Safely

        Restore into a separate database first. Do not overwrite the live sub2api database during an upgrade rollback.

        1.create the restore database

        BACKUP_DIR=/home/aaron/Ops/backups/sub2api/<approved-backup-directory>
test -s "$BACKUP_DIR/sub2api.dump"

set +x
read -rsp 'PostgreSQL admin password: ' POSTGRES_ADMIN_PASSWORD; printf '\n'
test -n "$POSTGRES_ADMIN_PASSWORD"

printf '%s\n' "$POSTGRES_ADMIN_PASSWORD" | \
  kubectl -n database exec -i postgresql-0 -- \
  sh -c 'IFS= read -r PGPASSWORD; export PGPASSWORD; exec createdb -U postgres -O sub2api sub2api_restore'

unset POSTGRES_ADMIN_PASSWORD

        2.restore and verify with the application database user

        set +x
PG_PASSWORD="$(kubectl -n application get secret \
  sub2api-external-postgresql \
  -o jsonpath='{.data.postgres-password}' | base64 -d)"
test -n "$PG_PASSWORD"

{ printf '%s\n' "$PG_PASSWORD"; cat "$BACKUP_DIR/sub2api.dump"; } | \
  kubectl -n database exec -i postgresql-0 -- \
  sh -c 'IFS= read -r PGPASSWORD; export PGPASSWORD; exec pg_restore -U sub2api -d sub2api_restore --exit-on-error --no-owner --no-privileges'

printf '%s\n' "$PG_PASSWORD" | \
  kubectl -n database exec -i postgresql-0 -- \
  sh -c 'IFS= read -r PGPASSWORD; export PGPASSWORD; exec psql -U sub2api -d sub2api_restore -c "\\dt"'

unset PG_PASSWORD

        3.switch only through a reviewed Git recovery change

        Change externalPostgresql.database to sub2api_restore in manifests/sub2api-argocd.yaml, commit and push the reviewed recovery change, then reconcile ops-docs and sub2api. Restore sub2api-data only during a planned maintenance window with the workload quiesced; never extract the archive over a running Pod.

        argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600

curl -fsS https://token.72602.space/health
curl -fsS https://token.72602.space/api/v1/settings/public
kubectl -n application logs deployment/sub2api --since=10m

        Do not use kubectl rollout undo, delete PVCs, or delete Secrets as restore or rollback steps.

        App-Level S3 Backup Check

        The live S3 endpoint is https://api.minio.72602.space, and Sub2API uses the sub2api bucket. The MinIO API Ingress source keeps nginx.ingress.kubernetes.io/proxy-body-size: "0" scoped to that API host so backup uploads are not rejected by the default 1 MiB ingress limit. Verify a scheduled or manual backup from logs without printing credentials, tokens, object names, or backup content:

        kubectl -n application logs deployment/sub2api --since=30m
kubectl -n basic-components logs deployment/ingress-nginx-controller --since=30m
kubectl -n storage get ingress minio-api

        Install (ArgoCD)

        🚀Installation

        Environment
        Install By

        Preliminary

        1. Kubernetes is installed; if not, check 🔗link


        2. Helm is installed; if not, check 🔗link


        3. ArgoCD is installed; if not, check 🔗link


        1. PostgreSQL is available in namespace database, DNS resolves token.72602.space, and ingress-nginx with the lets-encrypt ClusterIssuer is ready.

        1.prepare runtime Secrets

        Details
        kubectl get namespace application >/dev/null 2>&1 || \
  kubectl create namespace application

set +x
read -rsp 'Sub2API admin password: ' ADMIN_PASSWORD; printf '\n'
read -rsp 'Sub2API PostgreSQL password: ' POSTGRES_PASSWORD; printf '\n'
read -rsp 'Sub2API Redis password: ' REDIS_PASSWORD; printf '\n'
JWT_SECRET="$(openssl rand -hex 32)"
TOTP_KEY="$(openssl rand -hex 32)"

test -n "$ADMIN_PASSWORD"
test -n "$POSTGRES_PASSWORD"
test -n "$REDIS_PASSWORD"

kubectl -n application create secret generic sub2api-auth \
  --from-literal=admin-password="$ADMIN_PASSWORD" \
  --from-literal=jwt-secret="$JWT_SECRET" \
  --from-literal=totp-encryption-key="$TOTP_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n application create secret generic sub2api-external-postgresql \
  --from-literal=postgres-password="$POSTGRES_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n application create secret generic sub2api-redis \
  --from-literal=redis-password="$REDIS_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

unset ADMIN_PASSWORD POSTGRES_PASSWORD REDIS_PASSWORD JWT_SECRET TOTP_KEY
kubectl -n application get secret \
  sub2api-auth sub2api-external-postgresql sub2api-redis

        Secret values must stay outside Git, terminal output, and this handbook.

        2.verify the GitOps source

        The parent argocd/ops-docs Application reads https://github.com/AaronYang0628/docs.git at path manifests. The canonical child declaration is manifests/sub2api-argocd.yaml; it configures the OCI chart source and must not be applied as an independent deployment route.

        Details
        git -C /home/aaron/Ops/docs fetch origin main
git -C /home/aaron/Ops/docs \
  show origin/main:manifests/sub2api-argocd.yaml >/dev/null

kubectl -n argocd get application ops-docs \
  -o jsonpath='{.spec.source.repoURL}{"\n"}{.spec.source.path}{"\n"}'

        Expected source values are https://github.com/AaronYang0628/docs.git and manifests.

        3.sync parent and child Applications

        Details
        argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300

argocd app get sub2api --hard-refresh
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600

        4.verify release, storage, ingress, and API path

        Details
        kubectl -n argocd get application sub2api \
  -o jsonpath='{.spec.source.repoURL}{"\n"}{.spec.source.chart}{" "}{.spec.source.targetRevision}{"\n"}'

kubectl -n application get deployment sub2api \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"="}{.image}{"\n"}{end}'

kubectl -n application rollout status deployment/sub2api --timeout=600s
kubectl -n application get pods,svc,ingress,pvc
kubectl -n application get certificate,certificaterequest,order,challenge
kubectl -n application get endpointslice \
  -l kubernetes.io/service-name=sub2api

curl -fsS https://token.72602.space/health
curl -fsS https://token.72602.space/api/v1/settings/public

set +x
read -rsp 'Sub2API API token: ' SUB2API_API_TOKEN; printf '\n'
curl -fsS \
  -H "Authorization: Bearer ${SUB2API_API_TOKEN}" \
  https://token.72602.space/v1/models

read -rp 'Model ID for smoke generation: ' MODEL_ID
curl -fsS https://token.72602.space/v1/chat/completions \
  -H "Authorization: Bearer ${SUB2API_API_TOKEN}" \
  --json "{\"model\":\"${MODEL_ID}\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with OK.\"}],\"max_tokens\":8}"
unset SUB2API_API_TOKEN MODEL_ID

        Expected release values are chart 0.1.22, application 0.2.16, and image ghcr.io/wei-shaw/sub2api@sha256:4a0d0196d3450c71da159137e3419fc144df0c038bbe3b9e87dbbccaef10cab7 (Linux amd64). The TLS certificate is Ready; the sub2api-data PVC is 10Gi and the Redis PVC is 8Gi, both local-path RWO.

        Ops

        Web Page

        sub2api web page (https://token.72602.space)

        Current Release

        argocd app get ops-docs --hard-refresh
argocd app get sub2api --hard-refresh

kubectl -n argocd get application sub2api \
  -o jsonpath='{.spec.source.repoURL}{"\nchart="}{.spec.source.chart}{" "}{.spec.source.targetRevision}{"\nimage.tag="}{.spec.source.helm.parameters[?(@.name=="image.tag")].value}{"\n"}'

kubectl -n application get deployment sub2api \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"="}{.image}{"\n"}{end}'

kubectl -n application get pods,svc,ingress,pvc
kubectl -n application get certificate,certificaterequest,order,challenge

        The expected values are OCI chart 0.1.22, application 0.2.16, image ghcr.io/wei-shaw/sub2api@sha256:4a0d0196d3450c71da159137e3419fc144df0c038bbe3b9e87dbbccaef10cab7 (Linux amd64), namespace application, and host token.72602.space. The application PVC is 10Gi; the Redis PVC is 8Gi with AOF enabled. Both use local-path and RWO.

        Sync From Git

        argocd/ops-docs owns manifests/sub2api-argocd.yaml from https://github.com/AaronYang0628/docs.git. Reconcile the Git parent before the OCI Helm child:

        git -C /home/aaron/Ops/docs fetch origin main
argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300

argocd app get sub2api --hard-refresh
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600

        Do not apply manifests/sub2api-argocd.yaml directly as a second ownership path.

        Rolling Upgrade

        1.back up PostgreSQL before changing chart or image values

        Use the Backup & Restore runbook. Store the scoped backup outside the repository and verify both artifacts before continuing. For the upgrade completed on 2026-10-10, the backup was /home/aaron/Ops/backups/sub2api/upgrade-20261010T092354Z; it contains only:

        ArtifactSizeSHA-256
        sub2api.dump45,787,015 bytes0e0195256cccbbd985df5079f1d9e0998cd12cfe5e101ddcf471f01d4b767b5f
        sub2api-data.tgz4,665,673 bytes1068b2efe24639bfba000daa90e365c80fd7b874ffc4dea847a1d8e9626acd1e

        pg_restore --list passed with 1,214 entries; listing the PVC archive passed with 10 entries. Verify the saved artifacts with:

        sha256sum \
  /home/aaron/Ops/backups/sub2api/upgrade-20261010T092354Z/sub2api.dump \
  /home/aaron/Ops/backups/sub2api/upgrade-20261010T092354Z/sub2api-data.tgz
pg_restore --list \
  /home/aaron/Ops/backups/sub2api/upgrade-20261010T092354Z/sub2api.dump
tar -tzf \
  /home/aaron/Ops/backups/sub2api/upgrade-20261010T092354Z/sub2api-data.tgz

        Do not infer or claim a particular database migration ran.

        For the verified 2026-10-14 upgrade, the backup directory was /home/aaron/Ops/backups/sub2api/upgrade-20261014T123853Z (outside the repository, directory mode 700). It contained exactly the database dump and /app/data archive, both mode 600; pg_restore --list passed with 1,214 entries, archive listing passed with 10 entries, and SHA-256 recomputation passed. See Backup & Restore for artifact sizes and hashes.

        2.update the Git source

        Edit only the reviewed targetRevision, image.tag, or required values in manifests/sub2api-argocd.yaml, then inspect and publish the change. For the 2026-10-10 upgrade, this manifest changed only at targetRevision. The chart mirror package SHA-256 was 2c82a680a66988ab034e7180d9b2871d49e69f9c4217ffc3173c3790230453d6; an anonymous Helm pull matched the package and index. Chart mirror PR #12 was merged in commit 9f8a943e93a1fe4bf92b8ae381537538ea18a6fc. The GitOps change was committed and pushed as ee451e9025fa0ad6cd26e8a849db530693bbc6c8.

        For the 2026-10-14 upgrade, chart mirror PR #13 merged as 440034baaaac6efc56eba1ae8c906a475f5f1518. The chart package SHA-256 was 718ff8e75d4e803d351da82e2d02161a86ac85178b8e569ce6b98f241672dccb; an independent anonymous Helm pull matched. Its GHCR OCI manifest digest was sha256:4f3fdb5942b218717099821597b29324f98400256adc560e11cc0b7922372fc8. Chart CI, Mirror consistency CI, Publish Sub2API chart, and Pages workflows succeeded. GitOps commit ff542ee8a14d2615233281cf6886ddc5ab8a59e9 changed only targetRevision, from 0.1.21 to 0.1.22.

        git -C /home/aaron/Ops/docs diff --check -- manifests/sub2api-argocd.yaml
git -C /home/aaron/Ops/docs diff -- manifests/sub2api-argocd.yaml
git -C /home/aaron/Ops/docs add manifests/sub2api-argocd.yaml
git -C /home/aaron/Ops/docs commit -m "chore: upgrade sub2api"
git -C /home/aaron/Ops/docs push origin HEAD:main

        3.reconcile parent and child Applications

        argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300

argocd app get sub2api --hard-refresh
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600
kubectl -n application rollout status deployment/sub2api --timeout=600s
kubectl -n application get endpointslice \
  -l kubernetes.io/service-name=sub2api -o wide
kubectl -n application get statefulset,pod,service,endpoints sub2api-redis
kubectl -n application exec deployment/sub2api -- wget -qO- http://127.0.0.1:8080/health

        For that upgrade, ArgoCD adopted the Git commit at 2026-10-10T09:34:53Z, observed chart revision 0.1.21, and reported Synced/Healthy; health became Healthy at 09:37:54Z. The rollout completed 1/1 with zero Pod restarts, the Service endpoint was ready on port 8080, and the internal /health response was {"status":"ok"}. The dedicated Redis StatefulSet, Pod, and endpoint were all ready 1/1.

        For the 2026-10-14 upgrade, parent ops-docs was Synced/Healthy at commit ff542ee8a14d2615233281cf6886ddc5ab8a59e9, operation Succeeded. Child sub2api target and observed chart were 0.1.22, Synced/Healthy, operation Succeeded. Deployment desired/updated/ready/available counts were 1/1/1/1, unavailable 0; Pod sub2api-c8d4cd46d-5lb4z was Running/Ready with 0 restarts and its live image ID matched the expected digest. The ready/ serving Service endpoint was 10.42.0.55:8080; internal /health returned HTTP 200 with {"status":"ok"}. Dedicated Redis StatefulSet and Pod were 1/1 ready. Redis had 2 historical restarts, the last terminating on 2026-10-07 with exit 255; there were no current Redis errors.

        Non-fatal observations were one startup timeout fetching the remote model-price hash, security allowlist warnings, and a historical readiness failure from the replaced Pod. No migration failure was observed; no specific migration is asserted.

        The Deployment explicitly uses maxUnavailable: 0 and maxSurge: 1. Kubernetes adds the new Ready Pod to the EndpointSlice before terminating the old Pod, and the Service selects only Ready endpoints. This protects new requests during rollout, but it does not guarantee completion of requests that are already attached to the terminating Pod. Long generations still require the application to handle graceful termination and draining correctly.

        4.verify the public and authenticated model path

        curl -fsS https://token.72602.space/health
curl -fsS https://token.72602.space/api/v1/settings/public

set +x
read -rsp 'Sub2API API token: ' SUB2API_API_TOKEN; printf '\n'
curl -fsS \
  -H "Authorization: Bearer ${SUB2API_API_TOKEN}" \
  https://token.72602.space/v1/models

read -rp 'Model ID for smoke generation: ' MODEL_ID
curl -fsS https://token.72602.space/v1/chat/completions \
  -H "Authorization: Bearer ${SUB2API_API_TOKEN}" \
  --json "{\"model\":\"${MODEL_ID}\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with OK.\"}],\"max_tokens\":8}"
unset SUB2API_API_TOKEN MODEL_ID

        Admin MCP

        The Ops Agent Pod runs a local MCP server sub2api-admin against the cluster-local endpoint http://sub2api.application.svc.cluster.local:8080. The Deployment injects the API key from the dedicated runtime Secret application/sub2api-mcp key admin-api-key. Provision the Secret through manifests/ops-agent/create-sub2api-mcp-secret.sh; the value never enters Git or opencode.json. The MCP only accepts the configured cluster-local host, every authenticated GET runs through a reviewed route allowlist, sensitive fields are recursively redacted, and no generic authenticated write tool is exposed.

        Enter the key only into the helper’s silent interactive prompt. Never place the key in shell arguments, chat, Git, OpenCode config, logs, or command output.

        Tools

        Nine tools are registered:

        ToolPurpose
        sub2api-admin_describeCapabilities and safety constraints
        sub2api-admin_healthAnonymous check of /health through the configured internal cluster-local URL (SUB2API_BASE_URL), not the public host
        sub2api-admin_readOne allowlisted authenticated GET route
        sub2api-admin_find_userResolve exactly one user by normalized email
        sub2api-admin_preview_rechargeNon-mutating preview of an additive credit
        sub2api-admin_rechargeAdditive credit with backend idempotency
        sub2api-admin_update_user_limitsPer-user RPM and/or concurrency
        sub2api-admin_set_account_schedulableEnable or disable scheduling for an entire upstream account
        sub2api-admin_model_control_optionsExplain the available scheduling boundaries and rate-limit scopes

        The global OpenCode permission set denies every sub2api-admin_* tool. The 72602-k3s-maintainer agent allows the six read-only tools directly and asks before recharge, update_user_limits, or set_account_schedulable. Use the maintainer for any authenticated mutation.

        Recharge protocol

        Recharge is additive USD-style internal credit only; set and subtract operations are not exposed. Before every charge:

        1. Collect an exact email and a positive amount.
        2. Call sub2api-admin_preview_recharge. The server fuzzy-searches but accepts only one exact normalized email match.
        3. Read back the matched email, the immutable user id, status, the current balance, the amount, and the expected balance; obtain explicit confirmation for that exact tuple.
        4. Generate one stable Idempotency-Key of 16-128 URL-safe characters. Keep and reuse that key when the result is uncertain; never generate a fresh key for a retry.
        5. Call sub2api-admin_recharge with operation: add, the exact email and id, the audit note, and the idempotency key.
        6. Re-read the user and balance history; record the exact verified or accepted_unverified outcome. Never rewrite an unverified accepted response as success.

        No account is recharged as part of installation. Do not recharge inactive users unless the operator explicitly confirms the inactive status.

        Limits and model availability

        Sub2API v0.1.176 has no configurable per-model RPM. Operator scope:

        • User RPM and concurrency via sub2api-admin_update_user_limits.
        • Group RPM is set through the admin UI; this MCP does not expose it.
        • rate_multiplier changes billing, not request rate.

        For temporary model unavailability, call sub2api-admin_model_control_options first and identify the model’s actual scheduling boundary:

        • Dedicated account: sub2api-admin_set_account_schedulable toggles every model served by that account; roll back with the inverse value.
        • Restricted channel: remove the exact model from the channel pricing/model list while restrict_models=true; preserve the previous payload for rollback. This MCP does not mutate channels, use the admin UI.
        • Composite model: disable the exact composite route; preserve prior state. Use the admin UI.
        • models_list_config only hides discovery. It does not block direct requests and must never be reported as a disable.

        Scheduling changes affect new requests only. They do not cancel requests or streams already in progress. If the model spans multiple accounts, channels, or routes, enumerate and confirm every affected target before mutation.

        Secret rotation

        Sub2API supports only one Admin API Key. Generating or regenerating the key in the admin UI immediately invalidates the previous key, so the MCP will briefly fail to authenticate until the Ops Agent workload restarts with the new value. Plan an expected maintenance window of one to two restart cycles. Rotate in this exact order without intermediate pauses:

        1. Generate the new key in the Sub2API admin UI and immediately enter it into the silent prompt of manifests/ops-agent/create-sub2api-mcp-secret.sh. Confirm the helper reports application/sub2api-mcp keys=admin-api-key.
        2. Restart the managed workload so the new env var is injected:
          kubectl -n application rollout restart deployment/ops-agent
kubectl -n application rollout status deployment/ops-agent --timeout=300s
        3. Verify global/health reports healthy: true, /mcp reports exactly {"sub2api-admin":{"status":"connected"}}, and the merged OpenCode config carries the cluster-local SUB2API_BASE_URL without the new key.
        4. Through 72602-k3s-maintainer, run sub2api-admin_read against /api/v1/admin/system/version; the live application must respond 0.1.176.

        The upstream “generate key” action is itself the invalidation step; no follow-up disable of the previous key is needed or supported.

        Tests: node --test .opencode/mcp/sub2api-admin/server.test.mjs passes three tests covering the allowlist, path-traversal rejection, and the recharge idempotency path.

        GitOps Rollback

        Restore the previous reviewed chart and image values with a new Git revert commit, then let ArgoCD converge. To return this release to chart 0.1.21, revert its GitOps change and sync the parent and child Applications:

        git -C /home/aaron/Ops/docs log --oneline -- manifests/sub2api-argocd.yaml
git -C /home/aaron/Ops/docs revert ff542ee8a14d2615233281cf6886ddc5ab8a59e9
git -C /home/aaron/Ops/docs push origin HEAD:main

argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600

        An image/chart rollback does not reverse a database change. If database rollback is required, restore the pre-upgrade dump into an isolated database and switch configuration deliberately through a reviewed Git change. Do not use kubectl rollout undo; do not delete PVCs or Secrets as a rollback step.

        Troubleshooting

        kubectl -n application logs deployment/sub2api --since=10m
kubectl -n application get events --sort-by=.lastTimestamp
kubectl -n application get endpointslice \
  -l kubernetes.io/service-name=sub2api -o yaml
argocd app get sub2api --hard-refresh

        🛎️FAQ

        Admin proxy test reports the wrong egress country

        Runtime observations below were verified on 2026-10-04. Researched upstream source builds the admin Test proxy URL from the stored proxy record, rather than the Deployment’s HTTP_PROXY, and does not translate port 17890 to 7890; deployed source parity was not established. Its built-in probes are http://ip-api.com/json/?lang=zh-CN and http://api64.ipify.org?format=json.

        The admin proxy record 72602-clash (ID 1) was active as HTTP 192.168.0.25:17890.

        If ip-api.com reports China while the Pod is reaching the proxy bridge, check Clash rule precedence. The subscription’s DOMAIN-SUFFIX,ip-api.com,DIRECT rule sent that probe through CN host ISP egress. Put this prefix rule in /home/aaron/clashctl/resources/mixin.yaml:

        rules:
  prefix:
    - DOMAIN-SUFFIX,ip-api.com,Ghelper

        The mixin prefix takes priority over the subscribed DIRECT rule and survives subscription refresh. Keep the existing api64.ipify.org DIRECT override removed. The verified controller selection was Ghelper directly pinned to 🇸🇬 新加坡07; profile.store-selected: true preserves the selection across the owning workflow restart.

        After applying through the owning mixin merge/restart workflow, verify both exact probe URLs from the Sub2API Pod through http://192.168.0.25:17890. The verified responses were HTTP 200: ip-api reported status=success, countryCode=SG; api64 returned valid IP JSON. https://ifconfig.co/country-iso also returned HTTP 200 with SG. Sub2API was Running/Ready with zero restarts and health ok; Argo CD was Synced/Healthy on chart 0.1.20. The proxy UI at https://clash.72602.space/ui/ returned HTTP 200. These are Pod-side equivalent URL checks, not an admin Test proxy action: no approved tool for its dedicated POST endpoint was available.

        To roll back the mixin change, restore /home/aaron/clashctl/resources/mixin.yaml.before-ip-api-direct.20261004-074639 and run the owning merge/restart workflow. If required, return the controller selection to 🌐 全球智能 through its owning workflow.

        Settings API reports column settings.id does not exist

        If /api/v1/settings/public or the admin settings API returns HTTP 500 and the logs contain pq: column settings.id does not exist, Sub2API is connected to a shared or legacy database with an incompatible settings table.

        Confirm that the live Application and Secret reference the dedicated sub2api database and user without printing the password:

        kubectl -n argocd get application sub2api \
  -o jsonpath='{.spec.source.helm.values}'
kubectl -n application get secret sub2api-external-postgresql
kubectl -n application logs deployment/sub2api --since=10m

        Keep externalPostgresql.username and externalPostgresql.database set to sub2api in manifests/sub2api-argocd.yaml. Create or recover the dedicated database first, then update the Secret through the approved secret-management process and reconcile through ArgoCD. Do not point Sub2API at the n8n database or patch the Deployment directly.

        Redis reports WRONGPASS

        WRONGPASS invalid username-password pair means the password used by the Sub2API Pod no longer matches the Redis Secret. The chart must continue to pin the stable Secret and key:

        redis:
  auth:
    existingSecret: sub2api-redis
    existingSecretPasswordKey: redis-password

        Verify references and workload state without reading the Secret value:

        kubectl -n application get secret sub2api-redis
kubectl -n application get deployment sub2api \
  -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}{" <- "}{.valueFrom.secretKeyRef.name}{"/"}{.valueFrom.secretKeyRef.key}{"\n"}{end}'
kubectl -n application logs deployment/sub2api --since=10m

        If rotation is required, update Redis and application/sub2api-redis as one planned operation, then reconcile the Git-owned Application. Do not generate a new password during a routine restart.