Install (ArgoCD)

🚀Installation

git -C /home/aaron/Ops/docs fetch origin main
git -C /home/aaron/Ops/docs \
  show origin/main:manifests/ollama-moderation-gateway-argocd.yaml >/dev/null

argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --timeout 300

kubectl wait --for=jsonpath='{.status.phase}'=Active \
  namespace/moderation --timeout=120s

test -s /home/aaron/Ops/docs/.moderation.env
test "$(stat -c %a /home/aaron/Ops/docs/.moderation.env)" = 600

kubectl -n moderation create secret generic ollama-moderation-gateway \
  --from-env-file=/home/aaron/Ops/docs/.moderation.env \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n moderation get secret ollama-moderation-gateway --output=name

# confirm parent application source
kubectl -n argocd get application ops-docs \
  -o jsonpath='{.spec.source.repoURL}{"\n"}{.spec.source.path}'

# refresh and sync the child (ollama-moderation-gateway)
argocd app get ollama-moderation-gateway --hard-refresh
argocd app sync ollama-moderation-gateway
argocd app wait ollama-moderation-gateway --sync --health --timeout 600

# chart source details
kubectl -n argocd get application ollama-moderation-gateway \
  -o jsonpath='{.spec.source.repoURL}{"\n"}{.spec.source.path}{"\n"}{.spec.source.targetRevision}{"\n"}'

# image and release verification
kubectl -n moderation get deployment ollama-moderation-gateway \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"="}{.image}{"\n"}{end}'

# rollout status
kubectl -n moderation rollout status deployment/ollama-moderation-gateway --timeout=600s
kubectl -n moderation get pods,svc,ingress
kubectl -n moderation get certificate,certificaterequest,order,challenge
kubectl -n moderation get endpointslice -l kubernetes.io/service-name=ollama-moderation-gateway

# health checks
curl -fsS https://moderation.llm.72602.space/health
curl -fsS https://moderation.llm.72602.space/readyz

# authenticated moderation request; do not echo the key or response body
set +x
read -rsp 'Gateway API key: ' GATEWAY_KEY; printf '\n'
curl -fsS -X POST https://moderation.llm.72602.space/v1/moderations \
  -H "Authorization: Bearer ${GATEWAY_KEY}" \
  -H "Content-Type: application/json" \
  --json '{"model":"moderation-fast","input":"hello"}' \
  | jq -e '.results | length == 1' >/dev/null
unset GATEWAY_KEY

📦Rollback Guidance

If a deployment must be reverted, create a reviewed Git revert for the deployment source commit, push it, and sync the parent application. Runtime secrets and TLS certificates are external resources and are not modified by the rollback.

# create a reviewed revert in the docs repository
cd /home/aaron/Ops/docs
git fetch origin main
git revert --no-edit 39c48b3
git push origin main
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300

After rollback, re-run the verification steps above to ensure the service is healthy.

Backup & Restore

Scope

• Database: PostgreSQL (database/postgresql-0), DB/user sub2api
• Git source: manifests/sub2api-argocd.yaml, owned by argocd/ops-docs
• Current release: OCI chart 0.1.15, application 0.2.5
• Application image: ghcr.io/wei-shaw/sub2api@sha256:4c5dffab6e5ba4d3bd5382f19aad9654847b4e23de1a3d48e190146a3e6eb977
• Runtime data: application/sub2api-data, 10Gi, local-path, RWO
• Redis data: 8Gi, local-path, RWO; AOF is enabled
• Runtime Secrets: application/sub2api-auth, application/sub2api-external-postgresql, and application/sub2api-redis

Sub2API executes PostgreSQL migrations automatically on startup. Migrations are forward-only, so every chart or image upgrade requires a verified pg_dump before the Git version change.

Pre-Upgrade Backup

1.create a protected operation directory

BACKUP_ROOT=/home/aaron/Ops/backups/sub2api
BACKUP_DIR="${BACKUP_ROOT}/upgrade-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$BACKUP_DIR"
chmod 700 "$BACKUP_DIR"
printf '%s\n' "$BACKUP_DIR"

Use the generated UTC directory for the entire operation. Do not hard-code a previous operation timestamp into future commands.

2.dump PostgreSQL without printing its password

set +x
PG_PASSWORD="$(kubectl -n application get secret \
  sub2api-external-postgresql \
  -o jsonpath='{.data.postgres-password}' | base64 -d)"
test -n "$PG_PASSWORD"

printf '%s\n' "$PG_PASSWORD" | \
  kubectl -n database exec -i postgresql-0 -- \
  sh -c 'IFS= read -r PGPASSWORD; export PGPASSWORD; exec pg_dump -U sub2api -d sub2api -Fc' \
  > "$BACKUP_DIR/sub2api.dump"

unset PG_PASSWORD

The password is passed on stdin and is not written into the backup directory.

3.capture application data and non-secret metadata

kubectl -n application exec deployment/sub2api -- \
  tar -C /app/data -czf - . > "$BACKUP_DIR/sub2api-data.tgz"

git -C /home/aaron/Ops/docs fetch origin main
git -C /home/aaron/Ops/docs rev-parse origin/main \
  > "$BACKUP_DIR/git-revision.txt"
git -C /home/aaron/Ops/docs show origin/main:manifests/sub2api-argocd.yaml \
  > "$BACKUP_DIR/sub2api-argocd.yaml"

kubectl -n application get pvc \
  -l app.kubernetes.io/instance=sub2api -o yaml \
  > "$BACKUP_DIR/pvc-metadata.yaml"

Do not export Kubernetes Secret objects into this directory. Back up Secret values only through the approved secret-management process. Redis AOF supports restart recovery on its PVC, but it is not a substitute for the PostgreSQL dump. Do not copy live AOF files as if they were a consistent database backup.

4.verify artifacts before upgrading

test -s "$BACKUP_DIR/sub2api.dump"
test -s "$BACKUP_DIR/sub2api-data.tgz"

kubectl -n database exec -i postgresql-0 -- pg_restore --list \
  < "$BACKUP_DIR/sub2api.dump" \
  > "$BACKUP_DIR/sub2api.dump.list"
test -s "$BACKUP_DIR/sub2api.dump.list"

sha256sum \
  "$BACKUP_DIR/sub2api.dump" \
  "$BACKUP_DIR/sub2api-data.tgz" \
  "$BACKUP_DIR/sub2api-argocd.yaml" \
  "$BACKUP_DIR/pvc-metadata.yaml" \
  "$BACKUP_DIR/git-revision.txt" \
  > "$BACKUP_DIR/SHA256SUMS"
sha256sum -c "$BACKUP_DIR/SHA256SUMS"

Verified backup: 2026-08-14

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260814T063611Z (directory mode 700, files mode 600). Only the shared PostgreSQL sub2api database was dumped; n8n and other databases were not touched.
• The host had no pg_restore; PostgreSQL Pod database/postgresql-0 had pg_restore 18.3 and GNU tar 1.34. The dump was copied temporarily with kubectl cp, checked with Pod-local pg_restore --list, and the temporary Pod file was removed. The previous kubectl exec -i streaming validation was not reused.
• Non-empty artifacts: sub2api.dump (19,640,824 bytes), sub2api-data.tgz (10,616,251 bytes), and sub2api.dump.list (77,853 bytes). Git revision, manifest, and PVC metadata were also captured.
• sha256sum -c SHA256SUMS and pg_restore --list both succeeded.
• The captured origin/main revision was ce838b424f10316fc604a4a21517c90c7b6b97ae. No failure occurred; no upgrade, Argo CD sync, or GitOps/cluster change was performed.

Verified upgrade: 2026-09-05 (application 0.2.0)

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260905T000035Z (upgrade commit 5230546610c437a02ee45f5b80cd804779c9170c). The scoped sub2api.dump (47,520,105 bytes) and sub2api-data.tgz (10,192,905 bytes) were non-empty; pg_restore --list and SHA-256 checks passed.
• schema_migrations advanced from 268 to 277 records; the latest migration is 233_group_free_openai_fast.sql.
• ArgoCD reported Synced/Healthy; Deployment was 1/1 on the pinned ghcr.io/wei-shaw/sub2api@sha256:271bb3b34661803681cabf54e99811ab8e248b0dd4c88b09ea1226e22dea5751 image.
• Existing application and Redis PVCs remained Bound, Service endpoints were Ready, and public health/settings checks succeeded.
• No rollback was required.

Verified upgrade: 2026-09-05 (application 0.2.1)

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260905T115349Z-178860 (upgrade commit 21d02345de47d75454841e464756f6cf9349cc3c). The scoped sub2api.dump and sub2api-data.tgz were non-empty; pg_restore --list and SHA-256 checks passed.
• ArgoCD reported Synced/Healthy at 2026-09-05T13:03:41Z; Deployment was 1/1, Pod ready with 0 restarts, internal /health returned {"status":"ok"}, Service endpoint ready, and Redis StatefulSet 1/1.
• Existing application and Redis PVCs remained Bound; health and settings checks succeeded.
• No rollback was required.

Verified upgrade: 2026-09-10

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260910T061248Z; PostgreSQL dump 46,520,009 bytes, /app/data archive 10,214,969 bytes; backup artifacts and pg_restore --list, archive listing, SHA-256 all passed; GitOps commit 7bd44adb5ed1b61976b507e1f32bc990738782b2; sub2api ArgoCD target/observed 0.1.14 Synced/Healthy operation Succeeded; runtime verification: Deployment 1/1, Pod ready 0 restarts, chart sub2api-0.1.14/app 0.2.4, actual image digest matches, Service endpoint ready/serving, internal /health HTTP 200, Redis StatefulSet 1/1.
• No rollback required.

Verified upgrade: 2026-09-16

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260916T053309Z (directory mode 700, files mode 600). PostgreSQL dump 27,593,406 bytes, /app/data archive 4,579,406 bytes; backup artifacts and pg_restore --list, archive listing, SHA-256 all passed; GitOps commit e2c0e4576276f6a299bbd27e80f40af883f8e877 (chore(sub2api): upgrade to chart 0.1.15); sub2api ArgoCD target/observed 0.1.15 Synced/Healthy operation Succeeded; runtime verification: Deployment 1/1, Pod sub2api-7c9c4b95bf-r6cpg Running/Ready, restarts 0, chart sub2api-0.1.15/app 0.2.5, actual image digest matches sha256:4c5dffab6e5ba4d3bd5382f19aad9654847b4e23de1a3d48e190146a3e6eb977, Service endpoint ready at 10.42.0.198:8080, internal /health HTTP 200 body {"status":"ok"}, Redis StatefulSet ready at 10.42.0.12:6379 with 1 historical restart.
• No rollback required.

Restore PostgreSQL Safely

Restore into a separate database first. Do not overwrite the live sub2api database during an upgrade rollback.

1.create the restore database

BACKUP_DIR=/home/aaron/Ops/backups/sub2api/<approved-backup-directory>
test -s "$BACKUP_DIR/sub2api.dump"

set +x
read -rsp 'PostgreSQL admin password: ' POSTGRES_ADMIN_PASSWORD; printf '\n'
test -n "$POSTGRES_ADMIN_PASSWORD"

printf '%s\n' "$POSTGRES_ADMIN_PASSWORD" | \
  kubectl -n database exec -i postgresql-0 -- \
  sh -c 'IFS= read -r PGPASSWORD; export PGPASSWORD; exec createdb -U postgres -O sub2api sub2api_restore'

unset POSTGRES_ADMIN_PASSWORD

2.restore and verify with the application database user

set +x
PG_PASSWORD="$(kubectl -n application get secret \
  sub2api-external-postgresql \
  -o jsonpath='{.data.postgres-password}' | base64 -d)"
test -n "$PG_PASSWORD"

{ printf '%s\n' "$PG_PASSWORD"; cat "$BACKUP_DIR/sub2api.dump"; } | \
  kubectl -n database exec -i postgresql-0 -- \
  sh -c 'IFS= read -r PGPASSWORD; export PGPASSWORD; exec pg_restore -U sub2api -d sub2api_restore --exit-on-error --no-owner --no-privileges'

printf '%s\n' "$PG_PASSWORD" | \
  kubectl -n database exec -i postgresql-0 -- \
  sh -c 'IFS= read -r PGPASSWORD; export PGPASSWORD; exec psql -U sub2api -d sub2api_restore -c "\\dt"'

unset PG_PASSWORD

3.switch only through a reviewed Git recovery change

Change externalPostgresql.database to sub2api_restore in manifests/sub2api-argocd.yaml, commit and push the reviewed recovery change, then reconcile ops-docs and sub2api. Restore sub2api-data only during a planned maintenance window with the workload quiesced; never extract the archive over a running Pod.

argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600

curl -fsS https://token.72602.space/health
curl -fsS https://token.72602.space/api/v1/settings/public
kubectl -n application logs deployment/sub2api --since=10m

Do not use kubectl rollout undo, delete PVCs, or delete Secrets as restore or rollback steps.

App-Level S3 Backup Check

The live S3 endpoint is https://api.minio.72602.space, and Sub2API uses the sub2api bucket. The MinIO API Ingress source keeps nginx.ingress.kubernetes.io/proxy-body-size: "0" scoped to that API host so backup uploads are not rejected by the default 1 MiB ingress limit. Verify a scheduled or manual backup from logs without printing credentials, tokens, object names, or backup content:

kubectl -n application logs deployment/sub2api --since=30m
kubectl -n basic-components logs deployment/ingress-nginx-controller --since=30m
kubectl -n storage get ingress minio-api

Install (ArgoCD)

🚀Installation

kubectl get namespace application >/dev/null 2>&1 || \
  kubectl create namespace application

set +x
read -rsp 'Sub2API admin password: ' ADMIN_PASSWORD; printf '\n'
read -rsp 'Sub2API PostgreSQL password: ' POSTGRES_PASSWORD; printf '\n'
read -rsp 'Sub2API Redis password: ' REDIS_PASSWORD; printf '\n'
JWT_SECRET="$(openssl rand -hex 32)"
TOTP_KEY="$(openssl rand -hex 32)"

test -n "$ADMIN_PASSWORD"
test -n "$POSTGRES_PASSWORD"
test -n "$REDIS_PASSWORD"

kubectl -n application create secret generic sub2api-auth \
  --from-literal=admin-password="$ADMIN_PASSWORD" \
  --from-literal=jwt-secret="$JWT_SECRET" \
  --from-literal=totp-encryption-key="$TOTP_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n application create secret generic sub2api-external-postgresql \
  --from-literal=postgres-password="$POSTGRES_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n application create secret generic sub2api-redis \
  --from-literal=redis-password="$REDIS_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

unset ADMIN_PASSWORD POSTGRES_PASSWORD REDIS_PASSWORD JWT_SECRET TOTP_KEY
kubectl -n application get secret \
  sub2api-auth sub2api-external-postgresql sub2api-redis

git -C /home/aaron/Ops/docs fetch origin main
git -C /home/aaron/Ops/docs \
  show origin/main:manifests/sub2api-argocd.yaml >/dev/null

kubectl -n argocd get application ops-docs \
  -o jsonpath='{.spec.source.repoURL}{"\n"}{.spec.source.path}{"\n"}'

argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300

argocd app get sub2api --hard-refresh
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600

kubectl -n argocd get application sub2api \
  -o jsonpath='{.spec.source.repoURL}{"\n"}{.spec.source.chart}{" "}{.spec.source.targetRevision}{"\n"}'

kubectl -n application get deployment sub2api \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"="}{.image}{"\n"}{end}'

kubectl -n application rollout status deployment/sub2api --timeout=600s
kubectl -n application get pods,svc,ingress,pvc
kubectl -n application get certificate,certificaterequest,order,challenge
kubectl -n application get endpointslice \
  -l kubernetes.io/service-name=sub2api

curl -fsS https://token.72602.space/health
curl -fsS https://token.72602.space/api/v1/settings/public

set +x
read -rsp 'Sub2API API token: ' SUB2API_API_TOKEN; printf '\n'
curl -fsS \
  -H "Authorization: Bearer ${SUB2API_API_TOKEN}" \
  https://token.72602.space/v1/models

read -rp 'Model ID for smoke generation: ' MODEL_ID
curl -fsS https://token.72602.space/v1/chat/completions \
  -H "Authorization: Bearer ${SUB2API_API_TOKEN}" \
  --json "{\"model\":\"${MODEL_ID}\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with OK.\"}],\"max_tokens\":8}"
unset SUB2API_API_TOKEN MODEL_ID

Ops

Web Page

sub2api web page (https://token.72602.space)

Current Release

argocd app get ops-docs --hard-refresh
argocd app get sub2api --hard-refresh

kubectl -n argocd get application sub2api \
  -o jsonpath='{.spec.source.repoURL}{"\nchart="}{.spec.source.chart}{" "}{.spec.source.targetRevision}{"\nimage.tag="}{.spec.source.helm.parameters[?(@.name=="image.tag")].value}{"\n"}'

kubectl -n application get deployment sub2api \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"="}{.image}{"\n"}{end}'

kubectl -n application get pods,svc,ingress,pvc
kubectl -n application get certificate,certificaterequest,order,challenge

The expected values are OCI chart 0.1.15, image ghcr.io/wei-shaw/sub2api@sha256:4c5dffab6e5ba4d3bd5382f19aad9654847b4e23de1a3d48e190146a3e6eb977, namespace application, and host token.72602.space. The application PVC is 10Gi; the Redis PVC is 8Gi with AOF enabled. Both use local-path and RWO.

Sync From Git

argocd/ops-docs owns manifests/sub2api-argocd.yaml from https://github.com/AaronYang0628/docs.git. Reconcile the Git parent before the OCI Helm child:

git -C /home/aaron/Ops/docs fetch origin main
argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300

argocd app get sub2api --hard-refresh
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600

Do not apply manifests/sub2api-argocd.yaml directly as a second ownership path.

Rolling Upgrade

1.back up PostgreSQL before changing chart or image values

Use the Backup & Restore runbook. Sub2API executes PostgreSQL migrations automatically at startup, and migrations are forward-only. Verify the dump before continuing.

2.update the Git source

Edit only the reviewed targetRevision, image.tag, or required values in manifests/sub2api-argocd.yaml, then inspect and publish the change:

git -C /home/aaron/Ops/docs diff --check -- manifests/sub2api-argocd.yaml
git -C /home/aaron/Ops/docs diff -- manifests/sub2api-argocd.yaml
git -C /home/aaron/Ops/docs add manifests/sub2api-argocd.yaml
git -C /home/aaron/Ops/docs commit -m "chore: upgrade sub2api"
git -C /home/aaron/Ops/docs push origin HEAD:main

3.reconcile parent and child Applications

argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300

argocd app get sub2api --hard-refresh
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600
kubectl -n application rollout status deployment/sub2api --timeout=600s
kubectl -n application get endpointslice \
  -l kubernetes.io/service-name=sub2api -o wide

The Deployment explicitly uses maxUnavailable: 0 and maxSurge: 1. Kubernetes adds the new Ready Pod to the EndpointSlice before terminating the old Pod, and the Service selects only Ready endpoints. This protects new requests during rollout, but it does not guarantee completion of requests that are already attached to the terminating Pod. Long generations still require the application to handle graceful termination and draining correctly.

4.verify the public and authenticated model path

curl -fsS https://token.72602.space/health
curl -fsS https://token.72602.space/api/v1/settings/public

set +x
read -rsp 'Sub2API API token: ' SUB2API_API_TOKEN; printf '\n'
curl -fsS \
  -H "Authorization: Bearer ${SUB2API_API_TOKEN}" \
  https://token.72602.space/v1/models

read -rp 'Model ID for smoke generation: ' MODEL_ID
curl -fsS https://token.72602.space/v1/chat/completions \
  -H "Authorization: Bearer ${SUB2API_API_TOKEN}" \
  --json "{\"model\":\"${MODEL_ID}\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with OK.\"}],\"max_tokens\":8}"
unset SUB2API_API_TOKEN MODEL_ID

Admin MCP

The Ops Agent Pod runs a local MCP server sub2api-admin against the cluster-local endpoint http://sub2api.application.svc.cluster.local:8080. The Deployment injects the API key from the dedicated runtime Secret application/sub2api-mcp key admin-api-key. Provision the Secret through manifests/ops-agent/create-sub2api-mcp-secret.sh; the value never enters Git or opencode.json. The MCP only accepts the configured cluster-local host, every authenticated GET runs through a reviewed route allowlist, sensitive fields are recursively redacted, and no generic authenticated write tool is exposed.

Enter the key only into the helper’s silent interactive prompt. Never place the key in shell arguments, chat, Git, OpenCode config, logs, or command output.

Tools

Nine tools are registered:

The global OpenCode permission set denies every sub2api-admin_* tool. The 72602-k3s-maintainer agent allows the six read-only tools directly and asks before recharge, update_user_limits, or set_account_schedulable. Use the maintainer for any authenticated mutation.

Recharge protocol

Recharge is additive USD-style internal credit only; set and subtract operations are not exposed. Before every charge:

1. Collect an exact email and a positive amount.
2. Call sub2api-admin_preview_recharge. The server fuzzy-searches but accepts only one exact normalized email match.
3. Read back the matched email, the immutable user id, status, the current balance, the amount, and the expected balance; obtain explicit confirmation for that exact tuple.
4. Generate one stable Idempotency-Key of 16-128 URL-safe characters. Keep and reuse that key when the result is uncertain; never generate a fresh key for a retry.
5. Call sub2api-admin_recharge with operation: add, the exact email and id, the audit note, and the idempotency key.
6. Re-read the user and balance history; record the exact verified or accepted_unverified outcome. Never rewrite an unverified accepted response as success.

No account is recharged as part of installation. Do not recharge inactive users unless the operator explicitly confirms the inactive status.

Limits and model availability

Sub2API v0.1.176 has no configurable per-model RPM. Operator scope:

• User RPM and concurrency via sub2api-admin_update_user_limits.
• Group RPM is set through the admin UI; this MCP does not expose it.
• rate_multiplier changes billing, not request rate.

For temporary model unavailability, call sub2api-admin_model_control_options first and identify the model’s actual scheduling boundary:

• Dedicated account: sub2api-admin_set_account_schedulable toggles every model served by that account; roll back with the inverse value.
• Restricted channel: remove the exact model from the channel pricing/model list while restrict_models=true; preserve the previous payload for rollback. This MCP does not mutate channels, use the admin UI.
• Composite model: disable the exact composite route; preserve prior state. Use the admin UI.
• models_list_config only hides discovery. It does not block direct requests and must never be reported as a disable.

Scheduling changes affect new requests only. They do not cancel requests or streams already in progress. If the model spans multiple accounts, channels, or routes, enumerate and confirm every affected target before mutation.

Secret rotation

Sub2API supports only one Admin API Key. Generating or regenerating the key in the admin UI immediately invalidates the previous key, so the MCP will briefly fail to authenticate until the Ops Agent workload restarts with the new value. Plan an expected maintenance window of one to two restart cycles. Rotate in this exact order without intermediate pauses:

1. Generate the new key in the Sub2API admin UI and immediately enter it into the silent prompt of manifests/ops-agent/create-sub2api-mcp-secret.sh. Confirm the helper reports application/sub2api-mcp keys=admin-api-key.
2. Restart the managed workload so the new env var is injected:

   kubectl -n application rollout restart deployment/ops-agent
   kubectl -n application rollout status deployment/ops-agent --timeout=300s

3. Verify global/health reports healthy: true, /mcp reports exactly {"sub2api-admin":{"status":"connected"}}, and the merged OpenCode config carries the cluster-local SUB2API_BASE_URL without the new key.
4. Through 72602-k3s-maintainer, run sub2api-admin_read against /api/v1/admin/system/version; the live application must respond 0.1.176.

The upstream “generate key” action is itself the invalidation step; no follow-up disable of the previous key is needed or supported.

Tests: node --test .opencode/mcp/sub2api-admin/server.test.mjs passes three tests covering the allowlist, path-traversal rejection, and the recharge idempotency path.

GitOps Rollback

Restore the previous reviewed chart and image values, and any reviewed OpenCode configuration, with git revert or a new commit, then let ArgoCD converge:

git -C /home/aaron/Ops/docs log --oneline -- manifests/sub2api-argocd.yaml .opencode/opencode.json
git -C /home/aaron/Ops/docs revert <change-commit>
git -C /home/aaron/Ops/docs push origin HEAD:main
# Record rollback as a new Git revert commit (upgrade commit f6bf8df86d40d07493dafbc713ca37b31f80dd4e); backup artifacts are in `/home/aaron/Ops/backups/sub2api/upgrade-20260908T032857Z` for isolated DB restore.

argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600

kubectl -n application rollout restart deployment/ops-agent
kubectl -n application rollout status deployment/ops-agent --timeout=300s

After the Ops Agent Pod rolls back and no Deployment references the helper Secret, remove it:

test -z "$(kubectl -n application get deployment -o json \
  | jq -r '.items[].spec.template.spec.containers[]
          | select(.env[]?.valueFrom.secretKeyRef.name=="sub2api-mcp")
          | .name')" && \
  kubectl -n application delete secret sub2api-mcp

An image/chart rollback does not reverse a forward-only PostgreSQL migration. Confirm compatibility with the migrated schema; when database recovery is required, restore the pre-upgrade dump into a separate database and switch via a reviewed Git change. Do not use kubectl rollout undo, and do not delete PVCs or Secrets as a rollback step until the Git revert and ops-agent restart are Healthy.

Troubleshooting

kubectl -n application logs deployment/sub2api --since=10m
kubectl -n application get events --sort-by=.lastTimestamp
kubectl -n application get endpointslice \
  -l kubernetes.io/service-name=sub2api -o yaml
argocd app get sub2api --hard-refresh

🛎️FAQ

kubectl -n argocd get application sub2api \
  -o jsonpath='{.spec.source.helm.values}'
kubectl -n application get secret sub2api-external-postgresql
kubectl -n application logs deployment/sub2api --since=10m

redis:
  auth:
    existingSecret: sub2api-redis
    existingSecretPasswordKey: redis-password

kubectl -n application get secret sub2api-redis
kubectl -n application get deployment sub2api \
  -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}{" <- "}{.valueFrom.secretKeyRef.name}{"/"}{.valueFrom.secretKeyRef.key}{"\n"}{end}'
kubectl -n application logs deployment/sub2api --since=10m