Proxy

Open WebUI

Web Page

https://txt2img.agent.72602.online

🚀Installation

Install By

Preliminary

1. Kubernetes is installed; if not, check 🔗link

2. Helm is installed; if not, check 🔗link

3. ArgoCD is installed; if not, check 🔗link

1.prepare Redis URL secret

```shell REDIS_PASS=$(kubectl -n storage get secret redis-shared-credentials -o jsonpath='{.data.redis-password}' | base64 -d) kubectl -n ai create secret generic open-webui-redis-url \ --from-literal=redis-url="redis://:${REDIS_PASS}@redis-shared-master.storage.svc.cluster.local:6379/0" ```

2.prepare TLS certificate

```shell kubectl -n ai apply -f - <<'EOF' apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: txt2img-agent-tls spec: secretName: txt2img-agent-tls dnsNames: - txt2img.agent.72602.online issuerRef: name: lets-encrypt kind: ClusterIssuer EOF ```

3.deploy via ArgoCD

```shell kubectl -n argocd apply -f - <<'EOF' apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: open-webui spec: project: default syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true source: repoURL: https://open-webui.github.io/helm-charts chart: open-webui targetRevision: 14.5.0 helm: releaseName: open-webui values: | image: repository: m.daocloud.io/ghcr.io/open-webui/open-webui pullPolicy: IfNotPresent ollama: enabled: false pipelines: enabled: false websocket: enabled: true manager: redis existingSecret: open-webui-redis-url existingSecretKey: redis-url redis: enabled: false ingress: enabled: true class: nginx host: txt2img.agent.72602.online tls: true existingSecret: txt2img-agent-tls annotations: cert-manager.io/cluster-issuer: lets-encrypt persistence: enabled: true size: 2Gi storageClass: local-path extraEnvVars: - name: OPENAI_API_BASE_URL value: "http://sub2api.application.svc.cluster.local:8080/v1" - name: DEFAULT_USER_ROLE value: "admin" destination: server: https://kubernetes.default.svc namespace: ai EOF ```

4.verify

```shell argocd app get open-webui kubectl -n ai get pods -l app.kubernetes.io/instance=open-webui ```

⚡72602 Cluster Config

配置项	值
域名	`txt2img.agent.72602.online`
AI 后端	`sub2api.application.svc:8080/v1`（OpenAI-compatible）
Redis	`redis-shared-master.storage.svc:6379`（共享）
数据库	SQLite（PVC 2Gi, local-path）
镜像	`m.daocloud.io/ghcr.io/open-webui/open-webui:0.9.5`
部署方式	ArgoCD Helm chart `open-webui-14.5.0`

🛠️2026-05 实际部署记录（72602）

当前集群已按本文部署，argocd/open-webui 已创建并进入 Synced，首次启动会执行 DB migration，StatefulSet 短时间显示 Progressing 属于正常现象。

准备 Redis URL Secret（ai/open-webui-redis-url）

REDIS_PASS=$(kubectl -n storage get secret redis-shared-credentials -o jsonpath='{.data.redis-password}' | base64 -d)
kubectl -n ai create secret generic open-webui-redis-url \
  --from-literal=redis-url="redis://:${REDIS_PASS}@redis-shared-master.storage.svc.cluster.local:6379/0" \
  --dry-run=client -o yaml | kubectl apply -f -

确认证书（已存在可复用）

kubectl -n ai get certificate txt2img-agent-tls

通过仓库清单部署 ArgoCD Application

kubectl -n argocd apply -f /home/aaron/Ops/docs/manifests/application/open-webui.yaml

验证

argocd app get open-webui
kubectl -n ai get pods,svc,ingress
kubectl -n ai logs statefulset/open-webui --tail=120

🔁回滚步骤（OpenWebUI）

kubectl -n argocd delete application open-webui --ignore-not-found
kubectl -n ai delete ingress,svc,statefulset,pvc,secret -l app.kubernetes.io/instance=open-webui --ignore-not-found
kubectl -n ai delete secret open-webui-redis-url --ignore-not-found

若需恢复旧版本：

kubectl apply -f /home/aaron/Ops/docs/manifests/application/openwebui-backup-ai-20260513.yaml

✅验证结果（2026-05）

argocd/open-webui: Sync=Synced, Health=Progressing（初始化阶段）
ai 命名空间资源：
- StatefulSet/open-webui
- Service/open-webui
- Ingress/open-webui（txt2img.agent.72602.online）
启动日志包含 Alembic migration，未出现 CrashLoopBackOff

📦Shared Dependencies

Redis: manifests/storage/redis.yaml → Bitnami Redis chart, standalone, 2Gi local-path
sub2api: manifests/application/sub2api-argocd.yaml → AI API proxy

🔧Operations (72602)

1.cleanup `open-webui`

kubectl delete application -n argocd open-webui --ignore-not-found
helm uninstall -n ai open-webui || true
kubectl -n ai delete ingress,statefulset,svc,secret,pvc -l app.kubernetes.io/instance=open-webui --ignore-not-found

2.rollback `open-webui`

kubectl apply -f /home/aaron/Ops/docs/manifests/openwebui-backup-ai-20260513.yaml

🧱Pitfalls we already hit (2026-05)

OpenWebUI image button only enables UI entry, not backend readiness.
OPENAI_API_BASE_URL and provider mapping may be overridden by persisted config (Redis/DB), so env var change alone may not take effect.
Using image-only model in chat path can trigger handler errors in 0.9.5:
- TypeError: 'JSONResponse' object is not subscriptable
- TypeError: 'StreamingResponse' object is not subscriptable
sub2api in-cluster endpoint may return 404 on /v1/images/generations; verify real upstream supports image endpoint before enabling image generation.
Browser cache / service worker can keep old API path and cause false 404 symptoms after frontend migration.

✅Monitor / Verify Checklist

argocd app get open-webui
kubectl -n ai get pods,svc,ingress
kubectl -n ai logs statefulset/open-webui --tail=200
curl -vkI https://txt2img.agent.72602.online

If image generation fails, verify in order:

upstream supports /v1/images/generations
provider/model mapping in OpenWebUI admin page
no stale client cache/service worker

Sub2API

Web Page

sub2api web page (https://sub2api.72602.online)

Current State

ArgoCD Application: argocd/sub2api
Namespace: ai
Chart: ghcr.io/ben-wangz/k8s-at-home-charts/sub2api (0.1.1)
Ingress host: sub2api.72602.online
Ingress class: nginx
ClusterIssuer: lets-encrypt
External PostgreSQL: postgresql.database.svc.cluster.local:5432 (database/user: sub2api)

Docs

Backup & Restore
Sub2API database and app-level backup runbook
Install (ArgoCD)
Use ArgoCD to deploy sub2api in application namespace
Ops
Daily operations for sub2api

Backup & Restore

Scope

Database: PostgreSQL (database/postgresql-0), DB sub2api
App config: ops-assets/sub2api-argocd.yaml + ai namespace secrets
Runtime data path: /app/data (mounted from PVC sub2api-data)

Database Backup

TS=$(date +%F-%H%M%S)
mkdir -p /home/aaron/Ops/backups/sub2api

kubectl -n database exec postgresql-0 -- \
  env PGPASSWORD='REPLACE_POSTGRES_PASSWORD' \
  pg_dump -U postgres -d sub2api -Fc \
  > /home/aaron/Ops/backups/sub2api/sub2api-${TS}.dump

Database Restore

# 1) create a restore database first (recommended)
kubectl -n database exec postgresql-0 -- \
  env PGPASSWORD='REPLACE_POSTGRES_PASSWORD' \
  psql -U postgres -d postgres -c "CREATE DATABASE sub2api_restore OWNER sub2api;"

# 2) restore dump to restore database
cat /home/aaron/Ops/backups/sub2api/sub2api-YYYY-MM-DD-HHMMSS.dump | \
kubectl -n database exec -i postgresql-0 -- \
  env PGPASSWORD='REPLACE_POSTGRES_PASSWORD' \
  pg_restore -U postgres -d sub2api_restore --clean --if-exists

# 3) verify key tables
kubectl -n database exec postgresql-0 -- \
  env PGPASSWORD='REPLACE_POSTGRES_PASSWORD' \
  psql -U postgres -d sub2api_restore -c "SELECT count(*) FROM users;"

Sub2API App-Level Backup

TS=$(date +%F-%H%M%S)
mkdir -p /home/aaron/Ops/backups/sub2api/${TS}

# 1) backup ArgoCD app manifest source
cp /home/aaron/Ops/docs/manifests/application/sub2api-argocd.yaml \
  /home/aaron/Ops/backups/sub2api/${TS}/sub2api-argocd.yaml

# 2) backup in-cluster objects
kubectl -n ai get deploy,svc,ingress,pvc sub2api -o yaml \
  > /home/aaron/Ops/backups/sub2api/${TS}/sub2api-k8s.yaml

kubectl -n ai get secret sub2api-auth -o yaml \
  > /home/aaron/Ops/backups/sub2api/${TS}/sub2api-auth.secret.yaml

kubectl -n ai get secret sub2api-external-postgresql -o yaml \
  > /home/aaron/Ops/backups/sub2api/${TS}/sub2api-external-postgresql.secret.yaml

Pre-Restore Safety Checklist

Confirm rollback target (sub2api or sub2api_restore) before switching DB.
Keep current dump before any restore/import.
After restore, verify login API and settings API:

curl -sk -o /dev/null -w '%{http_code}\n' https://sub2api.72602.online/api/v1/settings/public
kubectl -n ai logs deploy/sub2api --since=10m

Install (ArgoCD)

Preliminary

ArgoCD 已可访问: https://argocd.72602.online
cert-manager 与 lets-encrypt 已就绪
DNS 已解析: sub2api.72602.online -> 47.110.67.161
PostgreSQL(database ns) 可用

Deployment

Deploy

PG_PASSWORD=$(kubectl -n database get secret postgresql-credentials -o jsonpath='{.data.postgres-password}' | base64 -d)
JWT_SECRET=$(openssl rand -hex 32)
TOTP_KEY=$(openssl rand -hex 32)

kubectl get ns application > /dev/null 2>&1 || kubectl create namespace application

kubectl -n application create secret generic sub2api-auth \
  --from-literal=admin-password='CHANGE_ME_STRONG_PASSWORD' \
  --from-literal=jwt-secret="$JWT_SECRET" \
  --from-literal=totp-encryption-key="$TOTP_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n application create secret generic sub2api-external-postgresql \
  --from-literal=postgres-password="$PG_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n application create secret generic sub2api-redis \
  --from-literal=redis-password='CHANGE_ME_REDIS_PASSWORD' \
  --dry-run=client -o yaml | kubectl apply -f -

ARGOCD_PASS=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)
argocd login --insecure --username admin argocd.72602.online --password $ARGOCD_PASS

kubectl apply -f /home/aaron/Ops/docs/manifests/application/sub2api-argocd.yaml
argocd app sync sub2api
argocd app wait sub2api --health --timeout 300

Verify

argocd app get sub2api
kubectl -n application get pods,svc,ingress
kubectl -n application get certificate,certificaterequest,order,challenge
curl -vkI https://sub2api.72602.online

Ops

Web Page

sub2api web page (https://sub2api.72602.online)

Health Check

argocd app get sub2api
kubectl -n ai get pods
kubectl -n ai logs deploy/sub2api --tail=80

Ingress and TLS

kubectl -n ai get ingress sub2api -o wide
kubectl -n ai get certificate,certificaterequest,order,challenge
curl -vkI https://sub2api.72602.online

Restart

kubectl -n ai rollout restart deploy/sub2api
kubectl -n ai rollout status deploy/sub2api

Rotate Auth Secrets

kubectl -n ai create secret generic sub2api-auth \
  --from-literal=admin-password='CHANGE_ME_STRONG_PASSWORD' \
  --from-literal=jwt-secret="$(openssl rand -hex 32)" \
  --from-literal=totp-encryption-key="$(openssl rand -hex 32)" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n ai rollout restart deploy/sub2api

Sync From ArgoCD

ARGOCD_PASS=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)
argocd login --insecure --username admin argocd.72602.online --password $ARGOCD_PASS

argocd app sync sub2api
argocd app wait sub2api --health --timeout 300

Backup and Restore

Database + app-level backup runbook: Ops/Sub2API/Backup & Restore
File path: /home/aaron/Ops/docs/content/Installation/Application/Sub2API/backup.md

Troubleshooting: `column settings.id does not exist`

Symptom: GET /api/v1/settings/public or admin settings APIs return 500, logs show pq: column settings.id does not exist.

Root cause: sub2api is pointed to a shared/legacy database schema (for example n8n DB) where settings table structure is incompatible.

Fix (minimal):

# 1) use dedicated DB/user for sub2api
# update /home/aaron/Ops/docs/manifests/application/sub2api-argocd.yaml
# externalPostgresql.username= sub2api
# externalPostgresql.database= sub2api

# 2) create DB/user in PostgreSQL and rotate secret
# (run with postgres superuser)

# 3) apply and rollout
kubectl apply -f /home/aaron/Ops/docs/manifests/application/sub2api-argocd.yaml
kubectl -n ai rollout restart deploy/sub2api
kubectl -n ai rollout status deploy/sub2api

Troubleshooting: Redis `WRONGPASS`

Symptom: kubectl -n ai logs deploy/sub2api shows WRONGPASS invalid username-password pair.

Root cause: Redis password changed (for example chart regenerated secret), but sub2api pod still used old REDIS_PASSWORD env var.

Fix:

# 1) ensure ArgoCD values pin Redis secret (in sub2api-argocd.yaml)
# redis.auth.existingSecret: sub2api-redis
# redis.auth.existingSecretPasswordKey: redis-password

# 2) apply and restart sub2api to reload env
kubectl apply -f /home/aaron/Ops/docs/manifests/application/sub2api-argocd.yaml
kubectl -n ai rollout restart deploy/sub2api
kubectl -n ai rollout status deploy/sub2api

# 3) verify
kubectl -n ai logs deploy/sub2api --tail=120

Verify:

curl -sk -o /dev/null -w '%{http_code}\n' https://sub2api.72602.online/api/v1/settings/public
kubectl -n ai logs deploy/sub2api --since=10m

Proxy

Subsections of Proxy

Open WebUI

Web Page

🚀Installation

⚡72602 Cluster Config

🛠️2026-05 实际部署记录（72602）

🔁回滚步骤（OpenWebUI）

✅验证结果（2026-05）

📦Shared Dependencies

🔧Operations (72602)

🧱Pitfalls we already hit (2026-05)

✅Monitor / Verify Checklist

Sub2API

Web Page

Current State

Docs

Subsections of Sub2API

Backup & Restore

Scope

Database Backup

Database Restore

Sub2API App-Level Backup

Pre-Restore Safety Checklist

Install (ArgoCD)

Preliminary

Deployment

Verify

Ops

Web Page

Health Check

Ingress and TLS

Restart

Rotate Auth Secrets

Sync From ArgoCD

Backup and Restore

Troubleshooting: column settings.id does not exist

Troubleshooting: Redis WRONGPASS

Troubleshooting: `column settings.id does not exist`

Troubleshooting: Redis `WRONGPASS`