docs

Deployment

Production runs on DigitalOcean Kubernetes — one cluster, four apps, one ingress.

truelabel.fun runs on DigitalOcean Kubernetes (DOKS). A single ingress, backed by one DO Load Balancer, routes four hostnames to four Deployments; cert-manager keeps all four on real TLS via Let's Encrypt. Postgres and Redis are not in the cluster — Neon and Upstash instead, reached over the network like any other dependency.

DNS (truelabel.fun + 3 subdomains)
  → DO Load Balancer
    → ingress-nginx
        truelabel.fun           → web        (marketing)
        api.truelabel.fun       → backend
        dashboard.truelabel.fun → dashboard
        docs.truelabel.fun      → docs
              ↓
      Neon (Postgres) · Upstash (Redis) — external, not in the cluster

The droplet this replaced

Earlier versions of this doc described a single droplet running the backend behind nginx + systemd. That's retired — see deploy/README.md at the repo root, kept only as reference for a possible future move to self-hosted bare metal, where that shape is closer to right again than these Kubernetes manifests are.

One-time cluster setup

Needs doctl, terraform, kubectl, and helm locally. Run once, not on every deploy.

Authenticate doctl with a DigitalOcean API token (Settings → API → Generate New Token in the DO console). Run this yourself in your own terminal — it's interactive and the token should never be pasted anywhere else:

doctl auth init

Provision the cluster.

cd infra/digitalocean/cluster
cp terraform.tfvars.example terraform.tfvars   # fill in do_token — gitignored, stays local
terraform init
terraform apply
doctl kubernetes cluster kubeconfig save true-label
kubectl get nodes   # sanity check

s-2vcpu-4gb nodes, autoscaling 1–3, region blr1 by default — see infra/digitalocean/cluster/variables.tf to change either.

Install ingress-nginx — the one thing in this cluster that gets a real DO Load Balancer:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace
kubectl get svc -n ingress-nginx ingress-nginx-controller -w

Wait for EXTERNAL-IP to populate — that's the one address every hostname below points at.

Point DNS at it, at whatever registrar holds truelabel.fun:

truelabel.fun           A  <EXTERNAL-IP>
api.truelabel.fun       A  <EXTERNAL-IP>
dashboard.truelabel.fun A  <EXTERNAL-IP>
docs.truelabel.fun      A  <EXTERNAL-IP>

Install cert-manager, then apply the one-time ClusterIssuer (fill in a real contact email first — Let's Encrypt only uses it for expiry notices):

helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace --set crds.enabled=true

# edit k8s/base/cert-manager/cluster-issuer.yaml: CHANGE_ME@example.com → your email
kubectl apply -f k8s/base/cert-manager/cluster-issuer.yaml

This CR is cluster-scoped and applied standalone, once — it isn't part of any kustomize overlay (see that file and k8s/base/kustomization.yaml's comments for why).

Secrets — GitHub Actions, never this repo

Everything below is typed into GitHub's own Settings → Secrets and variables → Actions UI, never into a file this repo tracks. .gitignore already refuses .env, k8s/**/secret.yaml (only *.example.yaml is meant to be committed), and *.tfvars.

SecretWhere it comes from
DOCKERHUB_USERNAME / DOCKERHUB_TOKENDocker Hub → Account Settings → Security
DIGITALOCEAN_ACCESS_TOKENDO console → API. Scope it as narrowly as the console allows.
PROD_DATABASE_URLNeon project → Connection string (postgresql://...neon.tech/truelabel?sslmode=require)
PROD_REDIS_URLUpstash database → rediss://...upstash.io:6379 — TLS, matches the rediss:// scheme the backend's redis crate already supports
PROD_GITHUB_TOKENOptional. GitHub → Settings → Developer settings → a fine-grained PAT scoped to Issues: write on this repo. Powers the dashboard's "publish as GitHub issue" button only — omit it and everything else still works, that one button just fails with a clear error.
PROD_GITHUB_WEBHOOK_SECRETOptional. A random secret you generate (openssl rand -hex 32) and set on both sides: here, and on the repo's own webhook (Settings → Webhooks → Add webhook → Payload URL https://api.truelabel.fun/api/v1/webhooks/github, content type application/json, same secret, subscribed to Issues only). Lets closing/reopening an issue directly on GitHub sync back onto the linked crash report's status. Omit it and that one endpoint just refuses every delivery with a clear error.
PROD_GOOGLE_OAUTH_CLIENT_IDOptional. The Web application OAuth client ID from Google Cloud Console (not the separate Android-type client, which has no secret and is never sent here) — the JWT audience the backend validates a Google ID token's aud claim against. Powers the Android app's "Sign in with Google" only. Omit it and POST /api/v1/me/link/google just refuses every request with a clear error.

CI creates/updates backend-secrets and dashboard-secrets in the cluster from these on every deploy to main (see .github/workflows/ci-cd.yml's deploy job) — see k8s/base/backend/secret.example.yaml and k8s/base/dashboard/secret.example.yaml for the shape, if you ever need to create either by hand against the cluster instead.

Migrating off Neon/Upstash later

The backend only ever reads DATABASE_URL/REDIS_URL — nothing in the code or manifests assumes either provider specifically. Moving to self-hosted Postgres/Redis (bare metal, or back in-cluster) later is exactly two secret values changing, nothing else. k8s/base/postgres/ and redis/ still have the in-cluster manifests from before this migration, excluded from the kustomization but not deleted, for exactly that day.

Deploying

git push origin main

.github/workflows/ci-cd.yml builds and pushes all four images, then (only on main) authenticates doctl, syncs secrets, pins every image to this commit's SHA, and runs kubectl apply -k k8s/overlays/main.

Verifying it worked

curl -sI https://api.truelabel.fun/health
curl -sI https://truelabel.fun
curl -sI https://dashboard.truelabel.fun
curl -sI https://docs.truelabel.fun

# TLS actually issued — READY should read True for all four hosts
kubectl get certificate -n true-label

Rolling back a bad deploy

Every image is tagged with the exact commit SHA that built it (ci-cd.yml's deploy job pins k8s/overlays/main's image tags before applying), so kubectl already has everything it needs — nothing to rebuild or re-tag by hand.

# What actually happened — see the last few revisions and which rolled
# out when. Repeat per deployment: backend, web, dashboard, docs.
kubectl rollout history deployment/backend -n true-label

# Back out one step. Kubernetes re-applies the previous revision's pod
# spec (image tag included) and does a normal rolling update — no
# downtime, same as a forward deploy.
kubectl rollout undo deployment/backend -n true-label

# Watch it land.
kubectl rollout status deployment/backend -n true-label

# Need a specific older revision, not just "one back"?
kubectl rollout undo deployment/backend -n true-label --to-revision=<N>

Roll back only the app that's actually broken — backend, web, dashboard, and docs deploy and roll back independently, so reverting one doesn't touch the others.

This doesn't roll back the database

A rollback undoes the code; SQL migrations that already ran stay ran — sqlx migrations in this app are additive/forward-only by convention, not written with a paired down migration. If a bad deploy shipped a migration the old code can't handle, rollout undo alone isn't enough; that needs a manual fix, not this command.

Once you're confident the rollback is right (not just a stopgap), also revert or fix-forward the commit in git — rollout undo fixes the running cluster, not the repo, and the next git push to main would otherwise redeploy the exact thing you just backed out of.

Recovering a locked-out sole admin

Two API endpoints cover normal password recovery — see Authentication → Password recovery — and neither helps if you're the only admin and you're the one locked out: one needs your current password, the other needs a different admin to call it on your behalf. Neither exists yet in a fresh deployment, or ever again if you're truly the only one.

This is why backend's binary has a --hash-password mode — reads a password from stdin, prints an Argon2 hash using the exact same code path register/reset-password use, then exits without touching the network or starting a server:

# Wherever the binary (or its source) is available — your machine is fine,
# it never needs to reach the database itself.
echo -n "your-new-password" | ./truelabel-backend --hash-password
# $argon2id$v=19$m=19456,t=2,p=1$...

Then write that hash straight to Neon (its own SQL console, or psql $DATABASE_URL):

UPDATE dashboard_users
SET password_hash = '<the $argon2id$... output above>', updated_at = NOW()
WHERE email = 'you@example.com';

Log in with the new password immediately after — the row's token_hash isn't touched by this, so any session that was still valid stays valid too.

Never write a plaintext password into that column

password_hash is exactly that — a hash. Writing plaintext there doesn't fail loudly; it just means nothing can ever log in again until someone runs this same procedure to fix it. The --hash-password step above isn't optional.

Load testing — what the resource requests/limits are actually based on

k8s/base/backend/deployment.yaml's requests: 100m/64Mi, limits: 500m/256Mi were reasonable-looking defaults, not measured ones, until this: wrk -t4 -c50 -d20s against /health/ready (a real DB + Redis round trip, not a static response), backend built --release, against local Postgres/Redis —

164,986 requests in 20.01s → 8,245 req/s sustained
RSS:  ~16 MB   (limit: 256Mi — enormous headroom)
CPU:  ~160%    (limit: 500m = 50% of one core)

Memory is barely touched — 256Mi is safe to leave as-is, tightening it buys nothing. CPU is the real ceiling: 500m would throttle a single pod well before 8k req/s. That sounds alarming until the HPA is in the picture — k8s/base/backend/hpa.yaml targets 70% of the request (100m), i.e. ~70m CPU per pod, so it scales out to a second/third replica at a small fraction of what it took to saturate one pod here. Net: the configured numbers hold up, at least for a cheap DB-touching read — not proof against every request shape, but no longer a guess either.

The more concrete, actually-actionable number this surfaced: backend-config sets MAX_DB_CONNECTIONS=10 per pod. At minReplicas: 2 that's 20 connections to Neon; at maxReplicas: 6 under real scale-out, 60. Confirm whatever Neon plan is live has headroom for 60 concurrent connections (or sits behind Neon's own pooler) before assuming the HPA can actually reach 6 replicas without the database itself becoming the limit instead of CPU.

The one setting that matters here

TRUSTED_PROXY_HOPS=1

Set in k8s/base/backend/configmap.yaml. ingress-nginx is the only proxy that appends to X-Forwarded-For in this setup — DO's Load Balancer is L4 and doesn't touch HTTP headers — so 1 is correct as long as nothing else sits between it and the app. Get this wrong and the rate limiter in Rate Limiting either silently stops meaning anything (too high) or throttles every user as one shared bucket (should be higher than it is). The service logs an error at boot if it detects either mistake.

dev / sit / test overlays

k8s/overlays/{dev,sit,test} exist for local or manual multi-env parity — own namespace, smaller replica counts — but only main is wired into CI's deploy job, and only main includes the ingress (see k8s/base/kustomization.yaml's comment on why the others don't: an Ingress claiming api.truelabel.fun etc. from three namespaces at once would fight itself). Reach the others via kubectl port-forward instead of a hostname.

On this page