Upgrade Keycloak from 24 to 26

Platform release 3.27 upgrades the bundled Keycloak from 24.0.5 to 26.x. This is a two-major-version jump with several breaking configuration changes. This guide covers the values changes, the rollout, and the rollback path for self-hosted deployments.

One-way database migration

Keycloak migrates its database schema automatically on first boot of the new version, and the migration is not reversible. Back up (snapshot) the Keycloak database before upgrading. Rollback requires restoring this backup.

Service impact: schedule planned downtime

Keycloak is unavailable while its StatefulSet rolls, and the first pod of the new version runs the schema migration before it reports healthy. Allow up to 10 minutes for that migration (the startup probe is configured for it). During the window:

• Interactive logins and token refreshes fail.
• All existing user sessions are cleared by the Keycloak 26 cache marshalling change, so every user re-authenticates once after the upgrade.
• The API deployment rolls as part of the same change.

Run this upgrade in a planned maintenance window rather than during production traffic.

What changes in Keycloak 25/26

Area	Keycloak 24 (old)	Keycloak 26 (new)
Hostname config	KC_HOSTNAME_URL, KC_HOSTNAME_ADMIN_URL, KC_HOSTNAME_STRICT_HTTPS (hostname v1)	--hostname=<url>, --hostname-admin=<url> (hostname v2); v1 options are not recognized
Reverse proxy	KC_PROXY=edge / proxy: edge	Removed; replaced by proxyHeaders: xforwarded + httpEnabled: true
Health & metrics	Served on the main HTTP port	Served on the dedicated management port 9000; no longer reachable on 8080/8443 or through the public gateway
Container image	Bitnami-based (/opt/bitnami/keycloak/...)	Hardened rebuild (/opt/iamguarded/keycloak/...); volume mount paths change
User sessions	In-memory	Cache marshalling changed; all sessions are cleared at upgrade; users re-login once

Prerequisites

• Platform images that understand the split Keycloak URLs (KEYCLOAK_BASE_URL vs KEYCLOAK_MANAGEMENT_URL) for the API, init containers, and DB migrations; included in 3.26.4 and later, and backported to maintained earlier release lines. Images without this support check Keycloak health at a URL that no longer serves it and will block startup; confirm with DynamoAI if you are unsure whether your release line has it.
• The updated platform chart version supporting the new Keycloak sub-chart (provided with the 3.27 release package).
• The new Keycloak container image mirrored into your registry.
• A snapshot/backup of the Keycloak database (see warning above).
• A planned maintenance window (see the service impact warning above).

Values changes

1. Keycloak block

Replace the Keycloak 24 configuration with the new pattern (substitute your auth hostname):

keycloak:
  image:
    registry: <your-registry>
    repository: <keycloak-26-image>
    tag: "26.5.5"
  command:
    - /opt/iamguarded/keycloak/bin/kc.sh
  args:
    - start
    - --db=postgres
    - --hostname=https://<auth-hostname>
    - --hostname-admin=https://<auth-hostname>/
    - --import-realm
  proxyHeaders: "xforwarded"   # replaces proxy: edge / KC_PROXY
  httpEnabled: true
  metrics:
    enabled: true              # exposes the management server (health/metrics) on :9000
  cache:
    enabled: true
    stack: jdbc-ping
    configFile: ""             # must be explicitly empty; see gotchas below
  startupProbe:
    enabled: true
    initialDelaySeconds: 30
    periodSeconds: 10
    failureThreshold: 60       # first boot runs the 24→26 schema migration; allow up to 10 min
  externalDatabase:
    host: <postgres-host>      # literal value required; see gotchas below
    port: 5432
    database: keycloak
    existingSecret: <postgres-secret-name>
    existingSecretUserKey: username
    existingSecretPasswordKey: password

Remove from extraEnvVars (no longer recognized): KEYCLOAK_EXTRA_ARGS hostname flags, KC_PROXY, KC_HOSTNAME_URL, KC_HOSTNAME_ADMIN_URL, KC_HOSTNAME_STRICT, KC_HOSTNAME_STRICT_HTTPS. Keep the realm-import (DYNAMOAI_*) variables and KC_HEALTH_ENABLED: "true".

Update volume mount paths from the Bitnami layout to the new image layout:

  extraVolumeMounts:
    - name: keycloak-realm-config
      mountPath: /opt/iamguarded/keycloak/data/import   # was /opt/bitnami/keycloak/data/import
      readOnly: true

2. Component URLs: the sharp edge

Repoint KEYCLOAK_MANAGEMENT_URL

If your existing values set KEYCLOAK_MANAGEMENT_URL to the external auth URL, it worked on Keycloak 24 because health lived on the main port. On Keycloak 26 the health endpoints move to management port 9000, which is not exposed through the public gateway. If this variable is not repointed, readiness checks poll a URL that never answers and the platform API never starts.

Update the API deployment, init containers, and DB migrations job env:

KEYCLOAK_BASE_URL:
  { value: "http://<keycloak-service>.<namespace>:8080" }        # main server; API calls
KEYCLOAK_MANAGEMENT_URL:
  { value: "http://<keycloak-service>-metrics.<namespace>:9000" } # management server; health checks

In-cluster service URLs are recommended for both (faster, no gateway round-trip). The browser-facing UI variables (DFL_KEYCLOAK_BASE_URL, VITE_KEYCLOAK_BASE_URL) keep the external URL; do not change them.

3. Chart-level gotchas (validated in real upgrades)

• External database host must be a literal value. The new Keycloak sub-chart no longer reads the DB host/port from a secret (existingSecretHostKey is ignored, and values validation fails with a keycloak: database error). Set externalDatabase.host and externalDatabase.port explicitly; only the user and password may come from existingSecret.

• Remove any custom themes volume. The new chart mounts its own writable emptyDir at /opt/iamguarded/keycloak/themes; defining your own volume at the same path fails the upgrade with a duplicate-mountPath patch error.

• The Keycloak StatefulSet must be recreated. The new chart changes immutable StatefulSet fields, so an in-place upgrade fails with Forbidden: updates to statefulset spec .... Delete the old StatefulSet first; Keycloak state lives in the external database, so nothing is lost; the upgrade recreates it:

  kubectl --context <ctx> -n <namespace> delete statefulset <keycloak-statefulset-name>

• Disable cloud-specific logging subcharts that don't apply to your environment; for example, on non-AWS clusters set dynamoai-amazon-cloudwatch.enabled: false if your chart package includes it.

• cache.configFile must be explicitly set to an empty string. If omitted, the chart passes --cache-config-file=cache-ispn.xml, a file the image does not ship, and Keycloak crash-loops at boot with Cache config file 'cache-ispn.xml' does not exist in the conf directory.

Rollout

1. Snapshot the Keycloak database.
2. Mirror the new Keycloak image and chart into your registry.
3. Delete the existing Keycloak StatefulSet (see gotchas above).
4. Apply the updated values via your GitOps flow. First boot takes several minutes (schema migration); the generous startup probe above covers this; do not restart the pod mid-migration.

Verification

kubectl --context <ctx> -n <namespace> get pods                    # keycloak + api Running
kubectl --context <ctx> -n <namespace> get svc | grep keycloak     # main svc AND -metrics svc (9000) present
kubectl --context <ctx> -n <namespace> run kc-health --rm -i --restart=Never --image=curlimages/curl -- \
  curl -s http://<keycloak-service>-metrics.<namespace>:9000/health/ready   # {"status": "UP"}

Then confirm in the browser: the admin console loads at https://<auth-hostname>, a platform user can log in (all users must re-login once; sessions were cleared), and an API call with a fresh token succeeds.

Rollback

1. Revert the values commit and reconcile (restores the Keycloak 24 image and configuration). Delete the Keycloak 26 StatefulSet first (same immutable-fields constraint applies in reverse).
2. Restore the Keycloak database from the pre-upgrade snapshot; the migrated schema is not compatible with Keycloak 24.
3. Restart the platform API after Keycloak is healthy.

Troubleshooting

Symptom	Likely cause
API stuck in init / readiness probing forever	KEYCLOAK_MANAGEMENT_URL still points at the external URL (see the sharp-edge warning)
VALUES VALIDATION: keycloak: database at upgrade	externalDatabase.host must be a literal value on the new chart
doesn't match $setElementOrder list patch error	Duplicate volume mount at the themes path; remove the custom themes volume
Forbidden: updates to statefulset spec	Immutable StatefulSet fields changed; delete the old StatefulSet and re-run the upgrade
Crash loop: Cache config file 'cache-ispn.xml' does not exist	cache.configFile not set; it must be an explicit empty string
Config fix applied but pod keeps crashing with the old error	Keycloak env comes from a ConfigMap via envFrom, which is resolved at pod creation; container restarts reuse stale env. Delete the pod so the StatefulSet recreates it.
ConfigMap still shows a removed key after a successful upgrade	Drift from an earlier failed upgrade attempt: Helm's three-way merge only diffs against the last deployed manifest and will not delete keys it never deployed. Remove the stray key manually, then recreate the pod.
Admin console redirect loop or "HTTPS required"	proxyHeaders/httpEnabled missing, or old KC_PROXY/hostname v1 vars still present
Realm not imported on fresh installs	Realm volume mounted at the old /opt/bitnami/... path
Keycloak pod restart loop during first boot	Startup probe window too short for the schema migration; raise failureThreshold
All users logged out after upgrade	Expected; sessions are cleared by the 26 cache change; one-time re-login