Rotating Credentials

Rotate the credential in Keycloak first, then update your secret store to match. Updating the secret store alone rotates nothing.

Most credentials the platform uses are written into Keycloak or the application database once, during the first installation. After that, the platform never reads the secret store value for them again. A rotation that changes only the secret store leaves the old credential live, produces no error anywhere, and is usually discovered later as a failed login.

Which Credentials Read the Secret Store at Runtime

Two classes of credential behave in opposite ways. Identify the class before planning a rotation.

Credential	Consumer	When the value is read	Rotation path
Keycloak admin password	Keycloak admin, master realm	First installation only	Keycloak, then secret store
Platform admin password	Platform user in the dynamo-ai realm	User creation only	Keycloak, then secret store
Default user password	Platform user in the dynamo-ai realm	User creation only	Keycloak, then secret store
Default user API key	Platform API bearer token	User creation only	Platform API, then secret store
Keycloak client secret	Platform to Keycloak authentication	Realm import, then every API start	Keycloak, secret store, then restart
Database and object store passwords	Platform services	Every start	Secret store, then restart
License	Platform API	Every start	Secret store, then restart

Only the final two rows rotate the way an operator expects. Everything above them requires the credential to be changed in Keycloak or the platform API first.

Confirm which credentials your deployment holds:

kubectl -n NAMESPACE get secret CORE_SECRET_NAME -o jsonpath='{.data}' | tr ',' '\n' | cut -d'"' -f2

Rotating the Keycloak Admin Password

Keycloak creates the admin account in the master realm during the first server start and stores the password hash in its database. Restarting Keycloak with a new value in the environment has no effect, because Keycloak ignores the bootstrap variables once the master realm exists.

1. Open the Keycloak Admin Console and sign in to the master realm as admin.
2. Select Users, then admin, then the Credentials tab.
3. Select Reset password, enter the new value, and set Temporary to off.
4. Update the corresponding key in your secret store.
5. Trigger a sync if your secret operator caches values, then confirm the stored value matches.

kubectl -n NAMESPACE get secret CORE_SECRET_NAME -o jsonpath='{.data.platformAdminPassword}' | base64 -d

No restart is required. Keycloak does not read this value again.

warning

The same secret key supplies both the Keycloak admin account and a platform user in the dynamo-ai realm. Changing the key without also resetting the Keycloak admin account leaves the two out of step, and the Admin Console continues to accept only the previous password.

Rotating a Platform User Password

Platform users such as the default user and the platform admin are created in the dynamo-ai realm during the first API start. The API sets a password only when it creates the account. On every later start it finds the existing account and returns without touching the credential.

Rotate through Keycloak:

1. Open the Keycloak Admin Console and select the dynamo-ai realm.
2. Select Users, choose the account, then the Credentials tab.
3. Select Reset password, enter the new value, and set Temporary to off.
4. Update the corresponding key in your secret store.

The platform API also exposes reset endpoints for the same purpose:

curl -X POST https://API_HOSTNAME/v1/user/reset-password/onboarded-user \
  -H "Authorization: Bearer API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"email":"USER_EMAIL"}'

The new password must satisfy the realm password policy. A value that fails the policy is rejected, and if the rejection happens during user creation the API reports no error to the caller. Read the active policy before choosing a value:

Realm settings > Authentication > Policies > Password policy

Rotating the Platform API Key

Request a new key as the account that owns it. The endpoint replaces the stored hash immediately, which invalidates the previous key in the same call.

curl -X POST https://API_HOSTNAME/v1/user/token \
  -H "Authorization: Bearer CURRENT_API_KEY"

The response contains the new key. Update your secret store to match.

Changing the secret store value alone has no effect. The platform writes the API key hash only when the account has no key recorded.

Rotating the Keycloak Client Secret

The platform authenticates to Keycloak with a confidential client secret. Keycloak imports the realm once, so the secret must be changed on the client itself, and the platform reads the value at every start.

1. Change the secret on the Keycloak client through the Admin Console or the Admin REST API.
2. Update the corresponding key in your secret store.
3. Restart the platform API so it reads the new value.

kubectl -n NAMESPACE rollout restart deployment/API_DEPLOYMENT_NAME
kubectl -n NAMESPACE rollout status deployment/API_DEPLOYMENT_NAME

warning

A PUT to the Keycloak client endpoint replaces the entire client representation. Send a partial body and Keycloak discards every field you omitted. Retrieve the client with GET, change the single field, and send the complete representation back.

To keep the secret out of the Keycloak database entirely, see Storing OIDC Client Secrets via Vault.

Recovering a Lost Keycloak Admin Password

Recovery is required when the secret store value no longer matches what Keycloak holds, which happens whenever the key is rotated without resetting the account.

Retrieve the previous value first. Most secret stores retain prior versions of a secret. The version in effect when the deployment was first installed is the password Keycloak still accepts. Sign in with it, then reset the account as described above so the two agree.

If no previous version is available, create a temporary administrator with the Keycloak bootstrap-admin command.

Creating a temporary administrator

Keycloak requires every node to be stopped before this command runs, so the procedure involves downtime.

1. Scale the Keycloak workload to zero replicas.

2. Run bootstrap-admin in a pod using the same image and the same database configuration as the deployment:

   /opt/keycloak/bin/kc.sh bootstrap-admin user --username TEMP_ADMIN --password:env PASS_VAR

3. Scale the Keycloak workload back up.

4. Sign in as the temporary account, reset the admin password, then delete the temporary account.

The binary path varies by image. Container images built on Bitnami charts place it at /opt/bitnami/keycloak/bin/kc.sh.

Changing the Default Organization Administrator Password

The platform creates a default organization administrator account during installation using a built-in password that is identical across every deployment. Change it immediately after installation.

1. Open the Keycloak Admin Console and select the dynamo-ai realm.
2. Select Users, choose the default organization administrator, then the Credentials tab.
3. Select Reset password, enter a new value, and set Temporary to off.

This account has no secret store key and no configuration override. A rebuilt realm recreates it with the built-in password, so repeat this step after any procedure that reimports the realm.

Verifying a Rotation

Confirm the credential works before closing the change, because a rotation that only updated the secret store produces no error at any layer.

Credential	Verification
Keycloak admin password	Sign in to the Admin Console in a new private browsing window
Platform user password	Sign in to the platform user interface
Platform API key	curl -H "Authorization: Bearer API_KEY" https://API_HOSTNAME/v1/moderation/policy returns 200
Keycloak client secret	API pods reach a ready state after the restart

Keycloak records the reason for every failed sign-in. Read it before assuming the password is wrong:

kubectl -n NAMESPACE logs KEYCLOAK_POD --since=15m | grep -E "LOGIN_ERROR|invalid_user_credentials|expired_code"

An invalid_user_credentials entry means the password does not match. An expired_code entry means the authentication session was lost, which points at hostname or cookie configuration rather than the credential.