Disaster Recovery
Applies to every DynamoAI product. Tier and feature names are Azure; see Cloud Reference for equivalents.
Disaster recovery is a resilience strategy that uses manual procedures to recover a deployment when automatic failover cannot help. It covers regional outages, data deletion, and data corruption.
Prefer high availability where possible. Use disaster recovery for the scenarios high availability does not cover, not as a substitute for it. See High Availability for the automatic controls.
Recovery objectives by datastore
Recovery point objective (RPO) is the maximum data loss. Recovery time objective (RTO) is the time to restore service. Both depend on what you configure.
| Data | Mechanism | RPO | Notes |
|---|---|---|---|
| MongoDB | Cross-region replica set nodes | Near zero while a majority survives | Majority writes acknowledge only when a majority holds the write. In a two-region geography, loss of the majority region needs manual reconfiguration before writes resume |
| MongoDB | Continuous backup with point-in-time recovery | Minutes | Restore targets a cluster you choose |
| PostgreSQL | Cross-region read replica with a virtual endpoint | Minutes | The endpoint follows promotion, so the connection string does not change |
| PostgreSQL | Geo-redundant backup and geo-restore | Up to one hour | Restore creates a new server; move the virtual endpoint to it |
| Object storage | Scheduled export to geo-redundant storage | Export interval | No export mechanism ships in the current release; see Object Storage |
| Redis | Rebuild | Not applicable | Queued and in-flight work is discarded; an operator restarts the affected jobs |
| Model artifacts | Re-seed by hand into the models bucket | Not applicable | Derived, not primary data; the chart does not create or populate the bucket |
Replication is not backup. A replica reproduces a deletion or a corruption as faithfully as it reproduces a write. Recovering from either requires a backup that the primary cannot reach.
Selecting a PostgreSQL mechanism
A cross-region read replica with a virtual endpoint is the only PostgreSQL failover path where the application connection string survives promotion. Geo-restore is simpler but produces a new server, which means reconfiguring the platform during the incident.
Geo-redundant backup can only be enabled at server creation. Decide before provisioning.
A promoted replica runs without high availability and without geo-redundant backup until you re-enable one and rebuild the other. Plan for that posture during the recovery window.
Redis is rebuilt, not replicated
Do not use active geo-replication for Redis. Redis carries the platform task queue, and with active-active replication a list element is delivered at least once and may be delivered twice. On a task queue that means duplicate execution. Active geo-replication is also mutually exclusive with persistence.
Treat Redis as rebuildable. Jobs in flight at the time of failover are lost and are restarted by an operator.
Recovery time
Recovery time is dominated by bringing guardrail model serving back online, not by data restore.
| Phase | Typical duration |
|---|---|
| Detect and declare | Minutes |
| Promote datastores | Minutes |
| Platform pods ready | Minutes |
| Container image pull | Several minutes |
| Model weights loaded to GPU | 5 to 15 minutes |
The last two rows are most of the total. A standby that keeps GPU capacity provisioned and model weights synchronised removes them. A standby built on demand pays them in full.
These durations are a budget composed from configured readiness ceilings and observed image and weight load times. Treat them as estimates until a drill has measured them.
Failover procedure
In the event of a regional outage, follow these steps:
- Confirm the primary region is unavailable. Failover while the primary is partially alive risks split-brain, particularly for DNS records and datastore promotion.
- Promote the datastore replicas. For PostgreSQL, promote the read replica; the virtual endpoint follows. For MongoDB, confirm the surviving nodes hold a majority. A two-region topology that loses its majority region requires manual intervention to restore quorum.
- Verify configuration parity. The standby serves the policy set present in its database. Confirm the replicated configuration matches what the primary was enforcing before directing traffic.
- Bring up the platform and redeploy policies. Start order matters: object store, promoted datastores, identity, API, operator, then guardrail models. The Helm release recreates the out-of-the-box models; every custom policy must be redeployed from the UI or API in the standby before it is enforced. Confirm each policy reaches a ready state before proceeding.
- Cut traffic to the standby. Update DNS or the routing layer.
- Confirm service. Verify moderation requests succeed and compare the policy list against the primary's last known state. No endpoint reports policy parity in the current release, so this is a manual check.
Once the primary region recovers, reconcile data written to the standby before failing back.
What does not fail over
State these explicitly in any recovery plan.
- Identity configuration. A restored identity database retains the original region's redirect URIs, and login fails until they are corrected. Where an external identity provider was configured by hand and not declared in configuration, rebuilding the realm loses that integration. Keep identity configuration in version control.
- Encryption context. The platform image and its database are a matched pair. A restored database is readable only by an image built with the same encryption context. Record the image digest alongside every backup.
- DNS record ownership. Where DNS records are managed by a controller with per-cluster ownership, a second cluster serving the same hostnames will not publish records while the original owner exists. Confirm the primary is fully stopped, or clear the ownership records, before the standby can take over.
- In-flight work. Evaluation and background jobs running at the moment of failure are lost and must be restarted by an operator.
Failure scenarios
Regional outage
Follow the failover procedure above. Recovery time depends on whether the standby keeps GPU capacity warm.
Deletion of production data
Restore from backup to a point before the deletion. Replication does not help; the deletion replicated too. For MongoDB, point-in-time recovery restores to a timestamp. Snapshot-only recovery restores to the last snapshot, which can be hours earlier.
Data corruption
Identify the earliest known-good point, restore to it, and reconcile writes made after that point. As with deletion, replicas carry the corruption.
Cluster loss
The datastores survive if they are customer-managed and external to the cluster. Reinstall the platform, point it at the existing datastores, re-seed model artifacts into the models bucket, and redeploy custom policies. No data moves, so this is substantially faster than a regional failover.
Loss of the bundled object store
If you run the bundled object store without an export configured, object data is not recoverable. See Object Storage.
Testing
A recovery procedure that has not been executed is an estimate. Test on a schedule, and record the measured RTO.
At minimum, exercise a datastore promotion and a full platform rebuild against restored data. In practice the configuration parity check in step 3 fails more often than the restore.