Data Replication
Applies to every DynamoAI product.
What Replicates Across Regions
| Data | Mechanism | Replication | Recovery point | Failover |
|---|---|---|---|---|
| Policy and guardrail configuration | MongoDB replica set, majority writes | Synchronous to a majority | Zero while a majority survives | Automatic while a majority survives |
| Platform relational data | Application PostgreSQL cross-region read replica | Asynchronous | Replication lag | Operator promotion |
| Identity data | Keycloak PostgreSQL cross-region read replica | Asynchronous | Replication lag | Operator promotion |
| Moderation audit records | Follows PostgreSQL | Asynchronous | Replication lag | Unavailable until promotion |
| Task queue and in-flight jobs | Rebuilt in the standby | Not replicated | Work in flight is lost | Operator restarts the affected jobs |
| Model artifacts | Scheduled export to the standby | Asynchronous | Export interval | Present, then loaded |
| Identity realm configuration | Deployed by the chart | Not replicated | Not applicable | Redeploy |
Recovery point and recovery time definitions, and the reason a replica cannot substitute for a backup, are covered in Disaster Recovery.
Replica Set Quorum
A MongoDB replica set elects a primary while a majority of voting members is reachable. This rule decides whether the document store recovers on its own after a region is lost.
| Topology | Region lost | Result |
|---|---|---|
| Three regions, one member each | Any one | Two of three voters remain. Automatic election, no loss of acknowledged writes |
| Two regions, two members and one | The minority region | Two of three voters remain. Automatic election |
| Two regions, two members and one | The majority region | One of three voters remains. No election, no primary, and writes stop until an operator restores quorum |
A geography with two regions cannot meet the quorum requirement for automatic failover across regions. Either place a third voting member in a separate failure domain, or record in the recovery plan that losing one region requires manual reconfiguration before writes resume.
Node counts must be odd, with a maximum of seven electable members. Region priority determines which region is preferred for the primary. Member count does not.
PostgreSQL
Use a cross-region read replica with a virtual endpoint. The endpoint follows promotion, so the connection string the platform uses survives the failover. Without one, promotion produces a new host and the platform has to be reconfigured during the incident.
A promoted replica runs without high availability and without geo-redundant backup until one is re-enabled and the other rebuilt. Plan for that posture during the recovery window.
Geo-restore is the alternative. It is simpler to set up and produces a new server, which means reconfiguring the platform mid-incident.
The platform runs two separate PostgreSQL databases, one for application data and one for Keycloak. Both are checked independently by the API readiness probe, and both need a cross-region replica. Replicating one and not the other leaves the standby able to serve no traffic: platform data without identity cannot authenticate anyone, and identity without platform data has nothing to authorize against.
Task Queue
Redis carries the task queue, job progress, and the stage-completion map that evaluation restarts consult. The standby rebuilds it. Two constraints rule out replication: active-active list delivery is at-least-once, which on a task queue causes duplicate execution, and active geo-replication is mutually exclusive with persistence.
Jobs in flight at the moment of failover are lost. Restarting them is a manual step in the current release. Nothing re-queues them, so an operator has to restart each affected job. A restarted job resumes from its last checkpoint rather than from the beginning. Daily request rate-limit counters also reset.