Reliability Overview
DynamoAI separates resilience into two strategies. High availability (HA) uses automatic controls to keep the platform serving through component, node, and availability zone failures within a region. Disaster recovery (DR) uses manual procedures to recover the platform when automatic failover cannot help, such as the loss of an entire region or the deletion of production data.
Prefer high availability where possible. Use disaster recovery for the failure scenarios high availability does not cover, not as a substitute for it.
Regional failover uses both. Detection and withdrawal of a failed region are performed by the global load balancer you operate, not by the platform, and what a passing probe establishes depends on which health endpoint you point it at. See Traffic Routing. Promotion of the datastores and the decision to cut traffic over are operator actions with a documented time budget.
Which strategy covers which failure
| Failure scenario | Strategy | Typical recovery | Data loss | Automatic |
|---|---|---|---|---|
| Pod or container failure | HA | Seconds; several minutes for a guardrail policy while its replacement loads the model | None | Yes |
| Node failure | HA | Seconds to minutes | None | Yes |
| Availability zone failure | HA | Minutes | None | Yes |
| Datastore node failure | HA | Seconds to two minutes | None for PostgreSQL and MongoDB; queued background work for Redis | Yes |
| Storage volume failure | HA | Automatic rebuild, distributed mode only | None | Yes |
| Regional outage: detection and traffic withdrawal | HA | Minutes | None | Yes, by the load balancer you operate |
| Regional outage: datastore promotion, parity check, cutover | DR | Tens of minutes to hours, measured by drill | Up to replication lag | No, by design |
| Deletion of production data | DR | Hours | Since last backup | No |
| Data corruption | DR | Hours | Since last backup | No |
| Cluster loss | DR | Hours | None for customer-managed datastores; total for a bundled object store without an export | No |
Redundancy does not cover the last four rows. Replication and erasure coding reproduce a deletion as faithfully as they reproduce a write. Only a backup recovers data that was destroyed.
Shared responsibility
DynamoAI deploys into infrastructure you operate. Availability is shared, and the boundary matters when you assess coverage.
| Layer | Responsibility |
|---|---|
| Kubernetes control plane | You |
| Node pools, availability zones, capacity | You |
| PostgreSQL, MongoDB, Redis | You |
| Object storage | Shared when using the bundled store: you provide nodes, zones, storage class and quota; DynamoAI provides the deployment |
| Ingress, DNS, certificates | You |
| Platform workloads and replica configuration | DynamoAI |
| Guardrail model serving | DynamoAI |
Datastore high availability is a property of the services you provision. The platform connects to whatever you configure. See High Availability for the requirements each store must meet.
Service tiers
Recovery objectives depend on deployment topology. A tier is a set of deployment choices made together; no single setting selects one.
| Bronze | Gold | |
|---|---|---|
| Survives | Pod, node, zone and datastore-node failure | Everything in Bronze, plus a regional outage through the paired region |
| Recovery time objective | Minutes, automatic | Tens of minutes, measured by drill |
| Recovery point objective | None for acknowledged writes | Replication lag of the asynchronous stores |
| Topology | Single cluster, multiple zones | Paired clusters behind a gateway |
| Compute redundancy | Multiple replicas across zones | Multiple replicas across zones and clusters |
| Guardrail serving | Multiple replicas per built-in model; custom policy models run one | Everything in Bronze, across clusters |
| Object storage | Distributed, multi-zone | Distributed, plus geo-redundant export |
| Failover | Within region, automatic | Cross-region: withdrawal by your load balancer, promotion and cutover by an operator |
| Availability | Current release | Targeted, see release notes |
Recovery objectives are design objectives until a drill has measured them. DynamoAI does not publish an availability percentage for a deployment that runs on infrastructure you operate.
How This Section Is Organised
Reliability splits three ways: what the platform provides for every product, what DynamoGuard adds on top, and what changes when a deployment spans two regions. Cloud-specific commands and tier names are kept out of all three.
| Section | Scope | Covers |
|---|---|---|
| Platform | All products | Component behaviour, prerequisites, datastores, object storage, disaster recovery |
| DynamoGuard | DynamoGuard | Guardrail serving, what a moderation request returns on failure, known limitations |
| Paired Regions | All products | Two regions in one geography: routing, replication, recovery objectives |
| Cloud Reference | Per cloud | The commands and tier names that satisfy the requirements above |
Start with Platform. A DynamoGuard policy cannot be more available than the API, datastores and object storage beneath it.