Platform Reliability

Scope

Applies to every DynamoAI product. For behaviour specific to guardrail serving and policy enforcement, see DynamoGuard Reliability.

High availability keeps the platform serving through failures inside a single region. Automatic controls handle recovery: replicas absorb the loss of a pod or node, disruption budgets prevent maintenance from removing an entire service, and topology constraints keep replicas in separate availability zones.

This page covers a single region. For regional outages, see Disaster Recovery.

Component behavior

Component	Redundancy	On failure
API	Multiple replicas across zones	Traffic shifts to surviving replicas
UI	Multiple replicas across zones	Traffic shifts to surviving replicas
Guardrail serving	One replica per policy by default; built-in models accept more	A policy running one replica is unavailable until rescheduled
Evaluation workers	Scale from zero on demand	In-flight jobs are lost and must be restarted by an operator; completed stages are checkpointed and skipped
PostgreSQL	Customer-managed HA	Automatic failover to the standby
MongoDB	Replica set across zones	Automatic election, no data loss with majority writes
Redis	Customer-managed HA	Automatic failover; queued work is discarded
Object storage	Erasure coding across drives in distributed mode	Automatic rebuild, see Object Storage

Guardrail serving runs one model-server deployment per policy, and whether it uses a CPU or a GPU is set by the model class. Losing the only pod for a policy makes that policy unavailable until the replacement reschedules and loads its model, which takes several minutes and up to fifteen for large models.

Built-in guardrail models accept a replica count. Set replicas on the model to run more than one. A PodDisruptionBudget is created automatically once a model runs two or more replicas, so no separate budget is required.

models:
  input-mini:
    replicas: 2

important

Two constraints apply to this.

Custom policy models run a single replica. The platform manages their lifecycle and sets the replica count itself, so a custom policy is unavailable for the duration of any restart of its model. This is a current release limitation, listed in Configuration Requirements.

Guardrail models cannot express topology spread or pod anti-affinity. Extra replicas may be scheduled into the same zone, or onto the same node. Verify placement after deployment, and constrain scheduling through node pools where zone separation matters.

Failure scenarios

Pod failure

Kubernetes reschedules the pod. Surviving replicas absorb traffic. No data loss, no manual action.

For guardrail pods, the affected policy is unavailable until the replacement loads its model. A request that names that policy fails as a whole, including the other policies in the same request.

The platform does not decide what happens to unchecked content

The API returns an error rather than a verdict, and the platform has no fail-open or fail-closed setting. Whether the content then proceeds unchecked is decided by the calling application, in how it handles that error. Configure the calling application to treat an error from the analyze endpoint as a block wherever unchecked content is unacceptable.

A missing policy behaves differently again: the request returns a normal success with fewer checks applied. See Traffic Routing.

Node failure

Pods reschedule onto surviving nodes, provided capacity exists. Workloads with zonal volumes reschedule within their own zone.

Disruption budgets ensure a planned drain removes only one replica at a time. A budget whose minAvailable equals the replica count permits zero disruptions and blocks drains entirely.

Availability zone failure

Replicas in surviving zones continue serving. Managed datastores fail over automatically.

Erasure-coded object storage keeps serving reads, and keeps accepting writes only while the surviving drives still meet the write quorum. A four-drive deployment that loses a zone holding two instances becomes read-only until capacity returns. Six drives across three zones keep the store writable through the loss of one zone. See Object Storage Availability for the drive counts.

Recovery depends on spare capacity in the surviving zones. Without it, rescheduled workloads stay pending. This is the most common reason a correctly configured deployment still fails a zone test.

Regional outage

Not covered by high availability. See Disaster Recovery.