Verifying A Standby
Applies to DynamoGuard. The platform prerequisites a standby also needs are in Platform Reliability.
A standby that reports healthy and cannot enforce policy is worse than one that is visibly down. Nothing alerts, and the gap surfaces during a failover. Every check here exists because the obvious signal said a region was fine when it was not.
Run these after standing a region up, and again before recording a pair as ready.
Checks
| # | Check | Passing looks like |
|---|---|---|
| 1 | Artifacts are present under the prefix the model resources reference | Object listing under that exact prefix is non-empty |
| 2 | Every guardrail model pod is Ready | Replica count matches, no restart loop |
| 3 | Identity is serving | The identity pod is Ready, not merely Running |
| 4 | Metrics reach a store you can query | A query for the guardrail series returns data |
| 5 | A real analyze request returns the verdict you expect | A known-bad prompt is blocked |
Check 5 is the only one that proves enforcement. The first four are prerequisites.
What each check catches
1. Artifact sync reports success and writes to the wrong place
An artifact sync reports how many objects it transferred, not whether the model servers can find them. Model resources load from a path that includes a prefix, and a sync configured without that prefix completes cleanly, reports every object transferred, and leaves the servers looking at an empty location.
Verify the objects exist at the exact path the model resources name. A non-empty bucket proves nothing:
kubectl get models.apps.dynamo.ai -n <namespace> \
-o custom-columns='NAME:.metadata.name,PATH:.spec.path'
Then list that prefix in the object store and confirm the weights are under it.
2. A model pod can be serving long before it is used
A pod is withheld from traffic until its readiness delay expires, whatever its model finished
loading. Where the delay is tuned for the slowest model, every policy recovers at that pace.
Compare the time the container logged that its model initialised against the time the pod became
Ready. A large gap is configuration, not load time. See
Configuration Requirements, A5.
3. A summary that hides which check failed
Health endpoints aggregate several checks, and a composite DOWN can sit beside a database
check reporting UP. Read the individual checks. The summary tells you something failed, not
which one. A standby whose
identity component reports its database as reachable and its cluster as unreachable will never
become ready, and the failure reads as a database problem.
This is the shape to watch when a standby shares a database with the primary: components that discover their peers through that shared database will find the other region and try to reach it directly. If there is no network path between the two, that discovery never completes.
4. Telemetry can be wired at both ends and discarded in the middle
Exporters gated on an endpoint go quiet when it is unset, and an empty series is indistinguishable from an idle guardrail path. Setting the endpoint is necessary and not sufficient: a collector whose pipeline exports only to a debug destination receives every metric and drops it.
Verify at the far end. Query the metric store for one of the guardrail series and confirm data
comes back. If the collector is scraped by Prometheus, confirm the target reports up. A
configured scrape is not a working one. See Monitoring And Targets.
5. Deployment status is recorded, not observed
A policy is marked deployed when its model resource is submitted, not when a pod is serving it. Where two deployments share a document store, the second renders the first's status and shows policies as trained and deployed while holding neither the artifact nor the workload.
A policy list is not a parity check. Compare what each region can serve by sending a request. What a region displays is a different fact.
Symptoms
| Symptom | Usual cause |
|---|---|
| Model servers restart continuously | Artifacts absent, or present under the wrong prefix |
| A policy is unavailable for minutes after a restart | Readiness delay, not model load |
| Identity never becomes ready | Peer discovery reaching the other region across a path that does not exist |
| Guardrail metrics are empty | Export unconfigured, or the collector retains nothing |
| The region lists policies it cannot serve | Status recorded at submission; artifacts and workloads are regional |