Monitoring And Targets
Applies to every DynamoAI product. The guardrail path metrics are DynamoGuard specific.
Signals To Collect
The platform runs as standard Kubernetes workloads. Liveness and readiness probes drive in-region recovery, pod and container metrics reach whatever monitoring stack the cluster runs, and application logs are written to standard output for collection.
For a paired-region deployment, collect these five signals in both regions.
| Signal | Why it matters | Source |
|---|---|---|
| Regional health probe result | Drives automatic withdrawal from routing | Your global load balancer |
| Replica distribution across zones | A deployment configured for zone spread can still be co-located | kubectl get pods -o wide compared against node zone labels |
| Datastore replication lag | Sets the recovery point at the moment of promotion | Your managed datastore metrics |
| Guardrail pod readiness per policy | A policy without a ready pod fails every request naming it | Kubernetes readiness state |
| Enabled and trained policy count per region | Parity between primary and standby. A raw policy count is not a parity check on its own, because policies differ by enabled state and training status | Platform API, compared between regions |
Guardrail Path Metrics
The API and the moderation worker both export OpenTelemetry metrics on the guardrail path.
| Metric | Source | Labels |
|---|---|---|
dynamoguard_successful_requests | API, analyze and chat endpoints only | modelId |
dynamoguard_failed_requests | API, analyze and chat endpoints only | modelId, failureType (HTTP status) |
dynamoguard_e2e_latency | API, analyze and chat endpoints only, milliseconds | modelId |
dynamoguard_moderation_<guardrail>_num_requests | Moderation worker, one counter per built-in guardrail type | None |
The worker counters cover the built-in guardrail types: PII redaction, input toxicity, output toxicity, RAG hallucination, input relevance, response relevance, prompt injection, legal advice and financial advice. They count by guardrail type, not by policy, so they do not give you a per-policy request rate for custom policies.
Both exporters are gated on configuration. The moderation worker logs a warning and exports nothing when its OTLP metrics endpoint is unset. An empty series therefore means the export is unconfigured, not that the guardrail path is idle. Confirm the endpoint is set in both regions before treating these metrics as a failover signal.
Configuring Metric Export
Set these on both the API and the moderation worker, in every region. Without them the guardrail metrics above are never produced.
| Variable | Component | Carries |
|---|---|---|
OTEL_APPLICATION_METRICS_ENDPOINT | API and moderation worker | The dynamoguard_* series and the per-guardrail counters |
OTEL_METRICS_ENDPOINT | API | Runtime and system metrics |
OTEL_APPLICATION_TRACES_ENDPOINT | API | Traces |
Point them at an OTLP collector reachable from the cluster:
OTEL_APPLICATION_METRICS_ENDPOINT=http://<collector>:4318/v1/metrics
OTEL_METRICS_ENDPOINT=http://<collector>:4318/v1/metrics
OTEL_APPLICATION_TRACES_ENDPOINT=http://<collector>:4318/v1/traces
Both components use the OTLP HTTP exporter. Give each variable the full signal path on the
HTTP port, such as http://collector:4318/v1/metrics. A gRPC-style endpoint like
collector:4317, or a bare host with no path, does not report an error. It exports nothing, and
the result is indistinguishable from a guardrail path that is simply idle.
Recovery Objectives
The platform is deployed into infrastructure you operate: your clusters, node pools, datastores, network and identity provider. Availability is a property of that deployment, so DynamoAI does not publish an availability percentage for it.
The configuration is designed to meet these objectives.
| Objective | Design target |
|---|---|
| Recovery point, synchronously replicated data | No loss of acknowledged writes |
| Recovery point, asynchronously replicated data | Replication lag at the moment of promotion |
| Recovery time, failure within a region | Minutes, automatic |
| Recovery time, regional failover | Tens of minutes, dominated by model load unless the standby holds GPU capacity |
Every figure above is a budget until a drill has measured it in your environment.
Drills
Exercise the procedure on a schedule and record the measured recovery time.
At minimum, run a datastore promotion and a full platform rebuild against restored data. In practice the configuration parity check fails more often than the restore does, which makes it the part of the drill worth repeating.
# Replica placement in each region
kubectl --context <ctx> -n <ns> get pods \
-o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName'
# Zone of each node
kubectl --context <ctx> get nodes \
-o custom-columns='NODE:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone'
# Disruption budgets permit at least one disruption
kubectl --context <ctx> -n <ns> get pdb
Compare the policy list returned by each region before recording the pair as ready.