Cost of Running DynamoGuard

Running DynamoGuard costs three things, each sized on its own:

Cost	What it pays for	What drives it
Inference	GPU nodes that run the policies on live traffic	Traffic, prompt length, and how many policies are deployed
Training	GPU time to train custom policies, plus LLM tokens to generate their training data	How often policies are created or retrained
Platform	CPU nodes for the API, UI and core services	Fixed per environment

This page explains how each is sized. To compute a deployment, use the DynamoGuard Cost Calculator, which applies the same model.

Inference

What a GPU Can Do

One T4 has two limits, and both apply at the same time:

• Work per second: about 90 standard policy checks per second at 90% utilization. A PII check costs about 7 standard checks, so a T4 that runs only PII handles about 13 PII checks per second. Every check, from any policy on the card, draws on this shared budget.
• Policies held in memory: each deployed policy is a model server that stays loaded on a GPU whether or not requests use it. A T4 holds up to 11 on Standard_NC8as_T4_v3, and 7 on Standard_NC4as_T4_v3, whose 4 vCPU are the limit.

A GPU therefore needs to be both big enough for the policies deployed on it and fast enough for the checks they receive.

Two Counts

• n, the policies each request runs, not counting PII. This sets the traffic load.
• N, every policy deployed in the environment, across all use cases, including PII and any replicas. This sets GPU memory, because each deployed policy stays loaded whether or not a request uses it.

Custom policies count exactly like built-in ones in both n and N; each runs as its own model server. PII is counted separately only because a PII check is about 7 times heavier than a standard one.

The Formula

For one environment on T4 GPUs, with the recommended configuration. PII is 1 if requests run PII and 0 if not.

Inference GPUs	Formula
GPUs required, with the PII model on its own GPU	ceil(QPS × n ÷ 90) + PII × ceil(QPS ÷ 13)
GPUs required, with the PII model sharing a GPU	ceil(QPS × (n + 7 × PII) ÷ 65)
Memory floor	ceil(N ÷ 7) on Standard_NC4as_T4_v3, or ceil(N ÷ 11) on Standard_NC8as_T4_v3
Inference GPUs per environment	The lower of the first two rows, and at least the memory floor

The sharing row only wins at low traffic, when PII and a few policies fit on one card below 65% use; at higher traffic PII on its own GPU needs fewer GPUs. The calculator picks the cheaper of the two.

Examples

Small: 4 policies, no PII, 5 QPS. ceil(5 × 4 ÷ 90) = 1 GPU, and the memory floor is ceil(4 ÷ 7) = 1. One Standard_NC4as_T4_v3, about $4,608 a year in East US.

Medium: 5 policies including one PII, 20 QPS. ceil(20 × 4 ÷ 90) + ceil(20 ÷ 13) = 3 GPUs per environment, about $13,823 a year on Standard_NC4as_T4_v3.

At scale: 40 policies deployed, 15 QPS. Here there are two costs to think about:

1. Running the policies each request uses. If each use case's requests run 3 policies plus PII, traffic needs ceil(15 × 3 ÷ 90) + ceil(15 ÷ 13) = 3 GPUs.
2. Keeping every policy loaded. All 40 policies stay in GPU memory: ceil(40 ÷ 7) = 6 Standard_NC4as_T4_v3 or ceil(40 ÷ 11) = 4 Standard_NC8as_T4_v3.

Keeping the policies loaded decides the count: 4 Standard_NC8as_T4_v3 nodes, about $26,350 a year, against $27,647 for 6 Standard_NC4as_T4_v3. With many use cases, the number of deployed policies matters more than traffic, so consolidating overlapping policies and deploying to each region only the policies it uses lower cost the most. In the calculator, enter the deployed counts in the policy fields and the per-request count in Input policies per request.

The Full Formula

For planning across several use cases, with output scanning and varying prompt lengths, use the full form. It reduces to the formula above for a single use case that scans input only.

Policy GPUs  = max( ceil( Σ_u QPS_u × (in_u + chunks × out_u) × cost_u ÷ checks ),
                    ceil( deployed_policies × replicas ÷ slots ) )
PII GPUs     = ceil( Σ_u QPS_u × pii_u × cost_u ÷ pii_checks )
Serving GPUs = Policy GPUs + PII GPUs + spare

• u: one use case, with its peak traffic QPS_u.
• in_u and out_u: the input and output policies that one request of that use case runs, without PII.
• chunks: the number of checks for one streamed response (about 3). It is 0 when the use case does not scan output.
• cost_u: the mean of max(tokens, 150) ÷ 150 for the requests of that use case.
• pii_u: 1 when the use case runs PII, and 0 when it does not.
• deployed_policies: all policies in the environment, also the policies that get no traffic.
• slots: 7 on Standard_NC4as_T4_v3 and 11 on Standard_NC8as_T4_v3.
• checks and pii_checks: the standard checks (90) and the PII checks (13) that one T4 does each second at 150 tokens and 90% use.
• spare: 1 for each production environment.

Cost = Σ over environments of (serving GPUs × node price + platform CPU) + training in one environment.

The full formula always gives PII its own GPU, so for very small deployments it can need one GPU more than the sharing row above.

Where the Numbers Come From

The T4 figures come from the October 2026 load test on Azure East US (see Measured Load Test). The planning figures keep headroom below the measured limit of about 100 checks per second per T4.

Training

Training runs only when a policy is created or retrained, never on live traffic. A new policy trains once. Retraining happens when users apply feedback or recreate the policy. Inference never waits on training.

Each training run takes one whole GPU with at least 10 GB of memory for about 43 minutes on a T4, plus LLM tokens to generate its training data: about $2.68 per policy in total, most of it the LLM tokens. Training GPUs do not need to be in every environment: train in one, such as dev, and promote policies. A training GPU kept on all month costs a full month; one that scales to zero costs only the hours it trains.

Platform

The API, UI, Keycloak and other core services run on CPU nodes, sized as 16 vCPU Standard_F16s_v2 nodes: about $11,861 a year per environment at 32 vCPU. Platform cost does not grow with policies or traffic.

Putting It Together

Cost per Environment

Cost	How to calculate it
Inference	Inference GPUs per environment × node price
Training	Policies trained × cost per policy, in one environment; or a training GPU's full month if kept on
Platform	Platform CPU nodes, about $11,861 a year per environment

High availability

An HA secondary is a second environment: it repeats inference and platform cost. Training is not repeated. A warm standby that runs smaller than the primary costs less; size it for the traffic it must carry on failover.

Base Cost of an Environment

Every environment pays for its platform and for keeping its deployed policies loaded, before it serves any traffic. An environment with 5 standard policies, 3 custom policies and PII deployed and no traffic needs 1 Standard_NC8as_T4_v3 for memory plus the platform: about $18,449 a year, 64% of it platform. With an HA secondary, about $36,897 a year. To keep this low, deploy to each region only the policies its use cases run, and add policies as those use cases go live.

How the Three Costs Compare

For an environment with 10 use cases of 3 custom policies each plus PII (31 deployed), 5 QPS of traffic, and 4 policies trained a month:

Cost	Per year	Share
Inference (3 Standard_NC8as_T4_v3)	$19,763	62%
Platform	$11,861	37%
Training (GPU hours plus data generation)	$129	under 1%
Total	$31,752

The 3 GPUs are needed to hold the 31 policies; traffic uses only part of them, so the cost is the same at 15 QPS. A training GPU kept on all month would add about $4,608 a year.

Rules of Thumb

• Marginal cost of traffic: about $6,588 × (n ÷ 90 + PII ÷ 13) per QPS per year per environment, once traffic rather than memory decides the GPU count. Cost rises a whole GPU at a time, so budget with the formula.
• Floor: an environment never costs less than its platform plus the GPUs that hold its deployed policies.

Recommended Configuration

The formula and the calculator assume these settings. Default chart settings need more nodes.

• Node type: Standard_NC4as_T4_v3 for up to about 7 deployed policies per GPU; Standard_NC8as_T4_v3 when more policies share a GPU. Both have one T4.
• Time-slicing at 7 per GPU on NC4as, 11 on NC8as. A twelfth server runs out of GPU memory while loading. Use time-slicing, not MPS: MPS caps each server at its share of the card and was 23-49% slower.
• Right-sized requests. Input guardrails: 0.5 vCPU, 2 GiB memory, 3 GiB ephemeral storage. PII: 1 vCPU, 2 GiB, 2 GiB. The chart's defaults (0.6 vCPU, 4 GiB, 20 GiB ephemeral storage) cap a node at 5 servers by disk alone.
• PII on its own GPU node pool. Point the PII model class's node selector at its own pool. A time-sliced card gives PII a fair share of its time rather than the time it needs, and a second PII replica on the same GPU adds only about 10%.
• livenessProbe.initialDelaySeconds: 300 on the model classes. When a node is replaced, every server on it loads at once and can outlast the default 60 seconds. The operator does not apply startupProbe.

Custom Policies

Each custom policy runs as its own model server: an input policy on the input-mini model class, an output policy on output-mini. Output custom policies deploy only if the output-mini model class is enabled in the DynamoGuard values. On DynamoAI 3.26, a custom policy deployed from the UI runs 1 replica; if you need more, patch spec.replicas on that policy's Model.

Inference in Detail

The calculator works in tokens, which also covers prompt lengths and streamed output:

scans per second   = peak requests/s × policies per request          (input)
                   + peak requests/s × chunks × output policies      (output)
GPU load           = scans per second × tokens per scan ÷ cost factor
GPUs               = GPU load ÷ (card tokens/s × utilization target)
                     or, if fewer: PII load and the rest on separate GPUs, each at the no-PII target
nodes              = the larger of: GPUs needed, or deployed model servers ÷ servers per node
inference $/month  = nodes × $/hour × 730

GPU	Card throughput (tokens/s)	PII cost factor	Source
T4	15,000	7× an input scan	Measured, Azure East US, October 2026
A100 80GB	40,500	7×	Measured, Azure East US, October 2026
A10, half card	21,600	5×	Measured, Azure East US, 3.26.11
A10, whole card	43,200	5×	Measured, Azure East US, 3.26.11

• The A100 is sized at its 150-token figure. It scales better than this model at long prompts (see Measured Load Test), so a deployment with long scans is sized conservatively on A100.
• Tokens per scan. Enter what the guardrails scan, not the whole conversation. Scans shorter than 150 tokens are sized as 150.
• Streamed output. Output guardrails scan a streamed response once per chunk, each sized at the chunk's tokens or 150, whichever is larger. Scanning output in about 3 chunks per response roughly triples the checks.
• Latency grows with busy servers on a card: p95 at 150 tokens rose from 29 ms with one busy server to 252 ms with 11.

Measured Load Test

Azure East US, October 2026. A configuration passes when p95 latency is at most 3 seconds with under 1% errors. Prompts are 150 tokens unless stated. Latency stays flat (p95 about 100 to 250 ms) below about 80% of capacity, rises to 0.3 to 1 second between 80% and 100%, and requests time out above 100%.

GPU node (Azure)	Policy checks per second per GPU	PII requests per second per GPU	Model servers per GPU
Standard_NC4as_T4_v3	About 100 at 150 tokens (about 15,000 tokens/s), about 28 at 500	15 at 150 tokens, 6 at 350, 4 at 500, 2 at 1,000	7 (vCPU-bound)
Standard_NC8as_T4_v3	The same as Standard_NC4as_T4_v3	The same	11 (GPU memory, about 1.3 GB each)
Standard_NC24ads_A100_v4 (A100 80GB), time-sliced	About 270 to 300 at 150 tokens	40 at 150 tokens (ceiling about 53), 8 at 1,000	30 or more
Standard_NC24ads_A100_v4, 7 MIG 1g.10gb partitions	About 410 at 150 tokens, one server per partition; no gain over time-slicing at 1,000 tokens	Not measured	7 at most

On a T4, PII runs on its own GPU. On an A100, 4 built-in guardrails plus PII on one card pass 30, 12 and 6 requests/s at 150, 500 and 1,000 tokens. The calculator sizes the A100 at 270 checks per second and does not model MIG.

Measured Inference Hardware

Requests/s are for 4 input guardrails plus PII at 200 tokens on DynamoAI 3.26.11, with p95 latency under 1 second.

GPU node (Azure)	GPU memory	Requests/s per node	Model servers per node
Standard_NV6ads_A10_v5 (1/6 A10)	4 GB	Not viable	Three guardrails do not fit
Standard_NC4as_T4_v3	16 GB	7	Up to 7 (vCPU-bound)
Standard_NC8as_T4_v3	16 GB	7	Up to 11 (GPU memory)
Standard_NV12ads_A10_v5 (1/3 A10)	8 GB	10	Up to 5
Standard_NV18ads_A10_v5 (1/2 A10)	12 GB	12	Up to 8
Standard_NV36ads_A10_v5 (1 A10)	24 GB	24	High request rates

Measured Training Hardware

GPU node (Azure)	GPU memory	GPU time per policy	Can train
Standard_NV12ads_A10_v5 (1/3 A10)	8 GB	—	No, runs out of memory
Standard_NC4as_T4_v3	16 GB	43 minutes	Yes
Standard_NV18ads_A10_v5 (1/2 A10)	12 GB	14 minutes	Yes, smallest A10 that can
Standard_NV36ads_A10_v5 (1 A10)	24 GB	10 minutes	Yes

• Keep training off the inference GPUs. Training on a GPU that also served guardrails failed 71% of requests at 5 requests/s on a T4.
• Upgrades interrupt training. Upgrading the DynamoGuard chart restarts the training job definition, which stops a training in progress.

Using Other Clouds

Throughput transfers per GPU model: a T4 does the same work on any cloud. Price does not. To cost a deployment on another cloud, take the node counts for the same GPU model from this page, price the equivalent instance from that provider's price list, and add the provider's node overhead. Price per GPU, because some providers sell A100 only in multi-GPU instances. The calculator lists AWS and Google Cloud prices in its data, but offers a cloud only once one of its GPUs is measured there.

Quota

Azure grants GPU quota as vCPUs per VM family, per region. The calculator lists, for each region, inference and training nodes at peak, one spare node per GPU pool, and the platform CPU nodes; Total Regional vCPUs must cover all of it. An HA secondary in another region needs its own quota there. Compare each line with your current limits:

az vm list-usage --location canadacentral --output table

Limits

• Prices are Azure pay-as-you-go (on-demand Linux) list prices from the Azure Retail Prices API for the tested VM sizes, refreshed weekly. The load tests ran in East US; the calculator prices the region you pick. Reserved instances, savings plans and committed-use discounts lower them.
• The calculator offers only SKUs measured end to end: T4 (NC4as, NC8as), A100 80GB (NC24ads) and A10 (whole, 1/2, 1/3 and 1/6 slices), all on Azure. The data also holds estimates for A10G, A100 40GB, L40S and H100, and an earlier L4 benchmark; none is offered until it is measured.
• Costs exclude storage and networking. The GPU count excludes hallucination guardrails, which use an external model endpoint by default.
• Treat results as a starting point and validate with a load test using your real prompt lengths.