Dynamo AI Resource Provisioning Guidelines
This reference outlines recommendations on resource provisioning for the DynamoAI platform based on expected feature utilization and workloads. It ensures optimal performance across various use cases and scenarios.
Scaling Considerations
Dynamo AI platform resource recommendations are based on the following metrics:
- Throughput: Number of requests per second
- Guardrails: Number of guardrails applied per moderation request (DynamoGuard)
Throughput Scenarios
Below, we provide resource requirements for different throughput scenarios, ranging from < 1 QPS to 100 QPS. For context, we typically observe production workloads ranging from 0.1 - 10 QPS in our customer's AI use cases. Size higher throughput with the DynamoGuard Cost Calculator.
Example: For an AI use case with 100k global users, a throughput of 10 QPS equates to approximately 8-12 queries per user per day.
| Scenario | Expected Throughput (QPS) | Use Cases |
|---|---|---|
| Development | 1 QPS | Testing environments |
| Small | 5 QPS | Lightweight production scenarios |
| Medium | 10 QPS | Moderate-scale AI application |
| Large | 50 QPS | High-demand production applications |
| Extra Large | 100 QPS | Enterprise-scale, high-performance systems |
Resource Guidelines
General Platform
Base platform resources are used for API and UI servers. The table below outlines recommended configurations:
| Scenario | Recommended Resources | Example Cloud-Specific Details |
|---|---|---|
| Development | x32 vCPUs, 64GB memory | - AWS: x8 c7i.xlarge - Azure: x8 Standard_F4s_v2 - GCP: x8 c2d-standard-4 |
| Small | x32 vCPUs, 64GB memory | - AWS: x8 c7i.xlarge - Azure: x8 Standard_F4s_v2 - GCP: x8 c2d-standard-4 |
| Medium | x64 vCPUs, 128GB memory | - AWS: x16 c7i.xlarge - Azure: x16 Standard_F4s_v2 - GCP: x16 c2d-standard-4 |
| Large | x128 vCPUs, 256GB memory | - AWS: x32 c7i.xlarge - Azure: x32 Standard_F4s_v2 - GCP: x32 c2d-standard-4 |
| Extra Large | x128 vCPUs, 256GB memory | - AWS: x32 c7i.xlarge - Azure: x32 Standard_F4s_v2 - GCP: x32 c2d-standard-4 |
Note: This table is a general reference. You may need less resources than described in this table, because general platform components can run on the GPU nodes, using the shared vCPU and RAM. But provisioning this amount of resources guarantees the performance.
DynamoGuard Content Guardrails
DynamoGuard requires resources based on the number of guardrails applied to a workload. While CPUs can handle limited guardrails with higher latency, GPUs offer significantly improved latency (< 300ms). For lower latency when using CPUs, we recommend utilizing compute-optimized instances. Below, we provide the resource requirements for input content guardrails. For details around output content guardrails, please reach out to our team.
Tip: For non-development workloads, we recommend GPUs due to reduced latency and higher scalability.
Note: Calculate the GPUs you need from your guardrails, traffic and prompt length, and the number of policies you will deploy in the cluster.
| Scenario | CPU Option | GPU Option | Example Cloud-Specific Instances |
|---|---|---|---|
| Development | x8 vCPUs, 8GB memory per guardrail | Same as Small scenario | - AWS: c7i.xlarge - Azure: Standard_F4s_v2 - GCP: c2d-standard-4 |
| Small | Not Recommended | Sized from traffic, guardrails and prompt length with the calculator | - Azure: Standard_NC4as_T4_v3 or Standard_NC8as_T4_v3 (T4, measured) - AWS, GCP: an instance with one T4 (not yet measured there) |
| Medium | Not Recommended | Sized from traffic, guardrails and prompt length with the calculator | - Azure: Standard_NC4as_T4_v3 or Standard_NC8as_T4_v3 (T4, measured) - AWS, GCP: an instance with one T4 (not yet measured there) |
| Large | Not Recommended | Sized from traffic, guardrails and prompt length with the calculator | - Azure: Standard_NC4as_T4_v3 or Standard_NC8as_T4_v3 (T4, measured) - AWS, GCP: an instance with one T4 (not yet measured there) |
| Extra Large | Not Recommended | Sized from traffic, guardrails and prompt length with the calculator | - Azure: Standard_NC4as_T4_v3 or Standard_NC8as_T4_v3 (T4, measured) - AWS, GCP: an instance with one T4 (not yet measured there) |
Data Generation
Data generation is a required step in the custom content policy creation. Dynamo AI supports several external and in-cluster model configurations for data generation. If required, contact Dynamo support for additional model providers.
| Option | Description | Cloud-Specific Details |
|---|---|---|
| Option 1 | Azure Llama 3.1-8B: azure_ai/Meta-Llama-3-1-8B-Instruct | N/A |
| Option 2 | AWS Llama 3.1-8B: bedrock/llama/us.meta.llama3-1-8b-instruct-v1:0 | N/A |
| Option 3 | GCP Llama 3.1-8B: llama-3.1-8b-instruct-maas | N/A |
| Option 4 (less performant) | In-cluster model: x1 A10G GPU + x8 vCPUs | - AWS: x1 g5.2xlarge - Azure: x1 NV36ads_A10_v5 - GCP: x1 g2-standard-8 |
Guardrail Fine-Tuning
Guardrail fine-tuning is a required step in the custom content policy creation. DynamoGuard offers two options for fine-tuning guardrails. Choose between SaaS fine-tuning or in-cluster fine-tuning based on your infrastructure.
| Option | Description | Cloud-Specific Details |
|---|---|---|
| Option 1 | Fine-tune on Dynamo SaaS environment and import policies into your cluster | N/A |
| Option 2 | x1 A10G (or similar) GPU with x8 vCPUs, 32 GB RAM, and 24GB GPU memory | - AWS: x1 g5.2xlarge - Azure: x1 NV36ads_A10_v5 - GCP: x1 g2-standard-8 |
DynamoGuard Hallucination Guardrails
For hallucination guardrails, Dynamo supports both external and in-cluster configurations. If required, contact Dynamo support for additional model providers.
| Option | Description | Cloud-Specific Details |
|---|---|---|
| Option 1 | Azure-Open-AI-GPT-4o or Open-AI-GPT-4o | N/A |
| Option 2 | In-cluster: Requires x3 A10G GPUs at 1 QPS. Additional GPUs scale linearly. | - AWS: x3 g5.2xlarge - Azure: x3 NV36ads_A10_v5 - GCP: x3 g2-standard-8 |
DynamoEval
DynamoEval requires the following resource configurations. The API endpoints are used for the data generation and judgement.
| Requirement | Description | Cloud-Specific Details |
|---|---|---|
| API Endpoints | mistral-small-latest, open-mistral-nemo, Open-AI-GPT-4o | N/A |
| CPU and memory | x1 vCPU, 4GB memory | - AWS: x1 c7i.xlarge - Azure: x1 Standard_F4s_v2 - GCP: x1 c2d-standard-4 |
GPU Requirements by Feature
The table below lists the minimum GPU resources per feature. Requirements may change as models evolve; contact DynamoAI for the most recent figures. In production, auto-scaling based on traffic volume may require more resources.
| Feature | A10, A10g or L4 GPU | A100 GPU (40GB Accelerator-memory) | No GPU |
|---|---|---|---|
| Basic API and UI Servers | None | None | 32 vCPUs and 128GB memory |
| DynamoGuard Input Content Guardrails (i.e. toxicity, "no financial advice", etc.), Prompt Injection Guardrails, PII Guardrails | Sized from traffic, not a fixed count per GPU: see the Cost Model | Sized from traffic: see the Cost Model | 8 vCPU, 16 GB memory per input guardrail |
| DynamoGuard Output Content Guardrails (toxicity, "no financial advice", etc.) | Not validated on measured hardware; contact DynamoAI | Not validated on measured hardware; contact DynamoAI | Not supported |
| DynamoGuard Hallucination Guardrails | Option 1: 2 GPUs to support retrieval relevance + faithfulness capabilities; Option 2: 3 GPUs to support retrieval relevance + faithfulness + response relevance capabilities | Option 1: 1 GPU to support retrieval relevance + faithfulness capabilities; Option 2: 1+1/2 GPUs to support retrieval relevance + faithfulness + response relevance capabilities | You can use third-party model endpoints (e.g., OpenAI or Mistral) to run Hallucination guardrails |
| DynamoGuard Fine-Tuning Custom Content Guardrails (i.e. creating your custom No Financial Advice, No Legal advice, etc. guardrails) | 1 GPU | 1 GPU | You can use Dynamo AI SaaS platform to fine-tune the custom content policies and import the models to your cluster |
| DynamoGuard Synthetic Data Generation for creating Custom Content Guardrails | 1 GPU for lower-quality in-cluster synthetic data generation | Option 1: 1/2 GPU for lower-quality in-cluster synthetic data generation; Option 2: 4 GPUs for highest quality in-cluster synthetic data generation | You can leverage an external LLM as an alternative to hosting the model in-cluster |