Skip to main content

Troubleshooting

Pentest fails — Azure content filter blocks the prompts (ContentPolicyViolationError)​

Symptom​

Every model request is rejected and the job ends in ERROR:

generation failed (gpt-4o-mini): - 55/55 inputs failed with error
"litellm.ContentPolicyViolationError: The response was filtered due to the prompt
triggering Azure OpenAI's content management policy."
(error rate: 100.0% exceeds 10% threshold)

Cause​

The target is an Azure OpenAI deployment with Azure's default Guardrails (content filters) on. DynamoEval's jailbreak prompts trip them, so Azure returns HTTP 400 instead of a response and the job fails. To red-team the model you must turn the target deployment's Guardrails off.

This is a property of the target deployment, not DynamoEval. The judge model never sees attack prompts, so leave it alone.

Fix​

1. Get approved for Modified Guardrails (one-time, per subscription). Turning filters off requires Microsoft approval; red teaming is an eligible use case.

  • Apply: Limited Access Review: Modified Content Filters
  • Reason: "To facilitate red teaming and/or testing and evaluation."
  • ~10 business days. Approval is per subscription — a customer running against their own Azure must apply for their own subscription.

2. Create a guardrail with everything off and assign it to your deployment(s). Use whichever method you prefer:

UI (Azure AI Foundry)​

  1. Microsoft Foundry → make sure New Foundry is off (use Foundry classic).
  2. Guardrails + controls → Content filters → + Create content filter.
  3. Name it dynamoeval-redteam.
  4. On both Input filters and Output filters, set every filter to Off — the four harm categories, Prompt Shields (jailbreak), and protected material. Create the filter.
  5. Models + endpoints → your target deployment → Edit → select dynamoeval-redteam → Save. Repeat for every deployment you'll test.

CLI​

SUB=...; RG=...; ACC=...           # Azure OpenAI account, e.g. dfl-shared-openai
POLICY=dynamoeval-redteam

# Create the "all off" guardrail (enabled:false = filter off)
az rest --method put \
  --url "https://management.azure.com/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.CognitiveServices/accounts/$ACC/raiPolicies/$POLICY?api-version=2024-10-01" \
  --headers "Content-Type=application/json" \
  --body '{"properties":{"basePolicyName":"Microsoft.DefaultV2","contentFilters":[
    {"name":"Hate","source":"Prompt","enabled":false,"blocking":false},
    {"name":"Hate","source":"Completion","enabled":false,"blocking":false},
    {"name":"Sexual","source":"Prompt","enabled":false,"blocking":false},
    {"name":"Sexual","source":"Completion","enabled":false,"blocking":false},
    {"name":"Violence","source":"Prompt","enabled":false,"blocking":false},
    {"name":"Violence","source":"Completion","enabled":false,"blocking":false},
    {"name":"Selfharm","source":"Prompt","enabled":false,"blocking":false},
    {"name":"Selfharm","source":"Completion","enabled":false,"blocking":false},
    {"name":"Jailbreak","source":"Prompt","enabled":false,"blocking":false},
    {"name":"Protected Material Text","source":"Completion","enabled":false,"blocking":false},
    {"name":"Protected Material Code","source":"Completion","enabled":false,"blocking":false}
  ]}}'

# Assign it to the deployment (PATCH preserves the model + capacity)
az rest --method patch \
  --url "https://management.azure.com/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.CognitiveServices/accounts/$ACC/deployments/gpt-4o-mini?api-version=2024-10-01" \
  --headers "Content-Type=application/json" \
  --body "{\"properties\":{\"raiPolicyName\":\"$POLICY\"}}"

GitOps (Terraform)​

resource "azapi_resource" "redteam_guardrail" {
  type      = "Microsoft.CognitiveServices/accounts/raiPolicies@2024-10-01"
  name      = "dynamoeval-redteam"
  parent_id = azurerm_cognitive_account.openai.id
  body = {
    properties = {
      basePolicyName = "Microsoft.DefaultV2"
      contentFilters = [
        for f in [
          { name = "Hate",     source = "Prompt" },     { name = "Hate",     source = "Completion" },
          { name = "Sexual",   source = "Prompt" },     { name = "Sexual",   source = "Completion" },
          { name = "Violence", source = "Prompt" },     { name = "Violence", source = "Completion" },
          { name = "Selfharm", source = "Prompt" },     { name = "Selfharm", source = "Completion" },
          { name = "Jailbreak", source = "Prompt" },
          { name = "Protected Material Text", source = "Completion" },
          { name = "Protected Material Code", source = "Completion" },
        ] : merge(f, { enabled = false, blocking = false })
      ]
    }
  }
}

resource "azurerm_cognitive_deployment" "gpt4o_mini" {
  name                 = "gpt-4o-mini"
  cognitive_account_id = azurerm_cognitive_account.openai.id
  rai_policy_name      = azapi_resource.redteam_guardrail.name
  # ...model + sku blocks...
}

3. Re-run as a NEW evaluation. Allow ~5 minutes for the change to propagate.

Always validate with a fresh test

DynamoEval resumes from a checkpoint keyed on the test. Re-running the same test can reuse old (filtered) results and still show errors even after the filter is off. Start a new evaluation (or register a fresh AI System) so every prompt is re-sent to the corrected target.

Pentest fails — Rate limit exceeded (HTTP 429)​

Symptom​

Error processing request: Rate limit exceeded for model gpt-4o-mini.
- 104/104 inputs failed ... 429 ... (error rate: ... exceeds 10% threshold)

Cause​

DynamoEval sends prompts in large concurrent batches (50–100+ at once). If the target deployment's TPM/RPM quota is too low, Azure returns 429 and the job fails the 10% error threshold. (Relaxing Guardrails does not raise throughput.)

Fix — raise the deployment's rate limit​

# Check available quota, then raise capacity (PATCH preserves model + guardrail)
az cognitiveservices usage list -l eastus --subscription "$SUB" -o table \
  --query "[?contains(name.value,'gpt-4o-mini')].{name:name.value, current:currentValue, limit:limit}"

az rest --method patch \
  --url "https://management.azure.com/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.CognitiveServices/accounts/$ACC/deployments/gpt-4o-mini?api-version=2024-10-01" \
  --headers "Content-Type=application/json" \
  --body '{"sku":{"name":"GlobalStandard","capacity":500}}'

UI: Foundry → Models + endpoints → deployment → Edit → raise Tokens per Minute Rate Limit → Save. GitOps: set sku { capacity = 500 } on the deployment. Size capacity for the eval's peak concurrency, not its average.

References​

Last updated on