Custom Systems Overview

How to connect a custom REST AI endpoint to DynamoEval when it is not one of the built-in providers—including Custom API Language Models and Custom RAG Applications.

When do you need this?

When you add an AI system in DynamoEval, you can usually pick a supported provider—for example OpenAI, Azure OpenAI, Anthropic, or Mistral. For those, DynamoEval already knows how to authenticate, shape chat requests, and parse responses. You mainly supply credentials and a model or deployment name.

That path does not cover every setup. Many teams expose their own application as a REST endpoint: an internal gateway, a fine-tuned model service, an agent API, or any HTTP service whose request and response contracts are not the same as a standard provider API.

Custom Applications are for that case. You point DynamoEval at your URL and teach it how to talk to your contract—whether that is a Custom API Language Model or a Custom RAG Application.

How adaptation works

During evaluation, DynamoEval repeatedly sends a test prompt to the AI system and reads the reply—often thousands of times, including in parallel. Internally it uses one request shape and one expected response shape (documented in Custom API Language Models).

Your endpoint almost certainly speaks a different dialect: different field names, auth headers, envelopes, or even streaming (SSE) instead of a single JSON body.

So the core question is:

DynamoEval knows one way to send prompts and one way to consume answers. Your REST API has its own contract. How do we adapt between them?

The Custom Application adapter answers that:

DynamoEval request
        │
        ▼
 optional request JSONata  ──►  your API payload
        │
        ▼
   POST your endpoint  (+ auth / headers)
        │
        ▼
 your API JSON response
        │
        ▼
 optional response JSONata  ──►  DynamoEval response

1. DynamoEval builds its standard request.
2. An optional request JSONata transform rewrites that JSON into your API's payload.
3. DynamoEval POSTs to your endpoint with the auth (and headers) you configured.
4. An optional response JSONata transform maps your API's JSON back into the shape DynamoEval understands.

If your API already matches DynamoEval's formats, you can skip the transforms. If it only speaks SSE, WebSockets, or gRPC, put a REST aggregation proxy in front first—see Bridging Streaming APIs Behind a REST Proxy.

Requirements checklist

For the full requirements list used when preparing a Custom API Language Model, see:

Custom API Language Model Requirements

Summary:

#	Requirement	What we need from you
1	REST protocol	An HTTP endpoint that accepts POST and returns a single JSON response. SSE / event-stream, WebSockets, and gRPC are not supported directly—use a REST proxy (see below).
2	Fixed request/response shape	A stable schema. OpenAPI/Swagger, a sample curl, or example request/response JSON is enough to design JSONata transforms.
3	Long-lived authentication	One of: No Auth, Bearer Token, or API Key, or Microsoft Entra Workload Identity for an Entra-protected endpoint. Short-lived tokens are not supported, except Entra tokens that DynamoEval mints itself; see Custom API Language Model Requirements.
4	Custom headers	Any required headers beyond auth (for example model-routing or tenant headers). Call these out explicitly.
5	Network reachability	Reachable from the Dynamo deployment under test—public internet, or the same VPC / private network for private deployments.
6	Input size limits	Any max input length (characters or tokens) so evaluation payloads stay in bounds.

Non-REST endpoints

Custom Applications expect one JSON request and one JSON response over REST. DynamoEval cannot call SSE / text/event-stream, WebSockets, or gRPC endpoints directly.

For any of those protocols, run a REST proxy outside DynamoAI that speaks the upstream protocol and returns complete JSON. See Bridging Streaming APIs Behind a REST Proxy.

Credentials must be long-lived. Evaluations can run for hours; if a short-lived token expires midway, the run fails. Microsoft Entra ID tokens are the exception: DynamoEval mints and refreshes them itself. Interactive OAuth is also not supported—see Authentication notes.

Throughput, rate limits, and latency are optional for registration but recommended before large evaluation runs—see Custom API Language Model Requirements.

Authentication

DynamoEval supports these auth modes for Custom Applications:

Mode	Behavior
No Authentication	No credential headers.
Bearer Token	Sends Authorization: Bearer <token>.
API Key	Customizable header name and optional scheme prefix (for example X-API-Key: <key> or Authorization: Basic <key>).
Microsoft Entra Workload Identity	Sends Authorization: Bearer <Entra access token>, minted and refreshed by DynamoEval, plus an optional Azure API Management subscription key. Available when the deployment enables it.

Credentials must work on every evaluation request. Short-lived tokens that DynamoEval cannot mint itself, and interactive login, are not supported—see Authentication notes.

For UI fields and header construction examples, see Available Authentication Modes.

JSONata transformations

JSONata is a lightweight language for querying and reshaping JSON. DynamoEval uses it to bridge format gaps:

1. Request transform — DynamoEval standard request → your API payload
2. Response transform — your API response → DynamoEval expected format

Try expressions in the JSONata playground. A documented sample request/response (or OpenAPI / curl) for your endpoint is the fastest way to get the expressions right.

Minimal example

Input:  { "message": "Hello", "metadata": { "timestamp": 123 } }
Output: { "text": "Hello", "time": 123 }

{
  "text": message,
  "time": metadata.timestamp
}

Helper prompt

Use this with an AI assistant when drafting transforms. Replace X with your API's expected JSON:

Create a JSONata expression to transform the following JSON object:
Input: { "messages": [[{ "role": "user", "content": "Hello" }]], "N": 1, "temperature": 1 }
Output: X

For concrete transforms, see:

• Custom API Language Models for LLM-specific examples
• Custom RAG Applications for retrieval-specific examples

Next steps

1. Review Custom API Language Model Requirements (for custom LLMs).
2. If your API streams over SSE, follow Bridging Streaming APIs Behind a REST Proxy.
3. Configure the endpoint, auth, and transforms using Custom API Language Models or Custom RAG Applications.