Custom RAG Application

Overview

DynamoEval provides support for integrating custom RAG applications through a flexible adapter system. This enables enterprises to:

• Leverage existing RAG implementations without modification
• Maintain control over their proprietary RAG pipelines
• Integrate specialized document retrieval systems
• Support custom authentication mechanisms
• Transform requests and responses to match existing APIs

Custom RAG Adapter Features

The CustomRagDB wrapper by DynamoAI facilitates integration with your vector database through REST APIs. Key features include:

• Acting as a bridge between DynamoEval and custom vector stores
• Requiring only a REST API endpoint that adheres to DynamoEval's request/response contract
• Supporting various authentication methods (e.g., No Auth, Bearer Token)
• Enabling request/response transformations using JSONata expressions

Requirements

1. Your custom RAG application must have at least one route of type "retrieve"
2. The application must handle authentication if required (supported types: NO_AUTH, BEARER)
3. The application must either:
   • Accept DynamoEval's standard request format and return the expected response format, OR
   • Use JSONata transformations to convert between your format and DynamoEval's format.

For detailed API documentation and implementation guidelines, see the Support for Custom RAG Adapter section below.

Request/Response Contract

DynamoEval's standard format:

// DynamoEval Internal Request Format
{
    "query": "user query text",
    "top_k": 3
}

// DynamoEval Expected Response Format
{
    "retrieved": [
        {
            "id": "doc1",
            "score": 0.95,
            "content": "document text - 1"
        },
        {
            "id": "doc2",
            "score": 0.75,
            "content": "document text - 2"
        },
        {
            "id": "doc3",
            "score": 0.89,
            "content": "document text - 3"
        }
    ],
    "text_only": ["document text - 1", "document text - 2", "document text - 3"]
}

Example 1: Direct Integration (No Transformations)

If your API already matches DynamoEval's format:

from dynamofl.entities import CustomRagApplicationRoutesEntity, AuthTypeEnum, RouteTypeEnum

base_url = "https://api.example.com"
auth_type = AuthTypeEnum.BEARER
custom_rag_application_routes = [
    CustomRagApplicationRoutesEntity(
        route_type=RouteTypeEnum.RETRIEVE,
        route_path="/retrieve",
        request_transformation_expression=None,
        response_transformation_expression=None,
    )
]

dfl.create_custom_rag_application(
    base_url=base_url,
    auth_type=auth_type,
    auth_config={"token": "bearer-token"},
    custom_rag_application_routes=custom_rag_application_routes
)

Example 2: With Transformations

If your API uses a different format:

# Your API's format:
# Request: {"searchQuery": "text", "limit": 3}
# Response: {"results": [{"docId": "1", "relevance": 0.9, "text": "content"}]}
from dynamofl.entities import CustomRagApplicationRoutesEntity, AuthTypeEnum, RouteTypeEnum

base_url = "https://api.example.com"
auth_type = AuthTypeEnum.BEARER
request_transformation_expression = """{
    "searchQuery": query,
    "limit": top_k
}"""
response_transformation_expression = """{
    "retrieved": results.{
        "id": docId,
        "score": relevance,
        "content": text
    },
    "text_only": [results.text]
}"""
custom_rag_application_routes = [
    CustomRagApplicationRoutesEntity(
        route_type=RouteTypeEnum.RETRIEVE,
        route_path="/retrieve",
        request_transformation_expression=request_transformation_expression,
        response_transformation_expression=response_transformation_expression,
    )
]

dfl.create_custom_rag_application(
    base_url=base_url,
    auth_type=auth_type,
    auth_config={"token": "bearer-token"},
    request_transformation_expression=request_transformation_expression,
    response_transformation_expression=response_transformation_expression,
    custom_rag_application_routes=custom_rag_application_routes
)

JSONata Transformations

DynamoEval uses JSONata expressions to transform requests and responses. See the JSONata documentation and JSONata playground for more details.

1. Request Transformation: Converts DynamoEval's standard request format to your API's format
2. Response Transformation: Converts your API's response format to DynamoEval's expected format

The transformations are applied in this order:

1. DynamoEval request → Request Transform → Your API
2. Your API response → Response Transform → DynamoEval

Once the custom rag application adapter is ready, here's how to set up CustomRagDB:

custom_rag_arg = {
    "custom_rag_application_id": 12 # id of custom-rag-application
}

from dynamofl import CustomRagDB
custom_rag_setup = CustomRagDB(**custom_rag_args)

Pass this custom_rag_setup as the vector_db argument in create_rag_hallucination_test() function.

Helper Prompt for JSONata Transformations

To simplify the process of writing JSONata code for transforming JSON objects, you can use the following GPT prompt. This prompt helps generate JSONata expressions by providing a sample of how your custom RAG adapter understands the data. Replace X with your specific JSON format:

Create a JSONata expression to transform the following JSON object:
Input: { "query": "user query text", "top_k": 3 }
Output: X

Example prompt

Create a JSONata expression to transform the following JSON object:
Input: { "query": "user query text", "top_k": 3 }
Output: { "searchQuery": query, "limit": top_k }