Static Evaluation
Static evaluation is the build-time review step that helps teams understand what an agent can reach before runtime enforcement is enabled.
It looks at the agent's tools, MCP servers, capability labels, and observed trajectories. The output is risk evidence that can be reviewed directly and used by the policy workflow.
What It Looks For
Static evaluation focuses on risky capability combinations.
For data exfiltration, the core pattern is:
ingest untrusted content + access private data + communicate externally
For unsafe actions, the core pattern is:
ingest untrusted content + change state, especially irreversible actions
The point is not just whether a single tool is risky. The point is whether the agent can combine tools in a way that creates a meaningful path to data loss or uncontrolled action.
Inputs
Static evaluation can work from several kinds of evidence.
Input Modes
| Input mode | Use when |
|---|---|
| MCP discovery | You want AgentWarden to discover tools from connected MCP servers. |
| Raw tool inventory | You can describe tools directly and want AgentWarden to tag them. |
| Pre-tagged tools | You already have reviewed tool capability labels. |
| Combined inputs | You want to combine raw inventory, MCP discovery, and pre-tagged tools in one evaluation. |
When combining inputs, each tool identity should appear in only one source.
MCP discovery can read host-specific MCP configuration for supported agent runtimes. MCP servers may be remote services or local subprocesses. For local servers, the runtime environment must be able to start the server and resolve any required credentials.
Review Context Inputs
- Trajectories: recorded or representative agent runs that show which tools were actually used.
- Scope constraints: narrows evaluation to the tools or servers in scope.
- Policy constraints: captures environment limits or disabled risk conditions.
- Use-case context: explains what the agent is expected to do and which capabilities are acceptable.
- Server metadata: adds server-level context for raw MCP or tool-inventory inputs before tool tagging.
- Optional metadata: groups or labels evaluation jobs for later review.
Results
Static evaluation produces a report that helps reviewers answer:
- Which tools and MCP servers were found?
- Which tools can access private data, ingest untrusted content, communicate externally, or change state?
- Which risky combinations are possible from the available tools?
- Which risky combinations appeared in observed trajectories?
- Which high-risk tools appear unused?
- Which remediations would reduce the most risk?
Typical reports have four review areas:
| Report area | What it helps answer |
|---|---|
| Introduction and capabilities | What servers and tools were discovered, and whether connectivity was complete. |
| Tool understanding | Which tools carry which capability labels, and which tools are broad enough to deserve closer review. |
| Risk understanding | Which risky paths exist statically, and which ones appeared in observed trajectories. |
| Remediation ranking | Which tools or constraints would reduce the most risk first. |
Large evaluations can produce many example paths and observed witnesses. Saved examples can be capped to keep reports easier to download, while summary counts remain complete.
Static Exposure and Observed Exposure
Static exposure and observed exposure answer different questions.
- Static exposure: what could happen based on the tools and capabilities available to the agent.
- Observed exposure: what actually appeared in recorded or representative trajectories.
Both matter. Static exposure catches reachable risk before it happens. Observed exposure helps prioritize what the agent is already doing.
Reducing Risk
Static evaluation is most useful when teams run it iteratively:
1. Remove tools the agent does not need.
Use scope constraints to limit the tool inventory before analysis. An explicit include list is usually safer than a blocklist because newly added tools are not allowed by default.
2. Model environment constraints.
Use policy constraints when a tool is needed, but a risk condition does not apply in the customer environment. For example, a tool may still be available, while public writes or external-tenant communication are disabled for that server or tool.
3. Re-run and compare.
Compare the new report to the previous report. Useful comparisons include:
- whether static risky paths decreased,
- whether observed risky patterns decreased,
- how many tools remain in scope,
- whether the remaining high-risk tools are tools the agent actually uses.
Scope is best for removing unused tools. Policy constraints are best for keeping needed tools while modeling the conditions under which they are allowed to operate.
How It Feeds Policy
Static evaluation is the evidence layer for policy review.
The policy workflow uses the evaluation output to draft candidate controls, review which controls should exist, compile them for a runtime, and deploy an active policy.
Next, see Policy Workflow for how evaluation output becomes reviewed runtime policy.
For concrete commands, see CLI.