Amazon Bedrock brings DeepEval and AutoEval to AgentCore for better AI agent testing
According to AWS Machine Learning: Testing AI agents in production is tough because traditional metrics often hide critical failures like leaked personal data. Amazon Bedrock is updating its AgentCore Evaluations feature to solve this by adding built-in AWS evaluators alongside popular third-party tools like DeepEval and AutoEval.
According to AWS Machine Learning:
The Evaluation Challenge
When deploying AI agents to production, simple metrics like latency or basic output checks aren't enough. An agent might resolve a customer support ticket while simultaneously leaking sensitive personal data. To fix this, Amazon Bedrock AgentCore Evaluations provides a structured way to measure what truly matters by mapping business goals to observable behaviors.
A Catalog of Evaluators
AgentCore brings together evaluators from three main sources. AWS built-ins cover response quality, task completion, and safety. Third-party managed options from DeepEval and AutoEval tackle complex multi-turn issues like knowledge retention, conversational completeness, and security flags. Developers can also create custom dimensions to grade multiple criteria in a single judge call.
Flexible Running Modes
To fit different development stages, AgentCore supports three execution modes using the exact same evaluator IDs. On-demand evaluation helps during quick debugging, online evaluation samples live production traffic, and batch evaluation runs regression tests against a golden dataset before any new release.
View selection rationale
Importance 78/100
About this brief
- Content type
- News summary
- Source type
- First-party / official source
- Original source
- AWS Machine Learning ↗
- Source published
- Process
- AI-assisted discovery, translation, summarisation, classification and scoring; standards and corrections are maintained by the editorial desk.
- Verification
- The original URL and publication metadata are retained. This brief is not independently fact-checked unless explicitly stated.
- Published by
- Polyemergence Editorial Desk · PolyEmergence Pte Ltd.
- Conflict disclosure
- No financial or commercial conflict declared.
