AWS has published a reference implementation for testing AI agents through GitHub Actions, allowing developers to run automated evaluations as part of a continuous integration and continuous delivery (CI/CD) pipeline. The workflow can fail an evaluation job when an agent’s scores fall below a configured threshold.
The approach puts agent behaviour alongside other checks that can run before code is merged. Repositories can configure the evaluation job as a required GitHub status check through branch protection or rulesets, preventing collaborators from merging changes when the check fails.
The implementation uses Amazon Bedrock AgentCore Evaluations to assess an agent after changes are made to its code, system prompt, model, or tool configuration. AWS’s example deploys the agent to a development environment, invokes it with a set of test prompts, evaluates the resulting traces, and returns the results to the GitHub Actions workflow.
AWS published the example as a technical guide on September 8, alongside a reference implementation covering the agent, infrastructure, authentication, evaluation scripts, and GitHub Actions configuration. The workflow runs when a pull request to the main branch changes relevant agent code, infrastructure, MCP server code, or evaluation scripts.
AWS made AgentCore Evaluations generally available on March 31, 2026, with on-demand evaluation for development and CI/CD workflows and online evaluation for production traffic. AWS said at the time that on-demand evaluation supports regression testing in CI/CD pipelines.
Testing agent behaviour in CI
The reference pipeline uses a Strands-based agent running on Amazon Bedrock AgentCore Runtime. The agent connects to a Model Context Protocol (MCP) server, while AWS Cloud Development Kit code provisions the associated runtime, Amazon Cognito resources, and IAM roles.
When the GitHub Actions job runs, it deploys the development stack and invokes the agent using predefined evaluation prompts. The resulting OpenTelemetry traces are retrieved from Amazon CloudWatch and passed to AgentCore Evaluations for scoring.
The example uses four built-in evaluators: GoalSuccessRate, Correctness, ToolSelectionAccuracy, and ToolParameterAccuracy. These checks cover whether the agent completed the requested task, returned an appropriate response, selected an appropriate tool, and derived tool parameters from information available in the conversation context.
The checks are scored separately. In AWS’s deliberate regression example, an altered system prompt caused GoalSuccessRate, Correctness, and ToolParameterAccuracy to fail, while ToolSelectionAccuracy still passed.
The sample workflow sets an evaluation threshold of 0.8. When metrics fall below that level, the evaluation script exits with a failure. AWS restored the original system prompt in its example and reran the pipeline, after which all four evaluations cleared the threshold.
A failed GitHub Actions job only blocks merging when the repository configures it as a required status check. GitHub supports this through protected branches and rulesets.
Building regression test suites
The September 8 implementation uses a fixed group of evaluation prompts, but AgentCore also has a dataset evaluation capability for running collections of test scenarios. AWS lists regression testing and CI/CD pipelines among its intended uses, alongside benchmark testing and comparisons against established baselines.
The dataset runner automates several steps that developers would otherwise perform separately. It invokes the agent against the scenarios, waits for telemetry ingestion, and evaluates the resulting interactions. AWS currently labels dataset evaluation as a public preview.
A predefined scenario can contain more than the prompt presented to an agent. The dataset schema supports an expected response, natural-language assertions describing required behaviour, and an expected trajectory listing the tools the agent should call.
Those fields can be tied to different checks. Correctness can compare the generated response with an expected response, GoalSuccessRate can assess session-level assertions, and trajectory evaluators can compare actual tool calls with an expected execution path.
AWS also supports versioning evaluation datasets. Developers can publish the current Draft as an immutable numbered version while continuing to edit the Draft for subsequent test changes. AWS records whether the working Draft contains changes that have not yet been published.
AgentCore can evaluate an entire session, an individual trace, or specific tool-call spans. Tests can therefore cover task completion, generated responses, tool selection, parameters, and execution paths rather than relying only on an agent’s final response.
AWS has previously said that the same request can result in different tool selections, execution paths, and outputs across separate runs because of the non-deterministic nature of large language models. Its guidance recommends creating datasets that represent expected requests and behaviours, then running those scenarios repeatedly during evaluation.
The service supports both model-based and programmatic checks. Correctness and GoalSuccessRate can use LLM-as-a-judge scoring, while trajectory matching can compare recorded tool calls directly with predefined ground truth.
AWS provides exact-order, in-order, and any-order trajectory checks. These determine whether the agent used the required tools and, where applicable, whether it called them in the expected sequence.
AWS also supports custom code-based evaluators through AWS Lambda. These can be written in Python or JavaScript and used for application-specific deterministic checks alongside model-based evaluators.
AWS warns that LLM-as-a-judge scores have inherent variance, meaning the same trace can receive slightly different scores across repeated evaluations. The company recommends leaving some margin when setting evaluation thresholds.
Trajectory checks instead use programmatic scoring to compare the recorded execution path with the expected trajectory and do not require LLM calls.
Running evaluations in GitHub Actions
AgentCore supports on-demand, online, and batch evaluation. On-demand evaluation can be used for controlled development testing and CI/CD workflows, online evaluation samples production traffic, and batch evaluation scores multiple sessions asynchronously.
For dataset-based testing, AWS provides on-demand and batch runners. The on-demand runner collects spans and requests evaluation from the client side, while the batch runner delegates span collection and evaluation to the service.
The reference GitHub Actions pipeline uses OpenTelemetry data generated during the agent sessions. AWS’s starter toolkit handles collection of the relevant traces from CloudWatch before those traces are submitted for scoring.
AWS also documents a second option that avoids invoking a live agent during the pull-request check. Developers can capture representative traces from a staging environment and evaluate the stored OpenTelemetry spans in CI without invoking the live AgentCore runtime.
AWS notes that this method evaluates behaviour captured from the staging deployment rather than code contained in the current pull request.
The reference pipeline uses GitHub Actions OpenID Connect federation to assume an AWS IAM role without storing long-lived AWS credentials. When the agent needs access to OAuth-protected MCP tools, the sample obtains a machine-to-machine token through Amazon Cognito.
AWS notes that the sample CI configuration gives the pipeline access to all tools and does not test role-specific access controls.
The workflow deploys its resources before evaluation and destroys the development stack after the job completes, including when the evaluation fails. AWS said its end-to-end example takes around 10 minutes because deployment, runtime startup, trace propagation, and evaluation all take place during the workflow.
Trace propagation alone can take between 30 and 90 seconds in AWS’s tests. The reference implementation retries trace collection every 30 seconds for up to 10 minutes before giving up.
Each LLM-based evaluator also requires a model invocation. In AWS’s example, four evaluators running against five prompts result in 20 judge-model calls for each pull request. AWS said these model invocations contribute to the Bedrock usage costs of running the evaluation pipeline.
Trajectory checks avoid those judge-model calls, while custom code-based evaluators can apply deterministic rules specific to an application’s testing requirements.
(Photo by Steve Barker)
See also: Should AI coding agents test their own code?
Want to dive deeper into the tools and frameworks shaping modern development? Check out the AI & Big Data Expo, taking place in Amsterdam, California, and London. Explore cutting-edge sessions on machine learning, data pipelines, and next-gen AI applications. The event is part of TechEx and co-located with other leading technology events. Click here for more information.
Developer Tech News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.