← /editions
#133

Agents Explore, Replays Ship

// An open-source framework uses agents to test web and mobile apps, then replays verified runs with no model calls, while deepsense.ai asks whether passing an eval means an agent is ready to ship.

$ today_in_10_seconds

Today's theme is trust: how do you know an agent did the right thing? One open-source framework uses agents to test your app and then removes the model from the loop once a run is verified. Another piece asks the question every agent team eventually faces: passing the eval isn't the same as being ready to ship.

$ lead_story

E2E — an open-source agentic testing framework for web, iOS and Android

TesterArmy's e2e lets you write tests as goals (upgrade the workspace to the Pro plan). An agent drives the app to reach the goal, and ordinary locators and assertions check the result in the same test. The clever part is caching. When a later assertion verifies an agent step, e2e records the actions, and the next run replays them with zero model calls until the app changes. It runs on Chromium, Firefox and WebKit through Playwright, and on iOS and Android simulators and emulators. You bring your own model, including a local one. It's Apache-2.0 and pre-1.0, so expect API churn.

// why_it_matters: Agents explore, replays ship. That's how you get agentic flexibility in CI without paying model costs or accepting non-determinism on every run.

read more →

$ also_worth_attention

  • Your agent passed the eval. Should you ship it?

    deepsense.ai offers a practical framework for evaluating agentic systems, built around the gap between a passing score and production readiness. My take: multi-step agents tend to fail in the path they take, through wrong tool calls, wasted loops and lucky recoveries, not just in the final answer. If your eval only grades outputs, you're shipping blind.

    link →

$ bigger_picture

The trend behind the headlines

These two stories are the same problem seen from opposite sides. One uses agents to test software, and the other tests the agents. Both land in the same place: let the model be flexible where flexibility pays, then pin its behavior down with deterministic checks. The teams that win with agents won't have the cleverest prompts. They'll have the tightest verification loops.

$ tool_of_the_day

Playwright

Playwright is the engine under e2e's web runner, and it's worth knowing even if you never use e2e. Its locator model, auto-waiting and trace viewer make it a natural foundation for browser agents, because traces let you replay exactly what your agent saw and clicked.

explore Playwright →

$ subscribe --free

Get the next edition in your inbox

// DAIly Agent covers agentic AI three times a week

// free, no spam, exit anytime