← /editions
#22

LongMedBench — agents can recall your history, but deciding is another matter

// A wave of new benchmarks is exposing where agents break down: not on knowledge, but on long, interdependent task chains. LongMedBench, PolyWorkBench, and the HORIZON diagnostic converge on one point — retrieval is nearly solved, sustained decision-making is not.

$ today_in_10_seconds

A wave of new benchmarks is exposing where agents actually fall apart: not on knowledge, but on the long, interdependent task chains that real work demands. Two fresh papers push evaluation into hard territory — longitudinal clinical decisions and multilingual execution — while a third diagnoses exactly where agents break as tasks get longer. The uncomfortable consensus: retrieval is close to solved; sustained decision-making is not.

$ lead_story

LongMedBench — agents can recall your history, but deciding is another matter

A new benchmark from a team publishing ahead of MICCAI 2026 targets the gap most medical-AI evals skip: real care is longitudinal, not a one-shot question. LongMedBench builds on MIMIC-IV records — 335 patients averaging 19.72 visits each and 44.91 events per visit — turning them into time-series event streams for multi-session agent interaction, scored across three suites: fact-based QA, temporal reasoning, and long-horizon decision-making. The core finding is the one that should give builders pause: RAG and agent memory systems improve information-retrieval tasks, but decision-making performance stays highly dependent on the model's immediate context — and while models use explicit timestamps well, they struggle with implicit time inference. In other words, bolting memory onto an agent helps it find facts, not make judgments across time.

// why_it_matters: Memory fixes recall, not reasoning — the hard part of long-horizon work is still trapped in the context window.

read more →

$ also_worth_attention

  • PolyWorkBench: agents that reason and act across languages

    Most agent benchmarks quietly assume everything — reasoning, tool calls, outputs — happens in English. PolyWorkBench (submitted July 9) challenges that monolingual assumption, evaluating long-horizon agents in multilingual settings where the execution process itself spans languages. For anyone deploying agents outside English-first markets, this probes a failure mode that English-only evals structurally can't see: does the planning-and-tool-use loop survive a language switch mid-task?

    link →
  • The 'long-horizon task mirage' — where agents actually break

    The sharpest of the group treats horizon as a controlled variable instead of just reporting success rates. HORIZON is a cross-domain diagnostic benchmark that collected 3,100+ trajectories from GPT-5 variants and Claude models across four domains to study horizon-dependent degradation, paired with a trajectory-grounded LLM-as-a-Judge pipeline for failure attribution. The steepest collapse showed up in embodied manipulation, and different task structures broke at different points — composite workflows failed earliest. This is the diagnostic layer the benchmark boom needs: not just 'did it pass' but 'where and why did it fail.'

    link →
  • Context-folding: making long horizons fit

    If the bottleneck is immediate context, the fix is managing it smarter. Context-folding lets an agent match or beat ReAct baselines on Deep Research and SWE tasks while using an active context roughly 10x smaller, outperforming summarization-based approaches. It's a concrete answer to the LongMedBench problem: rather than hope memory retrieval compensates, compress the working trajectory itself so distant evidence stays usable.

    link →

$ bigger_picture

The trend behind the headlines

Three months of agent releases have sold capability; this week's research is quietly auditing it. The pattern across all four papers is the same: agents are strong on short, self-contained tasks and degrade as tasks stretch into long, interdependent chains — and our headline benchmarks mostly hid that by reporting aggregate success rather than treating horizon as the variable. LongMedBench localizes the failure to decision-making under accumulating context; HORIZON quantifies the degradation curve; Context-folding attacks the mechanism. It ties directly to Edition #20's thesis that agents fail at the level of state, not intelligence — and it should temper how anyone reads a shiny launch-day benchmark. The real question for production agents isn't peak capability on a clean task; it's how gracefully they fail on step 40 of 50.

$ tool_of_the_day

HORIZON Leaderboard

The public leaderboard and HuggingFace dataset accompanying the 'task mirage' paper, tracking how agents degrade as task horizon increases across web, OS, database, and embodied domains. If you're building long-horizon agents, it's a rare resource that reports failure taxonomies and breaking points rather than a single success number — far more actionable for debugging than a leaderboard rank.

explore HORIZON Leaderboard →

$ subscribe --free

Get the next edition in your inbox

// DAIly Agent covers agentic AI three times a week

// free, no spam, exit anytime