← /editions
#98

Your Agents Cost More Than You Think

// A CTO-focused look at why most agentic AI ROI math is wrong, and how to measure cost per completed task instead of cost per API call.

$ today_in_10_seconds

The agentic AI pilot phase is over, and finance teams are asking the question engineering has been dodging: are these things paying for themselves? A new piece aimed at CTOs argues that most ROI math for agents is wrong because it measures the wrong unit — API calls instead of completed tasks. If you can't attach a dollar figure to your agents by Q4, expect someone else to do it for you.

$ lead_story

Are AI Agents Actually Saving Money?

Security Boulevard lays out how CTOs should measure return on agentic AI, and the core argument is uncomfortable: most teams are tracking inference spend and "hours saved" while ignoring the costs that dominate agent deployments — retries, tool-call failures, human review of agent output, and the engineering time spent maintaining prompts and guardrails. Agents are different from chatbots here because a single task fans out into dozens of model calls, tool invocations, and branches, so per-call pricing tells you almost nothing about per-outcome cost. The piece pushes for a task-level ledger: cost to complete, success rate, and the fully loaded cost of the failures, compared against the human baseline for the same task. For builders, the implication is that instrumentation is no longer a nice-to-have — if your agent can't report what it cost to finish a job, you can't defend its budget.

// why_it_matters: Agents that can't prove their ROI at the task level will lose their budget to ones that can.

read more →

$ also_worth_attention

  • The unit of measurement is the task, not the token

    An agent that burns 40 calls to finish a job it completes 90% of the time has a real cost that no dashboard shows by default. OpenTelemetry's GenAI semantic conventions now give you a standard way to trace spans across model calls and tool use, so cost-per-outcome becomes a query instead of a guess. Insight: if you can't roll up a trace into a single task, you don't have an agent, you have a pile of API calls.

    link →
  • "Hours saved" is not a number your CFO will accept

    Time-saved estimates assume the reclaimed hours convert to output, and they rarely do. The credible version is a controlled comparison: same task, agent versus human, measured on completion, quality, and rework. METR's work on measuring AI task capability is the closest thing to a rigorous template for this. Insight: measure tasks retired, not minutes shaved.

    link →
  • Governance overhead is a line item, not a footnote

    Every autonomous action needs a review policy, an audit trail, and someone who owns the failure. That cost scales with autonomy, and it's usually the thing that turns a "positive ROI" pilot negative in production. NIST's AI Risk Management Framework is the reference most security teams will hold you to. Insight: budget the guardrails before you budget the agent.

    link →

$ bigger_picture

The trend behind the headlines

The agent conversation has shifted from "can it do the job" to "what does the job cost," and that shift changes who's in the room. Engineering teams that were optimizing for capability now have to optimize for cost-per-successful-outcome, which is a very different engineering problem: fewer retries, tighter tool contracts, cheaper models for cheaper steps, and humans only where they're worth it. The winners of this phase won't be the teams with the smartest agents — they'll be the teams whose agents come with a receipt. Expect observability, evals, and cost attribution to become the real moat, and expect the next round of "agents don't work" headlines to come from companies that never instrumented theirs.

$ tool_of_the_day

Langfuse

Open-source LLM observability that traces every step of an agent run, attaches token cost to each span, and lets you roll it up by session, user, or task. It's the fastest way to get from "we think it's cheaper" to a per-task cost number you can put in front of finance. Self-hostable, which matters if your security team has opinions.

explore Langfuse →

$ subscribe --free

Get the next edition in your inbox

// DAIly Agent covers agentic AI three times a week

// free, no spam, exit anytime