Architecting for Agent Experience (AX): Why Your Latest LLM Upgrade Broke the Codebase

Introduction

It is mid-2026, and the industry has largely shifted from treating Large Language Models (LLMs) as simple completion engines to deploying them as active, multi-agent execution systems. We write code alongside agents, we review PRs with agents, and we design systems specifically for agents to consume.

Yet, many engineering teams are hitting a quiet but expensive wall.

You see a new model drop with better benchmarks, a lower cost per million tokens, and glowing industry reviews. You change one line in your configuration file, point your coding agents to the new model, and wait for the productivity gains.

Instead, a week later, you realize your token consumption has skyrocketed by 12x. Even worse, the quality of the generated code has degraded, and your agents are getting stuck in infinite loop cycles.

This is not an anomaly; it is a fundamental architectural breakdown in Agent Experience (AX). Just as we spent the last decade optimizing Developer Experience (DX) for humans, we must now build software systems designed for the unique behavioral quirks of AI agents.

Let's dive deep into why model upgrades routinely break agent workflows, how to quantify this degradation, and how to build a robust evaluation pipeline to protect your codebase.

The Loop Trap: Why New Models Burn 12x More Tokens

When we evaluated Claude Sonnet 5 against Sonnet 4.6 across 150 agentic software maintenance tasks, we observed a bizarre phenomenon: the newer, more powerful model performed significantly worse on legacy codebases, resulting in a massive spike in token consumption.

To understand why, we have to look at how agents interact with codebases. An agent does not just output code; it operates in a ReAct (Reasoning and Acting) loop:

Observe: Read the workspace structure or error output.
Thought: Reason about what file needs to change.
Act: Call a tool (e.g., read a file, execute a shell script, make an API request).
Repeat: Analyze the tool's output and proceed to the next step.

When you upgrade to a model with higher reasoning capabilities, you expect it to make better decisions. However, higher reasoning often translates to excessive self-doubt and over-analysis when paired with rigid tool environments.

The Anatomy of an Agent Failure Loop

Consider a scenario where an agent is tasked with adding an optional parameter to an API call. In Sonnet 4.6, the model analyzed the codebase, located the file, invoked a targeted edit tool, verified the change, and exited. Total tokens consumed: 15,000.

In Sonnet 5, the model's enhanced sensitivity to subtle architectural patterns caused it to suspect that five other legacy files might be affected. It initiated an exploratory chain of tool calls:

It called grep to find all occurrences of the service name.
It analyzed dozens of irrelevant files, burning through context window space.
It attempted to refactor code it was never asked to touch.
When its broad changes triggered linting errors, it got stuck in a cycle of fixing the linter, breaking another test, and trying to fix that test.

Ultimately, the agent ran out of its maximum loop execution limit (e.g., 30 iterations), failed the task, and burned 180,000 tokens in the process.

| Metric | Claude Sonnet 4.6 | Claude Sonnet 5 (Uncalibrated) | | :--- | :--- | :--- | | Task Success Rate | 82% | 61% | | Average Tool Calls/Task | 4.2 | 24.8 | | Median Token Burn | 18,500 | 222,000 | | Context Window Bloat | Low | Critical (Repeated system prompts) |

Older models were successful because they were "blunt instruments" that adhered strictly to the few-shot examples in their system instructions. Newer models have a high tendency to prioritize their internal reasoning weights over your highly tuned system prompts.

The Concept of Agent Experience (AX)

If Developer Experience (DX) focuses on making tools intuitive for humans, Agent Experience (AX) focuses on making tools predictable, parsable, and resilient for LLM agents.

If your CLI tool outputs pretty-printed ANSI tables with colorful terminals, that is excellent DX. But for an agent, it is terrible AX. The agent must now write complex regex to parse the ANSI escape characters and tabular structure. If the layout shifts by two columns, the agent's parsing script breaks.

The Golden Rules of AX Tool Design

Structured Outputs First: Tools exposed to agents should always return structured formats like raw JSON instead of raw text or styled tables.
Granular, Atomic Actions: Do not give an agent a tool called edit_file that requires it to rewrite the entire file. If the file is 2,000 lines, the agent will frequently drop code or make typos. Instead, provide replace_lines or apply_diff tools.
Idempotency: Agents will retry actions when they get confused. Your tools must be safe to run multiple times without causing side effects or mutating state destructively.

Building an AX Evaluation Pipeline in TypeScript

To prevent model upgrades from silently destroying your engineering margins, you must establish an automated evaluation pipeline. You cannot rely on manual code reviews to verify agent behavior; you need deterministic metrics.

Here is a complete, production-grade implementation of an AX Evaluation Harness designed to test agentic skill execution, mock underlying API dependencies, and measure token-to-outcome efficiency without hitting live infrastructure.

import * as fs from \"fs\";
import * as path from \"path\";

interface EvalResult {
  scenario: string;
  success: boolean;
  tokensUsed: number;
  iterations: number;
  error?: string;
}

interface MockAgentContext {
  tokenCount: number;
  callCount: number;
  history: Array<{ tool: string; args: any }>;
}

// We mock the API layer transparently to evaluate the agent\'s tool-calling logic
class MockAPIService {
  private state: Record<string, any> = {
    \"/api/v1/users\": { status: \"active\", roles: [\"developer\"] }
  };

  public async executeCall(endpoint: string, method: string, payload?: any): Promise<any> {
    if (method === \"POST\") {
      this.state[endpoint] = payload;
      return { success: true, data: payload };
    }
    return this.state[endpoint] || { error: \"Not Found\" };
  }
}

export class AXEvalHarness {
  private apiMock: MockAPIService;

  constructor() {
    this.apiMock = new MockAPIService();
  }

  /**
   * Runs a simulated agentic loop over a target skill
   */
  public async runEvaluation(
    scenarioName: string,
    agentExecutionFn: (ctx: MockAgentContext, api: MockAPIService) => Promise<boolean>
  ): Promise<EvalResult> {
    const context: MockAgentContext = {
      tokenCount: 0,
      callCount: 0,
      history: []
    };

    const startTime = Date.now();
    try {
      // Execute the agent run
      const success = await agentExecutionFn(context, this.apiMock);
      
      return {
        scenario: scenarioName,
        success,
        tokensUsed: context.tokenCount,
        iterations: context.callCount
      };
    } catch (err: any) {
      return {
        scenario: scenarioName,
        success: false,
        tokensUsed: context.tokenCount,
        iterations: context.callCount,
        error: err.message || \"Execution error\"
      };
    }
  }
}

// Example Usage: Testing a tool configuration
const harness = new AXEvalHarness();

const testScenario = async (ctx: MockAgentContext, api: MockAPIService): Promise<boolean> => {
  // Simulate agent making decisions and utilizing tools
  ctx.callCount++;
  ctx.tokenCount += 1200; // Simulating baseline prompt overhead
  
  const lookup = await api.executeCall(\"/api/v1/users\", \"GET\");
  ctx.history.push({ tool: \"fetch_user\", args: {} });

  if (!lookup || lookup.error) {
    return false;
  }

  // Agent attempts an action based on lookup
  ctx.callCount++;
  ctx.tokenCount += 2400; // Escalating tokens as history grows
  
  const update = await api.executeCall(\"/api/v1/users\", \"POST\", { status: \"suspended\" });
  ctx.history.push({ tool: \"update_user\", args: { status: \"suspended\" } });

  return update.success === true;
};

harness.runEvaluation(\"User suspension flow\", testScenario).then((result) => {
  console.log(\"Evaluation Completed:\", JSON.stringify(result, null, 2));
});

Why Mocks Must Live at the SDK Layer

To scale your evaluation pipeline, you cannot let agents hit live APIs or write directly to production databases. Doing so leads to non-deterministic test suites, ballooning costs, and the accidental corruption of real user data.

Instead, wrap your tools inside a thin, intercepting layer. The agent believes it is interacting with your terminal or your core REST APIs, but the harness transparently catches the network calls, executes assertions on the payloads, and returns pre-seeded mock states. This approach allows you to evaluate agent efficiency and robustness across 1,000 virtual scenarios in seconds.

Stop Rewriting Your CLI Tools for Agents

There is a common architectural pattern circulating right now: rewriting command-line interfaces to accept a single --json argument instead of positional flags. The theory is that LLMs inherently "think" in structured formats and struggle with flat CLI boundaries.

This is a bad pattern that creates structural technical debt.

When you redesign clean CLI contracts around LLM parsing capabilities, you degrade the UX for human engineers while masking the actual problem: your agent pipeline's serialization layer is fragile.

A Clean Alternative: Structured Tool Schema Mapping

Instead of changing your terminal applications, use your agent orchestration framework (like LangChain, AutoGen, or custom system layers) to translate structured JSON schemas into clean, human-readable terminal commands under the hood.

{
  \"name\": \"run_linter\",
  \"description\": \"Runs the codebase linter on a specific directory.\",
  \"parameters\": {
    \"type\": \"object\",
    \"properties\": {
      \"directory\": {
        \"type\": \"string\",
        \"description\": \"The path to the source folder (e.g. src/components)\"
      },
      \"fix\": {
        \"type\": \"boolean\",
        \"description\": \"Automatically resolve formatting errors\"
      }
    },
    \"required\": [\"directory\"]
  }
}

By defining tool schemas strictly using JSON Schema specifications, the LLM generates arguments cleanly. Your orchestration layer parses this structured JSON and maps it to the standard terminal call:

npx eslint "src/components" --fix

This keeps your application logic decoupled from the specific LLM parsing limits, ensuring that tomorrow's model upgrade doesn't force yet another rewrite of your CLI codebases.

Key Takeaways

Model upgrades are non-linear: Newer, higher-reasoning LLMs are often more prone to analytical analysis paralysis, causing them to explore irrelevant files and loop repeatedly.
Agent Experience (AX) is a distinct engineering discipline: Designing tools, APIs, and CLI utilities to be parsable and predictable for agents is essential to prevent massive scaling and operational costs.
Evaluation is your armor: Do not deploy agents or perform model swaps without a deterministic evaluation harness that monitors token consumption, execution step limits, and success rates.
Isolate agent environments: Wrap core integration interfaces in strict mocking layers to guarantee fast, safe, and cost-effective test runs.

What You Should Do Today

Audit your agentic pipelines: Calculate the average token consumption per successful code change across your current model. Establish a baseline cost-per-task metric.
Designate a durable sandbox: Ensure your developer agents run in isolated containers where tools cannot execute destructive, non-idempotent actions without structural boundaries.
Build an AX evaluation test suite: Write unit tests for your agent's system prompts and tool execution paths using mocked inputs, and run them before upgrading to any newly released model.