Observability: Understanding What Your Agent is Doing

Your agent is tested, and it's running in production. A week later, a user complains that it's "slow." Another user says it "gave a weird answer." Your CFO asks, "How much is this thing costing us?"

How do you answer these questions?

Welcome to the world of observability. It's the art and science of understanding the internal state of your system by observing its external outputs. For complex, non-deterministic systems like AI agents, observability isn't a luxury; it's a fundamental requirement for operation. You cannot manage, debug, or improve what you cannot measure.

This practice is built on three pillars: Logs, Traces, and Metrics.

Pillar 1: Logging (The "What")

We touched on logging for debugging, but for production systems, we need to be more disciplined. The key is structured logging. Instead of printing plain text, you log JSON objects with clear, consistent key-value pairs.

• Bad: console.log("Processing research request for topic X")
• Good: console.log(JSON.stringify({ level: "info", message: "Request received", requestId: "xyz-123", endpoint: "/research", topic: "X" }))

Why? Because structured logs are machine-readable. You can send them to a logging service (like Cloudflare Logpush) and then easily search, filter, and create dashboards. You can instantly find all logs for a specific requestId, calculate the average processing time, or alert on a spike in level: "error" messages.

For every significant AI transaction, you should log at a minimum:

• A unique requestId that follows the request through your entire system.
• The validated input from the user.
• The final, complete prompt sent to the LLM.
• The model used (e.g., @cf/meta/llama-3-8b-instruct).
• The raw text response from the LLM.
• The result of your Zod validation on that response (success or failure).
• The final, structured data returned to the user.

This level of detail turns a mysterious failure into a straightforward bug report.

Pillar 2: Tracing (The "Where" and "When")

In our architecture, a single user request might trigger a cascade of events: the initial worker calls an AuthAgent, which then writes to D1, after which the initial worker calls a ResearchAgent, which itself sends a message to a queue that's picked up by an IndexingAgent. This is a distributed system.

When a request is slow, where is the bottleneck? This is a question that distributed tracing answers. A trace follows a single request—using the requestId we logged earlier—as it travels between all your different agents and services. It visualizes the entire journey as a flame graph, showing you exactly how much time was spent in each step.

• Cloudflare Workers has built-in support for tracing. When you look at a worker invocation in the Cloudflare dashboard, you can see a detailed breakdown of its execution, including the time spent in fetch calls to other services (like other workers or the AI Gateway) and I/O operations (like KV or D1 calls).
• LangSmith is a specialized observability platform from the creators of LangChain. It's designed from the ground up to trace and visualize the execution of your LLM chains. It can show you the full context of every prompt, the output of every step, and exactly how long each part of the chain took. For debugging complex, multi-step agent competences, it is an invaluable tool.

Tracing allows you to move from "the request was slow" to "the request was slow because the third step in our summarization chain took 4.5 seconds."

Pillar 3: Metrics (The "How Much")

Metrics are aggregated numbers that give you a high-level, real-time view of your system's health and efficiency. For AI agents, a few key metrics are non-negotiable. Luckily, the Cloudflare AI Gateway provides most of them right out of the box.

On your AI Gateway dashboard, you should be constantly monitoring:

1. Cost: This is the most obvious one. The dashboard gives you a real-time view of your spending across all models and providers. You should set up budget alerts to get notified before costs run away from you.
2. Latency: How long are your users waiting? The dashboard will show you the average latency (and P95/P99 percentiles) for your LLM calls. If this number starts to creep up, it's a sign that the provider is having issues or your prompts are becoming too complex.
3. Error Rate: What percentage of your AI calls are failing? The gateway breaks this down into different error types, allowing you to distinguish between an outage at the AI provider versus an issue with your own code.
4. Tokens per Request: This is a direct driver of both cost and latency. The dashboard shows you the average number of input tokens (your prompt) and output tokens (the model's response) for each model. If you see your input tokens unexpectedly spiking, it could be a sign that you are feeding too much context (e.g., an overly long chat history) into your prompt.

By monitoring these four metrics, you can get an instant, high-level understanding of the health, cost, and performance of your entire AI infrastructure.

Building a Glass Box

The goal of observability is to turn your agent from an opaque "black box" into a transparent "glass box."

• Metrics give you the 30,000-foot view of your system's health.
• Traces let you zoom in on a single slow or failed request to see its journey.
• Logs give you the ground-level, "source code" view of what exactly happened during that single step.

By building a system with good observability from day one, you build a system you can understand, trust, and operate effectively at scale.