Trace and debug eve agent runs

eve's OpenTelemetry spans record model calls and tool execution. Use them to investigate wrong answers, slow steps, or unexpected token usage.

On Edgenivo, Agent Runs receives traces without a custom instrumentation file once sampling is configured. Capture policies determine which spans and content are available. You can also export traces to Datadog, Langfuse, or Braintrust.

Copy link to headingWhat an eve run records

An eve session can contain several turns. Each root agent turn starts a trace with an agent.step span for each loop step and the model and tool calls nested beneath it:

invoke_agent <agent>
  └── agent.step
        ├── chat <model>
        └── execute_tool <tool>

Skill loads appear as execute_tool eve:load-skill spans. If eve receives the caller's trace context, a local subagent's first turn joins that trace beneath the dispatch span. Without that context, it starts a separate trace.

Remote subagents start separate traces and link their first turn back to the caller. Later turns in child sessions start new traces.

Use these eve trace schema v4 attributes to connect related spans:

Model spans carry input, output, and cached token counts, and step spans add cost when Edgenivo AI Gateway served the call. Your trace content policy decides whether a span also holds prompts, responses, tool arguments, and tool results.

You can inspect the same spans locally before you deploy. eve dev writes traces to .eve/traces/v1, eve traces list lists them, and eve traces --verbose expands each span with all of its attributes:

eve traces list
eve traces --verbose

Copy link to headingFind and inspect runs in Agent Runs

Agent Runs is eve's built-in OpenTelemetry destination on Edgenivo. Preview and production deployments export to it by default without an instrumentation file.

Edgenivo bills Agent Runs usage at Tracing rates during the beta.

Copy link to headingTurn on trace sampling

When eve deploy creates a new Edgenivo project, it configures 100% trace sampling for all environments. Existing projects collect no traces until they have a sampling rule, so open your project's Settings → Tracing, then:

1. Select Add Sampling Rule.
2. Choose All Environments, Production, or Preview.
3. Set Rate to 100% to sample every matching request.
4. Leave Request Path Prefix blank to sample every path, then select Save.

Edgenivo applies the first matching rule, which you can update in the dashboard or with vercel traces config. Set its rate to 100%.

If you restrict sampling by path, cover /eve/v1 (or a named prefix such as /eve/support/v1), /.well-known/workflow, and any custom channel routes. Agents using dynamic schedules also need /.well-known/eve/v1/schedules/consumer.

Missing routes can leave gaps. Sampling does not restore spans or content excluded by eve's policies.

Copy link to headingInspect a run in the dashboard

Open your project and go to Observability → Agent Runs. The overview charts runs by trigger (such as Slack or HTTP) and token usage split into input, output, and cached tokens. Below the charts, a table lists each run with its triggering message, tokens, turn count, and duration.

Select a run to see its model, trigger, deployment, and per-turn details:

• Timing for every step, including skill loads and individual tool calls
• The turn's input and output
• The model's reasoning
• Each tool call with its arguments and results
• Input, cached, and output token counts

Prompts, responses, reasoning, and tool payloads appear only when recorded. Models may not expose reasoning, and capture policies can exclude content.

Copy link to headingInspect a run from the terminal

Use vercel agent-runs in a terminal or coding agent's shell. Replace the team and project names below, then choose a run ID from list:

vercel agent-runs projects --scope my-team
vercel agent-runs list --scope my-team --project my-app
vercel agent-runs inspect run_1234567890 \
  --scope my-team --project my-app
vercel agent-runs trace run_1234567890 \
  --scope my-team --project my-app \
  --json --max-field-length 0

Commands query production by default. Add --environment preview for preview runs and use the same environment when inspecting the selected run. inspect returns metadata, lifecycle events, usage, and subagents.

trace returns the recorded conversation. Combine --json with --max-field-length 0 to preserve full field values; Markdown output still shortens long content without --json.

For additional pages, follow the continuation commands printed by the CLI. These retain the relevant run and filter context. Untruncated fields do not combine pages or restore content that was never recorded.

If your editor uses Edgenivo MCP, it can inspect runs with list_agent_run_projects, list_agent_runs, get_agent_run, and get_agent_run_trace.

Trace results can be paginated or unavailable. Follow the response and pagination instructions before treating an empty page as evidence that a run had no activity.

Copy link to headingControl what traces capture

eve controls trace content at two levels. The process-wide tracePolicy in otel() sets the maximum content any OpenTelemetry destination can receive. Each destination's exportPolicy can only narrow that content.

Copy link to headingKnow the default

Channels set a session's audience to public, private, or unknown when creating it. Audience determines content capture, not who can access traces. The default policy retains metadata and captures content as follows:

Inputs cover model prompts, tool arguments, and channel input. Outputs cover model responses, reasoning, tool results, and error details.

Metadata includes span names, timing, model and tool names, and reported token usage. These fields remain available when content capture is disabled.

Copy link to headingSet the process-wide policy

Declare otel() once, in agent/instrumentation/otel.ts. This policy keeps the default rules for public and private sessions and drops traces for unknown-audience sessions outside development:

import { otel } from "eve/instrumentation/otel";

export default otel({
  resource: {
    "deployment.environment": process.env.VERCEL_ENV ?? "development",
  },
  tracePolicy: ({ audience, environment }) => {
    if (audience === "unknown" && environment !== "development") {
      return { emit: false };
    }
    const recordContent =
      audience === "public" || environment === "development";
    return {
      emit: true,
      recordInputs: recordContent,
      recordOutputs: recordContent,
    };
  },
});

Keep the policy deterministic because eve can evaluate it more than once while preparing a session. Its resolved decision sets the session's default Workflow run-data retention when the session starts.

If the policy drops a trace or excludes inputs or outputs, eve asks the Workflow backend to delete stored run payloads and streams when the run ends. Completed transcripts then become unreadable on backends that support deletion.

Set experimental.workflow.retention: "default" in defineAgent() to retain run data for the backend's default period independently of tracing. Destination exportPolicy filters do not change this retention decision.

Deleting run data leaves local traces and exported telemetry intact.

Copy link to headingNarrow what Agent Runs receives

Default-export agentRuns() to set the policy for the Agent Runs destination.

This policy keeps the spans it receives, removes inputs and outputs from non-public sessions, and drops the custom customer.id attribute:

import { agentRuns } from "eve/instrumentation/otel";

export default agentRuns({
  exportPolicy: {
    span: ({ audience }) =>
      audience === "public"
        ? { emit: true }
        : { redact: true, inputs: true, outputs: true },
    attribute: ({ key }) =>
      key === "customer.id" ? { emit: false } : { emit: true },
  },
});

The span callback keeps, drops, or redacts each span. Return { emit: true } to keep it or { emit: false } to drop it. Use { redact: true } with inputs, outputs, or both to remove content. If the callback throws, Agent Runs drops the span.

The attribute callback can keep, remove, or replace a value. Return { emit: true }, { emit: false }, or { replace: true, value } for each action. Use it to filter custom attributes outside eve's built-in redaction.

Input redaction removes eve's prompt, instruction, document, and tool-argument attributes. Output redaction removes response, reasoning, tool-result, and exception details, plus status messages. The span's status code is preserved.

Redaction in agentRuns() affects only Agent Runs. To turn Agent Runs off, default-export disableInstrumentation() from eve/instrumentation in the same file.

Local traces from eve dev include content by default and obey the process-wide policy. Set EVE_TRACES_CONTENT=off in .env.local to redact local inputs and outputs. This setting affects only local traces.

Warning: Captured model and tool content is visible to project members with trace access. If your agent handles personal, sensitive, or regulated data, review what it records and whether your privacy disclosures need to cover it.

Copy link to headingExport traces to your observability tool

To send traces somewhere besides Agent Runs, add one file per destination under agent/instrumentation/ and default-export otelIntegration() from each.

eve already emits OpenTelemetry spans, so you don't register a tracer provider yourself. Every otelIntegration() destination shares the process-wide tracePolicy, and Agent Runs keeps receiving spans alongside it.

Copy link to headingExport to Datadog

This example sends eve traces to Datadog APM over OTLP/HTTP.

Copy link to heading1. Install the exporter

Install @vercel/otel, which provides the OTLP exporter:

pnpm i @vercel/otel

npm i @vercel/otel

yarn add @vercel/otel

bun add @vercel/otel

Copy link to heading2. Add your Datadog credentials

The exporter below builds Datadog's Edgenivo OTLP endpoint from DD_SITE. Set this variable to your Datadog site, such as datadoghq.com or datadoghq.eu.

Add the site and API key as production environment variables:

vercel env add DD_SITE production
vercel env add DD_API_KEY production

Copy link to heading3. Declare the destination

Add the exporter in this instrumentation file:

import { OTLPHttpProtoTraceExporter } from "@vercel/otel";
import { otelIntegration } from "eve/instrumentation/otel";

const site = process.env.DD_SITE ?? "datadoghq.com";

export default otelIntegration({
  exportPolicy: {
    span: () => ({ redact: true, inputs: true, outputs: true }),
  },
  traceExporter: new OTLPHttpProtoTraceExporter({
    url: `https://vercel.integrations.otlp.${site}/v1/traces`,
    headers: { "dd-api-key": process.env.DD_API_KEY! },
  }),
});

The exportPolicy keeps span names, timing, and reported token usage in Datadog, but removes prompts, responses, and tool payloads. Delete it if your tracePolicy already limits content to what Datadog may store.

eve wraps traceExporter in its own batching processor, so you don't add one.

Copy link to heading4. Deploy and send a message

Deploy the agent to production:

eve deploy

Send your agent a message, then open Datadog APM and find the service named after your agent. eve sets service.name to the agent's name unless you set OTEL_SERVICE_NAME. Root agent turns appear as separate traces, with the same agent.step, chat, and execute_tool spans you see in Agent Runs.

Datadog's managed-platform endpoint doesn't attach host metadata, so features such as the Infrastructure Host List stay empty for these traces. If you already run an OpenTelemetry Collector with the Datadog exporter, point url at your Collector's OTLP/HTTP traces endpoint.

Copy link to headingExport to Langfuse

Langfuse accepts OTLP over HTTP/protobuf at /api/public/otel/v1/traces. Authenticate with Basic auth built from your project's public and secret keys. The x-langfuse-ingestion-version: 4 header selects its v4 ingestion model.

Install @vercel/otel if needed, then set LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY in your deployment environment. Add the exporter below:

import { OTLPHttpProtoTraceExporter } from "@vercel/otel";
import { otelIntegration } from "eve/instrumentation/otel";

const auth = Buffer.from(
  `${process.env.LANGFUSE_PUBLIC_KEY}:${process.env.LANGFUSE_SECRET_KEY}`,
).toString("base64");

export default otelIntegration({
  traceExporter: new OTLPHttpProtoTraceExporter({
    url: "https://cloud.langfuse.com/api/public/otel/v1/traces",
    headers: {
      Authorization: `Basic ${auth}`,
      "x-langfuse-ingestion-version": "4",
    },
  }),
});

The example URL is Langfuse's EU region. Use https://us.cloud.langfuse.com for the US region, or your own host for a self-hosted instance.

Copy link to headingExport to Braintrust

Braintrust accepts OTLP at https://api.braintrust.dev/otel/v1/traces, or https://api-eu.braintrust.dev/otel/v1/traces on the EU data plane.

Install @vercel/otel if needed and add BRAINTRUST_API_KEY to the deployment environment. Requests use that key as a Bearer token.

Set x-bt-parent to select the destination project. In the file below, replace my-eve-agent with your Braintrust project name:

import { OTLPHttpProtoTraceExporter } from "@vercel/otel";
import { otelIntegration } from "eve/instrumentation/otel";

export default otelIntegration({
  traceExporter: new OTLPHttpProtoTraceExporter({
    url: "https://api.braintrust.dev/otel/v1/traces",
    headers: {
      Authorization: `Bearer ${process.env.BRAINTRUST_API_KEY}`,
      "x-bt-parent": "project_name:my-eve-agent",
    },
  }),
});

This OTLP integration sends only content allowed by tracePolicy.

The registry command eve add instrumentation/braintrust installs a different provider: braintrustEveInstrumentation() from braintrust. It uses lifecycle events and does not apply tracePolicy or exportPolicy.

In [email protected], that provider records inputs and outputs regardless of audience and listens for event names used before eve 0.84. Check its compatibility with your eve version before installing it.

Copy link to headingInvestigate a wrong answer and add a regression eval

Trace the answer back through the model and tool calls to find where it diverged from the available evidence. Suppose an order-support agent tells a customer "Order 4417 shipped yesterday" while the order is still being packed.

Copy link to heading1. Find the run

Filter the Agent Runs table by time and trigger, or list runs from the terminal and pick the run ID:

vercel agent-runs list --scope my-team --project order-agent

Under the default policy, deployed traces for private or unknown-audience sessions contain metadata only. You can still see the order of tool calls and which calls failed.

To inspect prompts and results, reproduce the conversation with eve dev using test data. Local traces include content by default unless your policy excludes it. Select the saved trace with eve traces list, then inspect its ID:

eve traces show your_trace_or_conversation_id

Copy link to heading2. Walk the steps in order

Open the run and read each turn's agent.step spans from top to bottom. For each step, compare what the model received with what it decided:

• In the chat span, compare the prompt with the model's response. Check whether it calls a tool or answers directly.
• Read each execute_tool span's arguments and result. Use its error.type attribute to investigate failures.
• Check slow skill loads and tool calls for timeouts using the step timings.

In this example, step 1's chat span answered directly with no execute_tool span beneath it. The model stated an order status without calling get_order_status.

If the tool had failed, inspect the call and the answer that followed. Fixing the tool alone does not correct an invented order status.

Copy link to heading3. Fix the cause

Match the fix to the step that failed:

Copy link to heading4. Test the fix with an eval

Create a regression eval that checks the lookup and the expected status. Use a test agent with fictional order 4417 fixed at status: "packing" so changes to real order data cannot change the result.

Each app needs one eval config in its evals/ directory:

import { defineEvalConfig } from "eve/evals";

export default defineEvalConfig({});

The path under evals/ gives this eval its ID: orders/status-uses-tool.

The assertion checks the tool result's status field. Adapt it to your output schema:

import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";

export default defineEval({
  description: "Look up order 4417 and report its packing status.",
  tags: ["regression"],
  async test(t) {
    const turn = await t.send("Where is order 4417?");
    t.succeeded();
    t.calledTool("get_order_status", {
      input: { orderId: "4417" },
      output: { status: "packing" },
      count: 1,
    });
    t.check(turn.reply, includes("4417"));
    t.check(turn.reply, includes(/packing|being packed/i));
  },
});

Run the eval locally with the test agent's model credentials and fixture data. eve eval starts its own agent server and makes real model requests.

For preview runs, configure target authentication with a service credential and use the same test data. Protected deployments may need an automation bypass secret.

Replace the URL below with your preview deployment:

eve eval orders/status-uses-tool
eve eval orders --url https://order-agent-git-fix-status-my-team.vercel.app

The command exits with 0 when every gate passes and 1 when an eval fails. Configuration errors return 2. Add --strict to also fail when a soft assertion scores below its threshold.

These checks catch a missing lookup, an unexpected fixture result, or a reply that omits the packing status. The wording check is limited; use additional cases or a gated judge to assess whether the full answer agrees with the tool result.

Eval reporters export results separately from traces and do not use tracePolicy. Use fictional test data or a destination approved to store the eval content.

Add a reporter to send results to the same service as your traces:

import { defineEvalConfig } from "eve/evals";
import { braintrustReporter } from "eve/evals/reporters";

export default defineEvalConfig({
  reporters: [braintrustReporter({ projectName: "order-agent" })],
});

braintrustReporter() requires the braintrust package and BRAINTRUST_API_KEY. It exports the eval description, answer, errors, and assertion details. Use --skip-report to run evals without uploading results.

datadogReporter({ projectName }) sends results to a Datadog LLM Observability Experiment. Install dd-trace and set DD_API_KEY, DD_APP_KEY, and DD_SITE.

Datadog reporter defaults include prompts, outputs, expected answers, and errors. Configure its recording options for the content your destination may store.

Copy link to headingUpgrade notes for eve 1.0

This guide is written for eve 1.0, which is currently a release candidate (1.0.0-rc.1). If you're upgrading a project from eve 0.83 or earlier, these related APIs were renamed. The Upgrade to 1.0 guide has the full list.

Code that matches events by name, such as a stream reader or dashboard query, can compile cleanly and still never fire after the rename. Search for the old names in your instrumentation, scripts, and dashboards when you upgrade.

Copy link to headingNext steps

• Connect eve to other services with the Agent Stack guides.
• Review Agent Runs sampling and access for production tracing.
• Configure OpenTelemetry resources, samplers, and Node.js instrumentation.
• Write judge-graded and multi-turn checks with eve evals.
• Query runs from your terminal with vercel agent-runs.