Tag: Distributed Tracing

  • Routing Traces, Metrics, and Logs for LLM Agents (Pipelines + Exporters) | OpenTelemetry Collector

    Routing Traces, Metrics, and Logs for LLM Agents (Pipelines + Exporters) | OpenTelemetry Collector

    OpenTelemetry Collector for LLM agents: The OpenTelemetry Collector is the most underrated piece of an LLM agent observability stack. Instrumenting your agent runtime is step 1. Step 2 (the step most teams miss) is operationalizing telemetry: routing, batching, sampling, redaction, and exporting traces/metrics/logs to the right backend without rewriting every service.

    OpenTelemetry Collector for LLM agents

    If you are building agents with tool calling, RAG, retries, and multi-step plans, your system generates a lot of spans. The Collector lets you keep what matters (errors/slow runs) while controlling cost and enforcing governance centrally.

    TL;DR

    • Think of the Collector as a programmable telemetry router: OTLP in → processors → exporters out.
    • For LLM agents, the Collector is where you enforce consistent attributes like run_id, tool.name, prompt.version, llm.model, and tenant.
    • Use tail sampling so you keep full traces for failed/slow runs and downsample successful runs.
    • Implement redaction at the Collector layer so you never leak PII/secrets into your trace backend.
    • Export via OTLP/Jaeger/Tempo/Datadog/New Relic-without touching app code.

    Table of Contents

    What is the OpenTelemetry Collector?

    The OpenTelemetry Collector is a vendor-neutral service that receives telemetry (traces/metrics/logs), processes it (batching, filtering, sampling, attribute transforms), and exports it to one or more observability backends.

    Instead of configuring exporters inside every microservice/agent/tool, you standardize on sending OTLP to the Collector. From there, your team can change destinations, apply policy, and manage cost in one place.

    Why LLM agents need the Collector (not just SDK instrumentation)

    • Central policy: enforce PII redaction, attribute schema, and retention rules once.
    • Cost control: agents generate high span volume; the Collector is where sampling and filtering becomes practical.
    • Multi-backend routing: send traces to Tempo for cheap storage, but also send error traces to Sentry/Datadog/New Relic.
    • Reliability: buffer/batch/queue telemetry so your app doesn’t block on exporter issues.
    • Consistency: align tool services, background workers, and the agent runtime under one trace model.

    Collector architecture: receivers → processors → exporters

    The Collector is configured as pipelines:

    receivers  ->  processors  ->  exporters
(OTLP in)       (policy)       (destinations)

    Typical building blocks you’ll use for agent systems:

    • Receivers: otlp (gRPC/HTTP), sometimes jaeger or zipkin for legacy sources.
    • Processors: batch, attributes, transform, tail_sampling, memory_limiter.
    • Exporters: otlp/otlphttp to Tempo/OTel backends, Jaeger exporter, vendor exporters.

    A practical telemetry model for LLM agents

    Before you write Collector config, define a small attribute schema. This makes traces searchable and makes sampling rules possible.

    • Trace = 1 user request / 1 agent run
    • Span = a step (plan, tool call, retrieval, final response)
    • Key attributes (examples):
    • run_id: stable id you also log in your app
    • tenant / org_id: for multi-tenant systems
    • tool.name, tool.type, tool.status, tool.latency_ms
    • llm.provider, llm.model, llm.tokens_in, llm.tokens_out
    • prompt.version or prompt.hash
    • rag.top_k, rag.source, rag.hit_count (avoid raw content)

    Recommended pipelines for agents (traces, metrics, logs)

    Most agent teams should start with traces first, then add metrics/logs once the trace schema is stable.

    Minimal traces pipeline (starter)

    receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  batch:
    timeout: 2s
    send_batch_size: 2048

exporters:
  otlphttp/tempo:
    endpoint: http://tempo:4318

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/tempo]

    Agent-ready traces pipeline (attributes + tail sampling)

    This is where the Collector starts paying for itself: you keep the traces that matter.

    processors:
  attributes/agent:
    actions:
      # Example: enforce a standard service.name if missing
      - key: service.name
        action: upsert
        value: llm-agent

  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 200
    policies:
      # Keep all error traces
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      # Keep slow runs (e.g., total run > 8s)
      - name: slow
        type: latency
        latency:
          threshold_ms: 8000
      # Otherwise sample successful runs at 5%
      - name: probabilistic-success
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

    Tail sampling patterns for agent runs

    Agent systems are spiky: a single run can generate dozens of spans (planner + multiple tool calls + retries). Tail sampling helps because it decides after it sees how the trace ended.

    • Keep 100% of traces where error=true or span status is ERROR.
    • Keep 100% of traces where a tool returned 401/403/429/500 or timed out.
    • Keep 100% of traces where the run latency exceeds a threshold.
    • Sample the rest (e.g., 1-10%) for baseline performance monitoring.

    Redaction, governance, and safe logging

    LLM systems deal with sensitive inputs (customer text, internal docs, credentials). Your tracing stack must be designed for safety. Practical rules:

    • Never export secrets: API keys, tokens, cookies. Log references (key_id) only.
    • Redact PII: emails, phone numbers, addresses. Avoid raw prompts/tool arguments in production.
    • Separate data classes: store aggregated metrics longer; store raw prompts/traces on short retention.
    • RBAC: restrict who can view tool arguments, retrieved snippets, and prompt templates.
    • Auditability: keep enough metadata to answer “who/what/when” without storing raw payloads.

    Deployment options and scaling

    • Sidecar: best when you want per-service isolation; simpler network policies.
    • DaemonSet (Kubernetes): good default; each node runs a Collector instance.
    • Gateway: centralized Collectors behind a load balancer; good for advanced routing and multi-tenant setups.

    Also enable memory_limiter + batch to avoid the Collector becoming the bottleneck.

    Troubleshooting and validation

    • Verify your app exports OTLP: you should see spans in the backend within seconds.
    • If traces are missing, check network (4317 gRPC / 4318 HTTP) and service discovery.
    • Add a temporary logging exporter in non-prod to confirm the Collector receives data.
    • Ensure context propagation works across tools; otherwise traces will fragment.

    Tools & platforms (official + GitHub links)

    Production checklist

    • Define a stable trace/attribute schema for agent runs (run_id, tool spans, prompt version).
    • Route OTLP to the Collector (don’t hard-code exporters per service).
    • Enable batching + memory limits.
    • Implement tail sampling for errors/slow runs and downsample success.
    • Add redaction rules + RBAC + retention controls.
    • Validate end-to-end trace continuity across tool services.

    FAQ

    Do I need the Collector if I already use an APM like Datadog/New Relic?

    Often yes. The Collector lets you enforce sampling/redaction and route telemetry cleanly. You can still export to your APM-it becomes one destination rather than the only architecture.

    Should I store prompts and tool arguments in traces?

    In production, avoid raw payloads by default. Store summaries/hashes and only enable detailed logging for short-lived debugging with strict access control.

    Related reads on aivineet

    OpenTelemetry Collector for LLM agents is especially useful for agent systems where you need to debug tool calls and control telemetry cost with tail sampling.

  • Lightweight Distributed Tracing for Agent Workflows (Quick Setup + Visibility) | Zipkin

    Lightweight Distributed Tracing for Agent Workflows (Quick Setup + Visibility) | Zipkin

    Zipkin for LLM agents: Zipkin is the “get tracing working today” option. It’s lightweight, approachable, and perfect when you want quick visibility into service latency and failures without adopting a full observability suite.

    Zipkin for LLM agents

    For LLM agents, Zipkin can be a great starting point: it helps you visualize the sequence of tool calls, measure step-by-step latency, and detect broken context propagation. This guide covers how to use Zipkin effectively for agent workflows, and when you should graduate to Jaeger or Tempo.

    TL;DR

    • Zipkin is a lightweight tracing backend for visualizing end-to-end latency.
    • Model 1 agent request as 1 trace; model tool calls as spans.
    • Add run_id + tool.name attributes so traces are searchable.
    • Start with Zipkin for small systems; move to Tempo/Jaeger when volume/features demand it.

    Table of Contents

    What Zipkin is good for

    • Small to medium systems where you want quick trace visibility.
    • Understanding latency distribution across steps (model call vs tool call).
    • Detecting broken trace propagation across services.

    How to model agent workflows in traces

    Keep it simple and consistent:

    • Trace = one agent run (one user request)
    • Spans = planner, tool calls, retrieval, final compose
    • Attributes = run_id, tool.name, http.status_code, retry.count, llm.model, prompt.version

    Setup overview: OTel → Collector → Zipkin

    A clean approach is to use OpenTelemetry everywhere and export to Zipkin via the Collector:

    receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  zipkin:
    endpoint: http://zipkin:9411/api/v2/spans

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [zipkin]

    Debugging tool calls and retries

    • Slow agent? Find the longest span. If it’s a tool call, inspect status/timeout/retries.
    • Incorrect output? Trace helps you confirm which tools were called and in what order.
    • Fragmented traces? That’s usually missing context propagation across tools.

    When to move to Jaeger or Tempo

    • Move to Jaeger when you want a more full-featured tracing experience and broader ecosystem usage.
    • Move to Tempo when trace volume becomes high and you want object-storage economics.

    Privacy + safe logging

    • Don’t store raw prompts and tool arguments by default.
    • Redact PII and secrets at the Collector layer.
    • Use short retention for raw traces; longer retention for derived metrics.

    Tools & platforms (official + GitHub links)

    Production checklist

    • Add run_id to traces and your app logs.
    • Instrument planner + each tool call as spans.
    • Validate context propagation so traces don’t fragment.
    • Use the Collector for batching and redaction.
    • Revisit backend choice when volume grows (Jaeger/Tempo).

    Related reads on aivineet

    Zipkin for LLM agents helps you debug tool calls, measure per-step latency, and keep distributed context across services.

  • Storing High-Volume Agent Traces Cost-Efficiently (OTel/Jaeger/Zipkin Ingest) | Grafana Tempo

    Storing High-Volume Agent Traces Cost-Efficiently (OTel/Jaeger/Zipkin Ingest) | Grafana Tempo

    Grafana Tempo for LLM agents: Grafana Tempo is built for one job: store a huge amount of tracing data cheaply, with minimal operational complexity. That matters for LLM agents because agent runs can generate a lot of spans: planning, tool calls, retries, RAG steps, and post-processing.

    Grafana Tempo for LLM agents

    In this guide, we’ll explain when Tempo is the right tracing backend for agent systems, how it ingests OTel/Jaeger/Zipkin protocols, and how to design a retention strategy that doesn’t explode your bill.

    TL;DR

    • Tempo is great when you have high trace volume and want object storage economics.
    • Send OTLP to an OpenTelemetry Collector, then export to Tempo (simplest architecture).
    • Store raw traces short-term; derive metrics (spanmetrics) for long-term monitoring.
    • Use Grafana’s trace UI to investigate slow/failed agent runs and drill into tool spans.

    Table of Contents

    When Tempo is the right choice for LLM agents

    • Your agents generate high span volume (multi-step plans, retries, tool chains).
    • You want cheap long-ish storage using object storage (S3/GCS/Azure Blob).
    • You want to explore traces in Grafana alongside metrics/logs.

    If you’re early and want the classic standalone tracing UI experience, Jaeger may feel simpler. Tempo shines once volume grows and cost starts to matter.

    Ingest options: OTLP / Jaeger / Zipkin

    Tempo supports multiple ingestion protocols. For new agent systems, standardize on OTLP because it keeps you aligned with OpenTelemetry across traces/metrics/logs.

    • OTLP: recommended (agent runtime + tools export via OpenTelemetry SDK)
    • Jaeger: useful if you already have Jaeger clients
    • Zipkin: useful if you already have Zipkin instrumentation

    Reference architecture: Agent → Collector → Tempo → Grafana

    Agent runtime + tool services (OTel SDK)
   -> OpenTelemetry Collector (batch + tail sampling + redaction)
      -> Grafana Tempo (object storage)
         -> Grafana (trace exploration + correlations)

    This design keeps app code simple: emit OTLP only. The Collector is where you route and apply policy.

    Cost, retention, and sampling strategy

    Agent tracing can become expensive because each run can produce dozens of spans. A cost-safe approach:

    • Tail sample: keep 100% of error traces + slow traces; downsample successful traces.
    • Short retention for raw traces: e.g., 7-30 days depending on compliance.
    • Long retention for metrics: derive RED metrics (rate, errors, duration) from traces and keep longer.

    Debugging agent runs in Grafana (trace-first workflow)

    • Search by run_id (store it as an attribute on root span).
    • Open the trace timeline and identify the longest span (often a tool call or a retry burst).
    • Inspect attributes: tool status codes, retry counts, model, prompt version, and tenant.

    Turning traces into metrics (SLOs, alerts, dashboards)

    Teams often struggle because “agent quality” is not a single metric. A practical approach is:

    • Define success/failure at the end of the run (span status and/or custom attribute like agent.outcome).
    • Export span metrics (duration, error rate) to Prometheus/Grafana for alerting.
    • Use trace exemplars: alerts should link to sample traces.

    Privacy + governance for trace data

    • Avoid raw prompts/tool payloads by default; store summaries/hashes.
    • Use redaction at the Collector layer.
    • Restrict access to any fields that might contain user content.

    Tools & platforms (official + GitHub links)

    Production checklist

    • Standardize on OTLP from agent + tools.
    • Use the Collector for tail sampling + redaction + batching.
    • Store run_id, tool.name, llm.model, prompt.version for trace search.
    • Define retention: raw traces short, derived metrics long.
    • Make alerts link to example traces for fast debugging.

    Related reads on aivineet

    Grafana Tempo for LLM agents helps you debug tool calls, measure per-step latency, and keep distributed context across services.

  • Debugging LLM Agent Tool Calls with Distributed Traces (Run IDs, Spans, Failures) | Jaeger

    Debugging LLM Agent Tool Calls with Distributed Traces (Run IDs, Spans, Failures) | Jaeger

    Jaeger for LLM agents: Jaeger is one of the easiest ways to see what your LLM agent actually did in production. When an agent fails, the final answer rarely tells you the real story. The story is in the timeline: planning, tool selection, retries, RAG retrieval, and downstream service latency.

    Jaeger for LLM agents

    In this guide, we’ll build a practical Jaeger workflow for debugging tool calls and multi-step agent runs using OpenTelemetry. We’ll focus on what teams need in real systems: searchability (run_id), safe logging, and fast incident triage.

    TL;DR

    • Trace = 1 user request / 1 agent run.
    • Span = each step (plan, tool call, retrieval, final).
    • Add run_id, tool.name, llm.model, prompt.version as span attributes so Jaeger search works.
    • Keep 100% of error traces (tail sampling) and downsample the rest.
    • Don’t store raw prompts/tool args in production by default; store summaries/hashes + strict RBAC.

    Table of Contents

    What Jaeger is (and what it is not)

    Jaeger is an open-source distributed tracing backend. It stores traces (spans), provides a UI to explore timelines, and helps you understand request flows across services.

    Jaeger is not a complete observability platform by itself. Most teams pair it with metrics (Prometheus/Grafana) and logs (ELK/OpenSearch/Loki). For LLM agents, Jaeger is the best “trace-first” entry point because timelines are how agent failures present.

    Why Jaeger is great for agent debugging

    • Request narrative: agents are sequential + branching systems. Traces show the narrative.
    • Root-cause speed: instantly spot if the tool call timed out vs. the model stalled.
    • Cross-service visibility: planner service → tool service → DB → third-party API, all in one view.

    Span model for tool calling and RAG

    Start with a consistent span naming convention. Example:

    trace (run_id=R123)
  span: agent.plan
  span: llm.generate (model=gpt-4.1)
  span: tool.search (tool.name=web_search)
  span: tool.search.result (http.status=200)
  span: rag.retrieve (top_k=10)
  span: final.compose

    Recommended attributes (keep them structured):

    • run_id (critical: makes incident triage fast)
    • tool.name, tool.type, tool.status, http.status_code
    • llm.provider, llm.model, llm.tokens_in, llm.tokens_out
    • prompt.version or prompt.hash
    • rag.top_k, rag.source, rag.hit_count (avoid raw retrieved content)

    The cleanest workflow is: your app logs a run_id for each user request, and Jaeger traces carry the same attribute. Then you can search Jaeger by run_id and open the exact trace in seconds.

    • Log run_id at request start and return it in API responses for support tickets.
    • Add run_id as a span attribute on the root span (and optionally all spans).
    • Use Jaeger search to filter by run_id, error=true, or tool.name.

    Common failure patterns Jaeger reveals

    1) Broken context propagation (fragmented traces)

    If tool calls run as separate services, missing trace propagation breaks the timeline. You’ll see disconnected traces instead of one end-to-end trace. Fix: propagate trace headers (W3C Trace Context) into tool HTTP calls or internal RPC.

    2) “Tool call succeeded” but agent still failed

    This often indicates parsing/validation issues (schema mismatch), prompt regression, or poor retrieval. The trace shows tool latency is fine; failure happens in the LLM generation span or post-processing span.

    3) Slow runs caused by retries

    Retries add up. In Jaeger, you’ll see repeated tool spans. Add attributes like retry.count and retry.reason to make it obvious.

    Setup overview: OTel → Collector → Jaeger

    A simple production-friendly architecture is:

    Agent Runtime (OTel SDK)  ->  OTel Collector  ->  Jaeger (storage + UI)

    Export OTLP from your agent to the Collector, apply tail sampling + redaction there, and export to Jaeger.

    Privacy + redaction guidance

    • Do not store raw prompts/tool arguments by default in production traces.
    • Store summaries, hashes, or classified metadata (e.g., “contains_pii=true”) instead.
    • Keep detailed logging behind feature flags, short retention, and strict RBAC.

    Tools & platforms (official + GitHub links)

    Production checklist

    • Define a span naming convention + attribute schema (run_id, tool attributes, model info).
    • Propagate trace context into tool calls (headers/middleware).
    • Use tail sampling to keep full traces for failures/slow runs.
    • Redact PII/secrets and restrict access to sensitive trace fields.
    • Train the team on a basic incident workflow: “get run_id → find trace → identify slow/error span → fix.”

    FAQ

    Jaeger vs Tempo: which should I use?

    If you want a straightforward tracing backend with a classic trace UI, Jaeger is a strong default. If you expect very high volume and want object-storage economics, Tempo can be a better fit (especially with Grafana).

    Related reads on aivineet

    Jaeger for LLM agents helps you debug tool calls, measure per-step latency, and keep distributed context across services.

  • LLM Agent Tracing & Distributed Context: End-to-End Spans for Tool Calls + RAG | OpenTelemetry (OTel)

    OpenTelemetry (OTel) is the fastest path to production-grade tracing for LLM agents because it gives you a standard way to follow a request across your agent runtime, tools, and downstream services. If your agent uses RAG, tool calling, or multi-step plans, OTel helps you answer the only questions that matter in production: what happened, where did it fail, and why?

    In this guide, we’ll explain how to instrument an LLM agent with end-to-end traces (spans), how to propagate context across tool calls, and how to store + query traces in backends like Jaeger/Tempo. We’ll keep it practical and enterprise-friendly (redaction, auditability, and performance).

    TL;DR

    • Trace everything: prompt version → plan → tool calls → tool outputs → final answer.
    • Use trace context propagation so tool calls remain linked to the parent run.
    • Model “one user request” as a trace, and each agent/tool step as a span.
    • Export via OTLP to an OpenTelemetry Collector, then route to Jaeger/Tempo or your observability stack.
    • Redact PII and never log secrets; keep raw traces on short retention.

    Table of Contents

    What is OpenTelemetry (OTel)?

    OpenTelemetry is an open standard for collecting traces, metrics, and logs. In practice, OTel gives you a consistent way to generate and export trace data across services. For LLM agents, that means you can follow a single user request through:

    • your API gateway / app server
    • agent planner + router
    • tool calling (search, DB, browser, CRM)
    • RAG retrieval and reranking
    • final synthesis and formatting

    Why agents need distributed tracing

    Agent failures rarely show up in the final answer. More often, the issue is upstream: a tool returned a 429, the model chose the wrong tool, or retrieval returned irrelevant context. Therefore, tracing becomes your “black box recorder” for agent runs.

    • Debuggability: see the exact tool call sequence and timing.
    • Reliability: track where latency and errors occur (per tool, per step).
    • Governance: produce audit trails for data access and actions.

    A trace model for LLM agents (runs, spans, events)

    Start with a simple mapping:

    • Trace = 1 user request (1 agent run)
    • Span = a step (plan, tool call, retrieval, final response)
    • Span attributes = structured fields (tool name, status code, prompt version, token counts)
    trace: run_id=R123
  span: plan (prompt_version=v12)
  span: tool.search (q="...")
  span: tool.search.result (status=200, docs=8)
  span: rag.retrieve (top_k=10)
  span: final.compose (schema=AnswerV3)

    Distributed context propagation for tool calls

    The biggest mistake teams make is tracing the agent runtime but losing context once tools run. To keep spans connected, propagate trace context into tool requests. For HTTP tools this is typically done via headers, and for internal tools it can be done via function parameters or middleware.

    • Use trace_id/span_id propagation into each tool call.
    • Ensure tool services also emit spans (or at least structured logs) with the same trace_id.
    • As a result, your trace UI shows one end-to-end timeline instead of disconnected fragments.

    Tracing RAG: retrieval, embeddings, and citations

    RAG pipelines introduce their own failure modes: missing documents, irrelevant retrieval, and hallucinated citations. Instrument spans for:

    • retrieval query + filters (redacted)
    • top_k results and scores (summaries, not raw content)
    • reranker latency
    • citation coverage (how much of the answer is backed by retrieved text)

    Privacy, redaction, and retention

    • Never log secrets (keys/tokens). Store references only.
    • Redact PII from prompts/tool args (emails, phone numbers, addresses).
    • Short retention for raw traces; longer retention for aggregated metrics.
    • RBAC for viewing prompts/tool args and retrieved snippets.

    Tools & platforms (official + GitHub links)

    Production checklist

    • Define run_id and map 1 request = 1 trace.
    • Instrument spans for plan, each tool call, and final synthesis.
    • Propagate trace context into tool calls (headers/middleware).
    • Export OTLP to an OTel Collector and route to your backend.
    • Redact PII + enforce retention and access controls.

    FAQ

    Do I need an OpenTelemetry Collector?

    Not strictly, but it’s the cleanest way to route OTLP data to multiple backends (Jaeger/Tempo, logs, metrics) without rewriting your app instrumentation.

    Related reads on aivineet