Schedule

AI is changing how engineers build, operate, and troubleshoot software. But AI is only as effective as the production context it can access. Today, that context is fragmented across traces, logs, metrics, infrastructure, deployments, and runtime behavior, leaving AI systems without the complete picture needed to understand what's actually happening in production.

This keynote explores why runtime context is becoming the foundation of the next generation of software operations.

We'll discuss how OpenTelemetry has standardized telemetry collection, why eBPF is making production visibility dramatically easier, and how organizations are moving beyond simply collecting more data toward delivering rich, connected runtime context that both engineers and AI systems can reason about.

Attendees will learn:

1. Why runtime context is becoming the new operational data layer.
2. How OpenTelemetry and eBPF are reducing friction in production visibility.
3. Why dynamic telemetry matters more than static instrumentation.
4. How rich runtime context enables AI to troubleshoot, investigate, and automate production operations with greater confidence.

As AI becomes part of every engineering workflow, organizations that can provide complete, trustworthy runtime context will be best positioned to unlock the next generation of intelligent operations.

Continuous profiling connects resource consumption to the code responsible for it, adding a code-level view alongside metrics, logs, and traces. The OpenTelemetry eBPF Profiler extends that view across Linux systems, crossing languages, runtimes, and shared libraries without requiring application changes or restarts. Alongside the profiler, the community is shaping the OpenTelemetry Profiles signal to create a common way to represent and transport profiling data.

This talk explores what profiling adds to observability, what we are learning by building the profiler and signal collaboratively, and how together they can make profiling open, portable, and broadly accessible.

OpenTelemetry helps us measure latency, errors, and availability. But as sustainability moves into engineering, one question remains unanswered: how do we observe carbon?

This talk explores Software Carbon Intensity (SCI) through the lens of OpenTelemetry semantic conventions. Rather than focusing on sustainability reporting, we examine the technical challenge of modeling carbon as telemetry: what entities should emissions be attributed to, which attributes should be standardized, and whether carbon belongs in metrics, traces, resource metadata, or all three.

We will explore how carbon-related signals can be collected, enriched, and visualized alongside traditional observability data. Attendees will gain a deeper understanding of semantic conventions, telemetry design trade-offs, and the opportunities for OpenTelemetry to support a new generation of operational signals.

The audience will leave with these take-aways:
* Understanding SCI as an observability signal
* Designing semantic conventions for carbon telemetry
* Trade-offs between metrics, traces, and resource attributes
* Practical implementation patterns using OpenTelemetry
* Future directions for green observability

Your tests pass. You enable a feature for a small group of users. What evidence justifies expanding the rollout? This five-minute keynote follows one illustrative change from feature flag to production feedback. OpenFeature provides a standard interface for evaluating flags. OpenTelemetry instrumentation connects flag evaluations with runtime behavior. Together, they help teams investigate which requests received a change and decide whether to expand exposure or disable the feature. The same discipline applies whether a human or an AI coding agent wrote the code. Attendees leave with a practical release habit: define the evidence for proceeding, limit initial exposure, and verify the outcome before taking the next step.

"Noisy neighbour" used to mean a container stealing CPU. Inside an LLM server, the neighbour is sitting on the same GPU, in the same batch, on the same iteration as you. Servers like vLLM and TGI use continuous batching – your request is fused onto the GPU alongside strangers', decoded token by token, in a batch that reshuffles every step. So when your p99 spikes, it's often not a slow request at all. It's a batch-mate generating 4,000 tokens, or the KV cache thrashing and preempting you. Your per-request trace can't see any of it.

This is an SRE's field guide to observing LLM inference honestly. We'll cover why classic RED and latency dashboards mislead you here, and the signals that actually matter: time-to-first-token vs inter-token latency, KV cache utilization and prefix-cache hit rate, queue depth, and batch occupancy. Using DCGM, Prometheus, and inference-server metrics, we'll trace a real serving stack and pin a latency spike on its true cause – scheduling, memory, or that neighbour.

You'll leave able to debug what dashboards can't tell you today: whose request is actually slow, and why?

Modern production systems are highly observable—until the thing that is broken is not your infrastructure.

Your dashboards are green. Latency is stable, error rates are low, and every infrastructure metric says your systems are healthy. Yet customers are quietly failing, and no one knows until they report it.

At TrueLayer, we realised we weren't missing telemetry—we were missing the right signals.

This talk walks through how we evolved from simple volume monitoring into a business-focused observability framework that detects subtle drops in customer activity before they become incidents.

We'll explore why traditional metrics weren't enough, how we balanced statistical accuracy with operational usefulness and how we combined prediction models with confidence bands to reduce alert fatigue.

Along the way, I'll share the lessons learned about defining "normal", avoiding noisy alerts, and building monitoring that engineers and customer-facing teams could both trust.

Whether you run distributed systems, APIs or SaaS platforms, you'll leave with practical ideas for making observability reflect the health of your customers—not just your infrastructure.

OpenTelemetry has graduated and is fulfilling its vision as the de facto standard for observability. It is now becoming the backbone for security, development, and infrastructure practices.

This talk explores three emerging use cases that demonstrate OpenTelemetry's expanding role:

• Security Integration: We show how OpenTelemetry interfaces with OCSF (Open Cybersecurity Schema Framework) events, enabling real-time threat detection.
• CI/CD and Deployment Validation: OpenTelemetry serves as a foundation for modern software delivery, enabling teams to place assertions on system behavior during deployments and track how applications evolve over time.
• Network Device Management: We demonstrate how OpenTelemetry standardizes network equipment management by integrating with YANG models, gRPC, and gNMI, bringing telemetry to networking devices.

Attendees will learn practical integration patterns to extend their OpenTelemetry investment beyond traditional APM use cases.

OpenTelemetry gave us a standard for telemetry data. OpAMP gave us a standard for controlling the agents that collect it. MCP now gives AI agents structured access to both. Put the three together and a long-promised idea becomes testable: telemetry pipelines that tune themselves.

At Nike, we operate a fleet of 100,000 OpenTelemetry Collectors managed through an enterprise-grade OpAMP control plane (presented at KubeCon NA 2025 http://youtube.com/watch?v=J68ThM9DqQ0). This talk covers what happened when we let an AI agent into that loop: exposing our telemetry backends and fleet state through MCP servers, and allowing an agent to propose and, within strict guardrails, apply configuration changes via OpAMP. Think detecting a cardinality explosion and drafting a filtering rule, or raising trace sampling on a service the moment an incident is declared.

We'll demo the closed loop end to end: the approval gates and blast-radius limits we enforce, the failure modes we hit, and the line we currently refuse to let an AI agent cross. Everything shown is built on open-source protocols like OTLP, OpAMP, and MCP; vendor-agnostic and portable to any telemetry backend.

How much energy is this pod actually consuming? Most Kubernetes energy metrics are modeled from CPU utilization because hardware energy counters cannot be attributed to a single workload. This case study presents a fully open-source path to measured per-pod energy, closing a critical observability gap on the path to making agentic AI more energy-efficient.

The Linux 7.0 resctrl filesystem now exposes real energy telemetry, and this session shows how to attribute it to individual Kubernetes workloads — the missing input for energy-aware scheduling, per-tenant billing, and workload-level efficiency tuning in AI-focused data centers. The key is binding each pod to a monitoring group through NRI container-lifecycle hooks, which captures a workload's processes reliably where userspace polling daemons miss short-lived and forking threads.

Attendees will follow the complete path from a silicon counter to a live Grafana panel, see how energy monitoring drops into an existing cluster without changing how workloads are scheduled or resourced, and learn how to deploy it themselves with a Helm chart.

AI is generating more code than ever and that's making the maintainer crisis worse, not better. Every AI-assisted PR still needs a human to review it, context to evaluate it, and trust to merge it. The bottleneck was never code production. It's people.

In the age of AI, mentorship matters more than ever. Real humans building real relationships with maintainers, that's what sustains a project. It takes partnership: internal shepherds who invest in their people, and maintainers who open the door.

We tested this with NumFOCUS and pandas, then expanded it: CNCF, OpenTelemetry, and Bloomberg ran a 10-week mentorship cohort: 48 engineers, 7 maintainers, weekly office hours, curated issues. The results? 55 PRs merged across 11 repos. A participant became the 2nd largest human contributor to the OTel Demo. Three teammates built a Collector extension that will outlive the cohort.

What we learned: the bottleneck is issue discovery, not motivation. Maintenance work is satisfying when people understand why it matters. The model is built on trust between organizations and the communities they depend on.

We'll share the blueprint, the mistakes, and what it takes to build this together.

OpenTelemetry's metrics performance has long been a sore point for high-throughput users. Recording a simple Counter with three attributes/labels cost around 50 nanoseconds per call – enough to show up in CPU profiles and rule OpenTelemetry out of the busiest code paths.

Pre-resolved metric handles are an old answer. Prometheus client libraries expose labelled-metric handles you can cache, and Windows Performance Counters have used the same pattern for decades. OpenTelemetry now offers it as a first-class API, called bound instruments, and on that same hot path, recording drops to under 2 nanoseconds – roughly 25× faster (Measured in OTel Rust Sdk)

This lightning talk shows where the speedup comes from, when to reach for it (high-frequency counters with a fixed, known attribute set), and – more importantly – when not to. Used the wrong way, the new fast path can be slower than the original.

Flexible by default, fast when you need it.

This talk explores a field-tested example of OpenTelemetry Collector's multi-topology design working as intended. OpenTelemetry Collectors are deployed as sidecars at scale – co-located with the workloads they observe, regardless of the tenancy model.

It offers a practical path to architectural consistency without forcing a one-size-fits-all deployment topology. Whether a service runs in a shared cluster serving many clients or in an isolated environment dedicated to one, the sidecar pattern keeps telemetry collection close to the source while decoupling it from the underlying infrastructure differences between tenancy models.

The real payoff, however, isn't just architectural symmetry – it's semantic consistency. Deploying collectors uniformly is only half the story; enforcing shared semantic conventions across every collector instance is what produces a coherent picture at scale.

This session is aimed at platform engineers, SREs, and observability practitioners responsible for telemetry strategy across complex, multi-deployment-model organizations.

At Engineering Ingegneria Informatica, we run an internal developer portal for 5,000 developers, provisioning ephemeral Kubernetes virtual clusters on demand.

Wiring a large language model agent to live telemetry through the Model Context Protocol (MCP) promises a tempting shortcut: ask "why did latency double last night?" in plain language instead of pivoting across a dozen dashboards. This ten-minute talk is a candid field report on what happened when we actually tried it, connecting an LLM agent to an OpenTelemetry-backed observability stack over MCP.

I'll cover the parts that genuinely worked, then spend most of the time on the parts that didn't: confidently hallucinated root causes, the access-control gaps that surface the moment an agent can read operational data, and query costs that scale in surprising ways. The uncomfortable takeaway is that telemetry standardization and data governance mattered far more than which model we picked.

You'll leave with a realistic mental model and concrete guardrails to put in place before you point an agent at your own production signals.

Mobile networks are typically imagined as wireless networks built on old-school telecom routing equipment. In reality, the core (in 5G, the 5G Core) is a fully cloud-native, microservice-based architecture, communicating over the same APIs and protocols as any modern distributed system. Yet unlike the cloud-native world, there's no standard for distributed tracing or any other observability in the 5GC.

Distributed tracing and OpenTelemetry emerge as the missing layer for 5G Core observability. Drawing on experience instrumenting an open-source 5G Core (Open5GS), this usecase comparse three approaches to capturing traces from telecom-deployments: manual SDK instrumentation, service mesh (Istio Ambient), and eBPF-based instrumentation (Grafana Beyla). Did you know mobile-network-specific identifiers like PDU session and IMSI are also enable post-hoc context propagation?

Beyond capturing traces in mobile networks, the question is what to do with them. Tracing and metrics feed a causal model linking control-plane messages to downstream network effects, enabling automated root-cause analysis and self-healing operations in tele-operation scenarios like remotely controlling ships.

How do teams monitor availability and response time SLAs? Standard HTTP metrics from their applications may not tell the whole story.

Between a customer's request and the service handling it sit several intermediaries. When a request is dropped at the CDN, rejected by the gateway, or timed out before it reaches a pod, no application metric records it. Dashboards may stay green while customers feel impact and error budgets deplete. Even where edge error logs exist, attributing them to the right team in a microservice architecture is non-trivial.

This session walks through a production OpenTelemetry Collector configuration that turns raw edge logs (Cloudflare, Envoy Gateway) into Prometheus metrics matching existing application metrics – an equivalent "customer's-eye view" of availability and response times.

The session also demonstrates how Kubernetes Gateway API route definitions are used to auto-generate processor rules that attribute every edge request to an owning service, product, and therefore team – so alerts route to the right on-call without a hand-maintained mapping.

Attendees will leave with a blueprint for edge observability built on CNCF projects alone.

What is Developer Experience to you?

For the OpenTelemetry Developer Experience SIG, the initial answer seemed obvious: better SDK configuration, clearer API usage, and smoother onboarding. But when we asked the community what they were struggling with, we actually discovered that our assumptions were only part of the story.
Users already knew how to get started. What they needed was help getting to production.

They wanted real use cases, production-ready examples, Collector deployment patterns, and scaling guidance. In response, the SIG began interviewing companies of different sizes and publishing their production OTel stories, including examples from Mastodon, Adobe, Skyscanner, and others.

In this talk, we will share how our understanding of developer experience changed through surveys, interviews, and community feedback. We will highlight lessons from real-world OTel adoption: what teams are doing well, where they still struggle, and which patterns are emerging across different environments.
Join us to learn what comes next for the DevEx SIG and how real production stories can shape the future of OpenTelemetry developer experience.

What if application logs did not have to start in a logging library queue?

Linux user_events lets applications register tracepoints and write structured events into Linux tracing. When a listener is enabled, events flow through per-CPU kernel/perf buffers. When no listener is attached, producers can cheaply skip the write path.

This talk shares lessons from building both sides of this path: a userevents log exporter for the OTel Rust SDK and a userevents receiver for the OTel-Arrow Dataflow Engine. An application writes on CPU X, the kernel stores the event in CPU X’s perf ring, and a pipeline pinned to CPU X drains that ring.

We will cover listener checks, bounded kernel and receiver-side buffering, lost-sample visibility, tracefs metadata decoding, optional EventHeader decoding, batching, CPU coverage, and NUMA locality.

The goal is to show how Linux-native application events can become structured OpenTelemetry logs while preserving backpressure, locality, and operability.

Writing OpenTelemetry instrumentation libraries is usually fun. Testing that the output adheres to the semantic conventions or reviewing that changes to the instrumentations don't break anything less so. In this talk we'll see how we can leverage OpenTelemetry Weaver, the jack of all trades for OpenTelemetry semantic conventions, to properly validate the telemetry exported.

Localization is a powerful, although often overlooked, gateway to open-source contribution. For projects like OpenTelemetry, translating documentation forces you to dive deep into the project, understand the core terminology, and navigate your first pull request.

It is a brilliant way to build confidence, until the friction hits. Managing formatting, context-switching, and OTel glossary alignment can easily turn a single-page translation into a repetitive 4-hour task. The cognitive toil is real, and it can cause localization efforts to stall.

What if we could automate the drag while preserving the human connection?

In this lightning talk, we will explore an AI-powered workflow that abstracts away the mechanical overhead of OpenTelemetry documentation localization while keeping the human in the loop.

And we will close with an open question for the ecosystem: If we lower the friction of contributing through automation, do we dilute the learning experience? Or can we actually make localization efforts sustainable, allowing new contributors to focus on learning about the project rather than fighting the linter?

Distributed tracing usually assumes context travels in HTTP headers or message metadata. Yet, many workflows cross process boundaries instead: CI jobs launch scripts, workflow engines start containers, and build tools fork subprocesses, losing trace context.

This talk presents the use of OpenTelemetry environment variables propagation carriers, showing how traces can flow across CLI apps, CI/CD systems, build pipelines, and subprocesses without relying on network transports. The carrier model supports multiple propagation formats, enabling interoperability without reliance on a single ecosystem-specific encoding.

Through two demos it demonstrates how injected TRACEPARENT enables end-to-end tracing, linking stages, subprocesses, and even data lineage in a single span tree. Attendees will gain a practical model for maintaining trace continuity beyond network protocols without custom formats.

Two years ago I sent two agents into the arena: Fluent Bit and the OpenTelemetry Collector. Since then the games have changed. Fluent Bit v5 is back stronger, and a new gladiator has entered the gate: OTel Arrow, built to move telemetry at scale for a fraction of the bandwidth.

So I am reopening the Colosseum, and this time three agents fight: the OTel Collector, Fluent Bit v5, and Otel-Arrow. I run all three under identical Kubernetes conditions and judge them on 5 trials:

• design
• architecture
• signal support across metrics, logs, and traces
• self-observability, the health telemetry each exposes by each agent
• and performance under load (CPU, memory, throughput, latency).

Run independently and vendor-neutral with the OTel-Arrow maintainers, the benchmark shows the crown depends on the arena you fight in. All configs and results will be published on GitHub.

Observability pipelines are the nervous system of modern cloud-native infrastructure. However, the streaming platforms, vector databases, and log aggregators powering these environments have become high-value targets for threat actors. This session presents a real-world case study centered on security research that uncovered and reported critical vulnerabilities across enterprise streaming architectures and API infrastructures. The presentation explores how systemic architectural misconfigurations can expose data pipelines to severe exploits like DNS rebinding, Server-Side Request Forgery (SSRF), and malicious payload manipulation.

Moving beyond the exploits, the case study provides a detailed account of the remediation and hardening process. Attendees will learn practical strategies for integrating robust DevSecOps practices directly into their observability deployments. The speaker outlines how to enforce secure network boundaries around centralized data stores and utilize open-source tools like Prometheus and Fluentd to actively monitor the pipeline's health, ensuring the integrity of the telemetry stream against malicious exploitation.

AI agents are beginning to participate in operational workflows: reading logs, querying traces, searching runbooks, opening incidents, and proposing actions. Traditional observability tells us what services did, and LLM observability shows prompts/model calls, but neither fully explains why an agent chose a tool, which evidence it trusted, or how a policy result shaped its recommendation.

This talk introduces decision traces: structured, OpenTelemetry-style events that connect incidentid, sessionid, trace_id, agent step, tool call, evidence reference, policy result, cost, and outcome. We walk through a checkout-api latency incident handled by a controlled multi-agent workflow—triage, log, trace, retrieval, planner, policy-recording, and eval agents—and show how the decision path becomes searchable operational evidence.

Attendees will learn a vendor-neutral architecture using OpenTelemetry Collector pipelines, LLM observability tools such as Langfuse/LangSmith for model traces, and an observability backend such as OpenSearch/Grafana stack for decision analytics, audit, and runaway-loop detection.

Most observability tooling quietly assumes an always-on connection to a cloud backend. But a large and often-ignored slice of the world runs on-prem and disconnected by necessity: regulated industries, government, defense, financial systems, and any environment where data cannot leave the perimeter. For these teams, "just send it to our SaaS" is a non-starter, yet they need modern observability just as much as everyone else, arguably more, because their operational and compliance stakes are higher.

This session confronts the air-gapped and on-prem observability problem head-on. I'll start from the reality that many enterprises will keep a meaningful portion of workloads on-premises for the foreseeable future, and that these workloads still deserve to be first-class citizens in an OpenTelemetry-centric world. From there, we'll dig into the architectural patterns that make open-source observability work without a cloud dependency.

We'll also cover: data redaction and residency, delivering telemetry in fully disconnected deployments, and designing so that the same software can serve both a connected cloud customer and an air-gapped one without forking the architecture.

Managing 250+ MySQL clusters and 10,000+ nodes requires robust observability. With Graphite not actively maintained in 2026, moving our metrics stack to OpenTelemetry became a priority to eliminate operational risks and align with industry standards. This was a massive redesign with unique challenges:

New Architecture: Replace heterogeneous/legacy collectors with a unified stack of well-connected components. Standardize on the OTEL collector as the single proxy on every database node pushing metrics to Grafana Mimir, orchestrated by Puppet.

Data Model: Use the OTEL SDK with delta temporality to replicate event-driven pushes and navigate Prometheus’ 5-minute staleness logic. Safely switch between Graphite and Mimir backends via a runtime config. Convert Graphite paths to OTEL attributes using Prometheus' file-based service discovery.

Mimir Tenant Tuning: Operating at ~10M active series required us to adjust query limits and ingestion rates to avoid rate limiting. OTTL functions allowed us to prevent cardinality explosion.

Safe Rollouts: Using the Grafana API we sorted dashboards based on popularity. Explain why AI was not enough to rewrite complex Graphite queries with PromQL.

OpenTelemetry is a beloved, now graduated, CNCF project. It goes without saying that it has become the de facto way to describe, collect and process telemetry in modern systems. However, the story does not end here. There is still plenty of work to be done and the project is focusing heavily on becoming stable by default. This translates into: no more sudden breaking changes from one release to the other. In this session, the speakers will share their insights, as active maintainers of the project, on how this ambitious goal is approached across the OpenTelemetry ecosystem, what are the key projects that show progress right now, what have been achieved so far and what is coming next.

Are you excited about OpenTelemetry and how it’s become the standard for open o11y? Join us to learn the latest and greatest from the source!

GPU observability in Kubernetes is fragmented. Each vendor ships their own metrics stack: NVIDIA DCGM, AMD ROCm SMI. Platform teams running heterogeneous clusters stitch together three different dashboards to answer one question: "Are my GPUs being used efficiently?"

HAMi (CNCF Incubation) sits at the scheduling layer and sees every GPU operation. That single integration point gives you centralized observability across NVIDIA, AMD, Ascend, and any accelerator with a device plugin. Rather than scraping vendor-specific endpoints, HAMi instruments the scheduling path itself and reports utilization, memory pressure, and allocation efficiency per workload regardless of the underlying hardware.

This talk covers:
– Why GPU observability is harder than CPU observability (CUDA context model, MIG partitioning, device-plugin opacity)
– How HAMi's scheduling-layer instrumentation provides vendor-neutral metrics without per-vendor exporters

The GenAI observability space is fragmented right now. OpenInference, OpenLLMetry, framework-specific conventions are all solving the same problems with incompatible attribute names. That made sense when OTel’s GenAI support was thin. It makes less sense today.

OTel is where this converges. Getting there from where most teams actually are isn’t obvious. Kasper and Adriana cover the current landscape, how the genainormalizer processor bridges the gap at the collector layer, and what a realistic path to OTel-native GenAI observability looks like.

Then the harder question: your agent just deleted a database. What does your telemetry actually tell you? Non-deterministic systems don’t leave obvious footprints, and most teams discover that at the worst possible time.

The OpenTelemetry project has been wildly successful for gathering trace telemetry from services that look and act like web request-response services – but there are many high-impact services that don’t perform work on relatively short time scales: ETL, workflow automation, and (our specialty!) deployments.

In this talk, you’ll hear about the crucial role “supertraces” have played in enabling our team to diagnose problems in our customers’ long-running workflows, and the specific choices and custom instrumentation made to fill the gap between the existing OpenTelemetry specs and our needs. Presented by an SRE and a dev, we’ll share how our solution navigates the tension between keeping enough data to be valuable and spending a fortune on storage and compute.

Join us for drinks and appetizers with your fellow speakers and attendees. All attendees of Observability Summit are welcome.