Schedule

Edge and on-prem AI inference increasingly runs on silicon that isn't a GPU. This talk tells the story of bringing ExecuTorch — PyTorch's on-device inference runtime — to IBM's AIU, a purpose-built accelerator for enterprise inference, entirely through ExecuTorch's public backend-delegate extension points: a custom partitioner, an AOT-only compilation path with well-defined CPU-fallback boundaries, a mixed-dtype tensor bridge, and static bucketing for dynamic shapes.

We'll walk through the concrete design decisions and dead ends behind a working, device-verified delegate running real language models (Granite) end-to-end on the AIU, then generalize them into a practical checklist for anyone extending ExecuTorch to a new, independent accelerator. Attendees will leave with a concrete map of ExecuTorch's backend extension surface — what's genuinely reusable, what's hardware-specific, and where the sharp edges are — whether their target is a novel edge chip, an emerging vendor's silicon, or an in-house accelerator.

Guided real-time chunking aligns a robot's next action sequence with the one already executing. That alignment depends on a backward pass through the denoiser. Exporting only the forward model leaves out the correction, so the deployed computation no longer implements the same policy.

This talk addresses that deployment gap by treating the complete guidance step as the unit of compilation. Using SmolVLA's action expert and ExecuTorch on Arm CPUs, we walk through making attention-cache state explicit and capturing the denoiser, action-input vector-Jacobian product and correction update in one static program. We build on ExecuTorch's existing forward/backward export, with model weights held fixed.

We examine the graph boundaries, backward-operator coverage and activation lifetimes needed to make this program executable. Validation against eager PyTorch and torch.compile covers gradient and action-chunk agreement, FP32 latency and peak memory. Attendees will learn how to carry gradient-dependent inference from a Python research loop into an edge runtime while preserving its action correction.

Run a GGUF model on a Mac and most of the work happens in a few hundred lines of Metal you've never read. This talk opens up one of them.

Q5_K is among the most-used quantization formats in the GGUF ecosystem, and until recently ExecuTorch's MLX backend couldn't execute it. I'll walk through the upstream contribution that added it.

We'll cover what's actually inside a Q5K super-block, an affine d/dmin block, 6-bit packed sub-block scales, and a separate array holding each weight's 5th bit, and why that layout forces real kernel design rather than a copy of Q4K. Then the two kernels every decoder needs: a mat-vec for decode, a tiled simdgroup mat-mat for prefill, and dynamic-M dispatch between them. With benchmarks: Q4K vs Q5K vs Q6_K tokens/sec and memory on Apple silicon.

You'll leave knowing how block quantization really works on-device, and how to get your own kernel merged into a PyTorch runtime.

Deploying large language models (LLMs) on limited hardware is challenging. Models may exceed available memory, and selecting quantization methods and bit widths requires specialist expertise. OneComp is an open-source Python library for resource-adaptive post-training quantization. With a single command, it detects available GPU memory, assigns layer-wise bit widths using AutoBit, performs quantization and error correction, evaluates the result, and saves the compressed model. OneComp incorporates Fujitsu’s latest model compression technologies, including Quantization Error Propagation (QEP) and JointQ. This talk will introduce how these techniques help preserve model quality at low precision and explain OneComp’s approach to making LLM compression more accessible.

How can a local PyTorch community turn learners into open-source contributors?

This talk shares lessons from eight years of PyTorch Korea, focusing on tutorials-kr, a community-maintained Korean translation of the official PyTorch tutorials. We explain how translated learning resources, university lectures, and open-source mentoring create entry points for developers, researchers, and students to contribute.

Our mentoring efforts supported 26 first-time contributors and 51 merged PRs to tutorials-kr, which has welcomed more than 160 contributors overall. We use this experience to discuss lowering the barrier to a first contribution, supporting contributors through review, and connecting learning activities with open-source participation. We also share how the vLLM Korea Meetup broadened knowledge exchange across the PyTorch ecosystem.

Attendees will leave with practical lessons for organizing contribution opportunities, connecting education with project maintenance, and measuring community reach and contribution outcomes separately.

LeRobot is Hugging Face's PyTorch library for real-world robotics, and its dataset format gave the community a shared standard. But robot learning data is multimodal and growing fast: long multi-camera videos, states, actions and language instructions in every episode. Training from object storage with global shuffling, and finding the right episodes first, still means copying data or stitching systems together.

LanceDB and the LeRobot team upstreamed LanceDB as the first non-LeRobot storage backend. Users keep the familiar LeRobotDataset API and read Lance datasets in place from the Hub, HF buckets or S3.

This session covers:
– How episodes, frames and videos are laid out as tables for random access
– Feeding PyTorch: map-style datasets, global shuffle from object storage, and pitfalls with parallel workers
– Using the same table to search, curate and train, with vector and full-text search
– What it took to move from an external plugin into LeRobot core

You'll leave with steps to try in your own LeRobot setup, and a clear view of when this design helps and its current limits.

Vision-Language-Action (VLA) models enable AI systems to perceive their environment, understand instructions, and perform physical actions; however, deploying VLA models on edge devices requires more than just getting inference running. Real-world systems have to satisfy strict requirements concerning latency, memory, power, thermal behavior, reliability, and security.

The talk describes a complete process for moving from a PyTorch VLA model to real-time on-device inference using torch.export and ExecuTorch on ARM-based edge devices. The deployment pipeline includes model optimization, quantization, hardware acceleration, and runtime execution.
Systematic examines the entire perception-to-action process, measuring end-to-end latency, memory usage, power consumption, sustained performance, thermal behavior, and task success. The presentation examines how various optimization techniques and hardware acceleration affect real-time performance.

The aim is to develop a practical method for determining when a PyTorch VLA model is ready for real-time Physical AI, moving beyond the simple criterion of "it runs" to "it responds, performs, and acts reliably on the edge."

High-performance PyTorch applications increasingly depend on specialized kernels, but writing a fast kernel is only part of the problem. Projects also need to build across hardware and software configurations, distribute kernels, manage compatibility, version them safely, and integrate them into their own stacks.

In this talk, Sayak Paul and Aritra Roy Gosthipaty will introduce Hugging Face Kernels and show how the Kernel Hub provides a common layer for building, versioning, distributing, and consuming optimized kernels across the PyTorch ecosystem.

We will show how this abstraction lets projects share optimized implementations without maintaining their own packaging and distribution infrastructure, from declaring kernel dependencies to selecting compatible builds and locking versions for reproducible deployments.

We will also discuss how Kernel Hub is evolving beyond PyTorch, with support for interfaces such as TVM-FFI, and examine adoption in projects such as SGLang to show how shared kernel infrastructure can reduce integration overhead.

Adapting inference frameworks to different AI accelerators requires coordinated work across operators, compilers, runtimes, and communication. This talk uses vLLM and SGLang to show how FlagOS organizes that adaptation while preserving familiar serving interfaces.

We connect engine plugins to FlagGems operators and explain the roles of FlagTree's supported compiler paths and optional FlagCX communication. A concrete RMSNorm example shows how two engine interfaces reach shared computation while retaining framework-specific handling. Device-independent installation and vendor fallback illustrate additional hardware integration requirements.

We then examine validation across eager and graph modes, text and image requests, and cold-start behavior, using public vLLM plugin tests and documented SGLang deployment cases.

Attendees will learn how to structure multi-accelerator adaptation, decide what can be reused across frameworks and chips, and verify correctness and performance. The session is grounded in open-source code, upstream changes, and deployment evidence.

At the start of 2026, running models on PyTorch's MPS backend would mean working around unsupported operations, performance bottlenecks and memory growth even for small models. Since then, MPS has become much more mature platform to run the regular models on locally.

This talk goes through the progress on the backend such as, fused attention for prefill/decode workloads, Flex Attention, native Metal kernels for linear algebra operations, convolutions, porting MPS Graph to Metal kernels and memory allocator changes which fixes unexpected memory growth. I'll connect these changes to real model workloads from Hugging Face across language, image and video models, showing how improvements across the backend add up to better user experience.

We'll look at where MPS stands today, where are it's limitations compared to MLX and what are the upcoming changes we can expect for the backend in the near future.

safetensors is widely used to distribute model checkpoints, especially through Hugging Face. fastsafetensors is a high-performance loader that reduces loading overhead through optimized file access and device transfers. The speaker is a maintainer of fastsafetensors, which is integrated with PyTorch systems including vLLM and SGLang.

Portable checkpoints do not always match the tensor formats accelerators need. Loading may require data-type and memory-layout conversions in addition to I/O and transfers, and these conversions can remain a bottleneck even after I/O is optimized.

Using IBM Spyre as a case study, this talk presents extensions to fastsafetensors. Converted weights are written directly into output files in parallel across shards, reducing intermediate copies. Later loads reuse cached weights when the source weights, conversion settings, and environment remain compatible.

We discuss cache validation, creation cost, storage overhead, and the boundary between generic file handling and device-specific conversion. Attendees will learn how to preserve portable model distribution while efficiently preparing tensors for accelerator execution.

Frontier labs post-train models across thousands of RL environments. At their core, these environments expose two things: tasks for models to solve and verifiers that determine whether they succeed. The challenge is scaling both: generating diverse, high-quality tasks from real-world data and building robust verifiers that provide reliable rewards. Yet very little of this work is being done openly at scale.
We walk through four key decisions in building RL environments: choosing domains with real, verifiable data; turning that data into tasks with controlled difficulty; designing robust rewards that resist hacking; and packaging everything with OpenEnv and Harbor for scalable training and evaluation.
We then explore how these environments integrate with TRL for GRPO training, including agents running behind a proxy, and share practical examples across coding, games, multilingual OCR and speech, design, biology, and more, highlighting how the environments were built and what training showed.
All environments and recipes are open in FineEnvs: https://huggingface.co/FineEnvs

Disaggregated inference separates prompt prefill from token generation, but adds routing and KV cache transfer costs. Whether it pays off depends on workload and hardware, making qualitative guidance such as “prefill-heavy” difficult to apply.
We present llm-d-diagnostics, a portable Python toolkit to measure monolithic vLLM and prefill/decode deployments on Kubernetes. It collects individual request data across latency decomposition, throughput, prompt and output length sweeps, mixed workload isolation, saturation, cache behavior, and fault recovery. Its analyzer reports latency distributions, statistical comparisons, and data quality indicators. An advisory module combines model architecture with GPU and network assumptions to estimate transfer overhead and identify crossover points by concurrency and prompt length. It uses cluster measurements when available and distinguishes them from estimates based on reference data.

The toolkit provides a repeatable measurement protocol that can be configured for different GPU types and Kubernetes cluster environments, helping practitioners determine where disaggregation’s isolation benefits offset its transfer costs.

An agent prompt on a phone can open with about 2,050 tokens of harness, 1,700 of them tool definitions. On a Dimensity 9500s, a 1,036-token prefix takes 9.3 s to prefill on LFM2.5-1.2B and 45 s on SmolLM3-3B; faster decode does nothing to that wait. We solved it with prefix reuse, built twice because our open source app ships two runtimes with different cache constraints.

ExecuTorch's Android API exposes prefill and full reset, but no partial rollback. We prefill the shared head at model load, then feed only the unmatched suffix while the prompt is an exact extension. On Qwen3-1.7B, a second turn reused 1,192 tokens and fed 20 new ones, cutting prefill to 215 ms, 27 times faster than the first turn. On our llama.cpp path, LFM2.5's recurrent state cannot rewind without per-token snapshots, so we snapshot the warmed 12 MB state and restore it; that 1,018-token prefix falls from 9,326 ms to 235 ms.

Correctness depends on byte identity. The date is pinned per conversation, history replays the engine's own bytes, changing reasoning mode invalidates the prefix, and truncated replies are never reused. We show cache rules, failures and regression tests; measurements and logs are public.

Quantized models are typically accepted when their perplexity or benchmark scores remain close to the original model. Production systems, however, depend on behavior, not perplexity.

This session tests whether perplexity is a reliable proxy for production behavior. A small open model is quantized using PyTorch-native tooling and compared against its full-precision baseline across a focused set of behavioral metrics, including output stability, JSON and schema validity, structured generation, and instruction-following consistency.

The results reveal where benchmark parity aligns with behavioral parity, where it breaks down, and which classes of regressions remain invisible to aggregate evaluation metrics. The session also introduces DriftBench, a lightweight open-source PyTorch harness for measuring behavioral regression between model variants.

Attendees will leave with practical acceptance criteria for deploying quantized models, an open-source benchmarking tool, and a reproducible methodology for validating model behavior before production deployment.

DeepSpeed is a pioneering distributed training framework built on top of PyTorch, combining performance and scalability with ease of use. It has over 43,000 GitHub stars and one million+ monthly PyPI downloads. DeepSpeed now delivers performance competitive with other state-of-the-art frameworks on frontier models, with just a configuration change. This talk presents the key recent optimizations and benchmark results.
Best known for ZeRO, DeepSpeed has since added expert, sequence, and tensor parallelism (AutoEP, AutoSP, AutoTP) that compose with ZeRO and work with a wide range of models, including many Hugging Face models. Much of this progress has come from contributions by researchers and engineers across many organizations. AutoEP, for instance, brings optimized kernels and communication to large-scale MoE training, while DeepCompile optimizes communication scheduling and memory offloading at the compiler level using torch.compile. This success reflects DeepSpeed's open development, which draws a wide variety of contributions, including experimental and research-oriented features.