PyHEP.dev 2026 - "Python in HEP" Developer's Workshop

7 Sept 2026, 08:50 → 9 Sept 2026, 22:30 Europe/Amsterdam

Veltman Centre (Nikhef)

PyHEP.dev is an in-person, informal workshop for developers of Python software in HEP to plan a coherent roadmap and make priorities for the upcoming year. It complements the PyHEP Users online workshop, which is intended for both developers and physicists.

You are encouraged to register to the HSF forum to receive further information concerning the organisation of the workshop.

Program

The agenda will consist of morning kick-off talks and afternoon coding sessions and discussions, in which the groups and topics are self-assigned.

Key Topics

• Agentic SWE and Tooling
• Distributed Computing and Workflows
• Data Analysis Tools, e.g., I/O, Histogramming, Statistical Tools, Visualization
• Packaging & Distribution

Organising Committee

Peter Fackeldey - Princeton University
Dmitry Kalinkin - Brookhaven National Laboratory
Alexander Heidelbach - Ludwig-Maximilians-University Munich (LMU)

Local Organising Committee

Jeff Templon - Nikhef
Joan Berger - Nikhef

---

The PyHEP.dev and the PyHEP User workshop are both supported by the HEP Software Foundation (HSF). Further information is on the PyHEP Working Group website. This PyHEP.dev workshop is also supported by the U.S. National Science Foundation cooperative agreement PHY-2323298 (IRIS-HEP) and by the Dutch National Institute for Subatomic Physics (Nikhef).

Monday, 7 September

09:25 → 09:30 Welcome

Speaker: Manfred Peter Fackeldey (Princeton University)

09:30 → 10:30 Talks

09:30 Coffea Framework: Current Status, Recent Updates, and Community Impact

The Coffea (Columnar Object Framework for Effective Analysis) framework continues to evolve as a cornerstone tool for high-energy physics data analysis, providing physicists with efficient, scalable solutions for processing complex event data. This talk presents the current status of Coffea, highlighting significant recent developments and their impact on the HEP analysis community.

A major milestone has been transitioning from Dask-Awkward to Awkward Array's new virtual arrays feature as the default backend. This shift improves Coffea’s user-friendliness, increases performance, and simplifies the execution model. The transition requires minimal to no code modifications for existing analyses, providing a seamless migration path with substantial improvements.

Recent enhancements include advanced workflow features such as checkpointing for robust, resumable analyses, improved branch preloading and caching for network-efficient data access, and workflow tracing to identify required data branches. These optimizations benefit both interactive and batch processing scenarios, allowing physicists to focus on physics rather than data management details.

This talk will also present usage and community feedback demonstrating Coffea's growing adoption across CMS and other HEP experiments. Looking toward the High-Luminosity LHC era, Coffea's architecture positions it as a key enabling technology for handling unprecedented data volumes while maintaining intuitive, user-friendly interfaces for data analysis at scale.

Speaker: Iason Krommydas (Rice University)

09:45 Evolving ROOT's Python Interfaces

ROOT provides automatic and dynamic Python bindings that expose its powerful C++ core to the broader scientific Python ecosystem. As Python usage continues to grow across the HEP community, ROOT continues to evolve to offer a more intuitive, Pythonic experience while ensuring its interfaces interoperate smoothly with the rest of the ecosystem.

This contribution presents recent developments across ROOT's Python interfaces, enabling more direct interaction with other Python packages, whether for analysis through the Unified Histogram Interface (UHI) or machine learning through RDataLoader. We also discuss underlying infrastructure upgrades supporting these interfaces, and the design considerations involved in extending and maintaining them.

Speaker: Silia Taider (EP-SFT)

10:00 Uproot: RNTuple support and future plans

The migration from TTree to RNTuple as the preferred format for data storage is now in full force. Uproot, the IO package in Scikit-HEP, has stayed at the forefront of this migration. It now has 100% bi-directional support between RNTuples and Awkward arrays, supports schema extensions, and defaults to writing RNTuples instead of TTrees. I will be discussing the status of RNTuple support, some limitations, and plans for improvement. Along with this, I will also talk about potential plans for Uproot as a whole, including refactors, improvements, and AI-assisted developments.

Speaker: Andres Rios-Tascon (Princeton University)

10:15 Pepper: a CMS columnar analysis framework

This talk presents Pepper, a coffea-based, general purpose, CMS-endorsed analysis framework. Pepper is designed to run on the CMS NanoAOD dataset without intermediate ntuples, aiming to allow full analyses with fast time-to-insight. Its features include substantial book-keeping and histogram-filling capabilities, implementation of standard CMS objects, problematic file handling, resumption from saved states and scale-out via HTCondor. A broad range of analyses are supported, with support for techniques from data-driven background estimation to unfolding. In particular, the talk will focus on the main performance bottlenecks in real analysis scenarios (very commonly memory limits) and the examples of common user experiences, which will hopefully be informative for developer of other packages in the scikit-HEP ecosystem, on which Pepper relies.

Speaker: Dominic Stafford (Deutsches Elektronen-Synchrotron (DE))

10:30 → 11:00 Coffee Break

11:00 → 12:00 Talks

11:00 FCCAnalyses: Python-Driven Analysis at Scale for the Future Circular Collider

FCCAnalyses is the common analysis framework for the Future Circular Collider (FCC) study, built on ROOT's RDataFrame and the EDM4hep event data model. Its performance-critical processing is written in C++, but the framework is designed and driven from Python end-to-end: users configure analyses, manage dataset metadata, and run their processing on anything from a laptop to a grid site through a single Python interface.

We present a set of recent additions that make this Python layer more capable and consistent. Analyses can now be submitted to grid resources via DIRAC, alongside the existing HTCondor support, through the same command. Reading and writing data now supports ROOT's new RNTuple format alongside the established TTree. Analyses using full fledged EDM4hep objects through PODIO DataSource can also access additional metadata stored in EDM4hep files, such as multiple event weights, directly from Python. Finally, a new fit subcommand lets users define statistical models natively in Python instead of separate configuration files, and generates the inputs needed for the CMS Combine tool automatically. We'll walk through these additions from a user's perspective and use the workshop's discussion sessions to compare notes with other framework developers on analysis tools, distributed computing, packaging, and where our tools overlap with the wider Python HEP ecosystem.

Speakers: Birgit Sylvia Stapf (CERN), Juraj Smiesko (CERN), Pantelis Kontaxakis (Universite de Geneve (CH))

11:15 coffea-workflow: coffea extension for HEP workflow management

This talk aims to present coffea-workflow package - workflow manager for coffea HEP analyses. This HEP-specific extension for coffea analyses does not replace existing workflow managers (Snakemake, LAW, …) - instead it focuses on three things coffea alone does not provide out of the box:

1. Partial results - split your fileset into independently cached chunks; if some fail you keep the rest.
2. Facility factories - one-line switching between local execution, coffea-casa, and CERN lxplus (HTCondor) without changing your analysis code.
3. Execution control - choose between sequential and parallel chunk submission, tune executor type and worker count per facility

These three aspects, as well as some of the key concepts underlying the implementation of coffea-workflow, will be presented and discussed.

Speaker: Yana Holoborodko (IRIS-HEP, Princeton University (US))

11:30 A python-based analysis pipeline for the ATLAS Integration Challenge

The "Integration Challenge" is the latest IRIS-HEP project exploring physics analysis pipelines for the HL-LHC. Compared to earlier iterations, this one focuses on capturing the detailed needs of experiments. The contribution presents the analysis pipeline built for the ATLAS version of the project, which is based on a variety of Scikit-HEP and IRIS-HEP projects (including uproot, awkward, coffea, hist, ServiceX). We provide an overview of the components involved and the optimizations that have led us to the current implementation. While still relying on Dask for distributed execution, a number of changes compared to traditional coffea-based approaches have both expanded functionality and increased robustness.

Speakers: Alexander Held (University of Wisconsin–Madison), Artur Cordeiro Oudot Choi (University of Washington)

11:45 An update on CMS analysis pipelines through the Integration Challenge

Analysis workflows for the High-Luminosity LHC (HL-LHC) will need to handle far more data than current approaches were designed for. To prepare for this, CMS analysis pipelines are being built on the scientific Python stack, with HEP-specific tools filling the gaps where general-purpose software falls short. This talk gives an update on that effort through the lens of the Integration Challenge, an IRIS-HEP initiative that tests the software stack and analysis facilities under realistic conditions before the HL-LHC era begins.

Rather than testing components in isolation, the Integration Challenge runs a realistic physics analysis from start to finish, exposing how well the tools work together and what the experience feels like for an analyst. Because the surrounding ecosystem keeps moving, the challenge has to move with it: a recurring theme of recent work has been picking up new capabilities from coffea and the Scikit-HEP libraries and working them into the pipelin. We discuss what this continuous integration of ecosystem developments looks like in practice, and outcomes from integrating new features into the workflow.

Speaker: Mohamed Aly (Princeton University)

12:00 → 13:30 Lunch

13:30 → 14:30 Talks

13:30 Omnifold- a Python package for publishing, and comparing unbinned, unfolded measurements

OmniFold gives you per-event weights instead of a histogram, so you can build whatever distribution you want afterward. But that makes results hard to share. A weight file is just a few hundred unlabeled columns, and readers are left guessing which one is the main result, which columns belong to the same uncertainty estimate, how to combine them, and whether the rows even still match the original data.
omnifold_publication is our attempt at a standard for publishing unbinned results. It stores the data and weights in a standard table format, plus a metadata file that says which weights go together, how to combine them, the binning convention, normalization, row alignment, and a checksum. The package also reads and analyzes results published this way: histograms, uncertainty estimates, correctness checks, and export for publication. We want unbinned measurements to be as easy to reuse as a histogram, without every analysis making up its own format.

Speaker: Aashirvad Bajpai (Manipal Academy of Higher Education)

13:45 Model Context Protocol for Scientific Software: Building AI Interfaces That Actually Work

Large language models are rapidly becoming useful assistants for software development, analysis, and documentation. In practice, however, the limiting factor is rarely the model itself. The hard part is giving the model reliable, secure access to the tools, data, and domain knowledge that scientists already use.

This talk introduces the Model Context Protocol (MCP) from the perspective of scientific software developers. Rather than focusing on prompt engineering or custom foundation models, we'll examine how MCP servers expose existing Python libraries, databases, workflow systems, and experiment services to AI assistants through well-defined interfaces.

Using examples from high-energy physics (e.g. dataset databases, analysis metadata, and workflow services) we will discuss practical patterns for building MCP servers, handling authentication without exposing user credentials, and designing interfaces that remain useful across different LLM providers and experiments.

You will hopefully have a concrete understanding of when an MCP server is appropriate, how to build one around an existing Python project, and why interoperable tool interfaces are likely to matter more than model-specific integrations.

Speaker: Giordon Holtsberg Stark (University of Chicago)

14:00 LLM explorations for statistical analysis workflows

Following the recent increase in LLM capabilities, there are many exploratory projects ongoing to identify the most promising use cases for HEP. In this contribution I report on two examples in the context of statistical model building. The first example focuses on testing the capabilities of an automated support system for complex software. The second investigates to what extent it may be feasible or useful to auto-generate analysis code from a declarative configuration, with a look at computational efficiency.

Speaker: Alexander Held (University of Wisconsin–Madison)

14:15 An AI Assistant for CMS Combine

Combine is used for most statistical results in CMS, but its many methods, options, and error messages make it hard to learn and lead to many repeated support questions. We present an AI assistant for Combine built around a large‑language‑model agent that answers using the tool's own trusted sources and cites the material it relies on. The assistant can also reproduce and diagnose a user's command by running Combine in an isolated, sandboxed environment, so it can both explain and check a result. It runs on standard CERN infrastructure, supports different language models, and needs no per‑user setup. We describe the system's design and how it can make statistical analysis in CMS easier to approach.

Speaker: Massimiliano Galli (Princeton University)

14:30 → 18:00 Discussion / Coding

Tuesday, 8 September

09:30 → 10:30 Talks

09:30 Luigi and Workflow Managers in HEP

Reproducible, fault-tolerant analysis pipelines are essential to modern HEP computing, and workflow managers provide the backbone for meeting these demands. In this talk, I present an overview of workflow management in HEP, with a focus on Luigi—a Python-based framework widely used in the community—and its HEP-specific extensions, b2luigi and Law. I discuss their design philosophies, key features, and ongoing development efforts. I then broaden the scope with a brief comparison to alternative workflow management tools from outside HEP, including Snakemake, Apache Airflow, Dask/Spark, and emerging commercial platforms, highlighting their relative strengths and trade-offs in the context of typical HEP analysis workflows.

Speaker: Jonas Eppelt

09:45 Did we accidentally build a compiler for HEP workflows? FAST-HEP Flow from declarative workflows to executable plans

Modern high-energy physics analyses rely on complex software workflows that must evolve alongside changing libraries, data formats, execution environments, and computing technologies. Experience with FAST-HEP has shown that declarative analysis descriptions improve readability, but do not by themselves make the complete analysis unambiguous, replaceable, or reproducible.
This talk introduces Flow, a workflow system that applies compiler techniques to scientific analyses. Flow consolidates author-facing inputs into a normalized workflow, constructs a logical graph, validates dependencies, and produces a backend-independent execution plan before execution begins. This enables earlier error detection, interchangeable implementations and execution environments, and traceable production of analysis artifacts.
Focusing on a small number of architectural examples, the talk will discuss why compilation was introduced, what it enables in practice, and the lessons learned from nearly a decade of developing and modernising FAST-HEP.

Speaker: Lukasz;Luke Kreczko (University of Bristol (GB))

FAST-HEP-pyhep2026.pdf

10:00 ParaO: Ease of Use for In Situ Workflows

Born for first-hand user experience of using Luigi based workflows for an HEP analyses, the ParaO Workflow Python package addresses several pain points that arise when developing and executing an analysis.
This contribution will introduce the core concepts, while relating to principles already found in Luigi, and will showcase the new features that attempt to address perceived shortcomings in Luigi.

Speaker: Benjamin Fischer (RWTH Aachen University (DE))

10:15 ParaO: First hand experience within a Run 3 analysis

This contribution will show how ParaO is used to conduct an analysis and related first hand experiences.

Speaker: Felix Philipp Zinn (RWTH Aachen University)

10:30 → 11:00 Coffee Break

11:00 → 12:00 Talks

11:00 Efficient Statistical Modeling for Particle Physics Using Computational Graphs in Python

Statistical modeling is central to discovery in particle physics, yet the tools commonly used to define, share, and evaluate these models are often complex, fragmented, or tightly coupled to legacy systems. In parallel, the scientific Python community has developed a variety of statistical modeling tools that have been widely adopted for their performance and ease of use, but remain under-utilized in particle physics. We attempt to bridge this gap with a lightweight python framework that calculates likelihood ratios through the construction and evaluation of computational graphs. With modularity, auto-differentiability, and computational efficiency in mind, we designed the framework to integrate with modern scientific computing ecosystems while providing a clean, well-documented, and extendable API. This implementation makes published particle physics results more transparent, reproducible, and accessible for reanalysis. We present the initial framework, validate its results against established calculations, examine its performance relative to existing systems, and outline future development plans.

This work was supported by the U.S. Department of Energy (DOE) Office of High Energy Physics under Grant No. DE-SC0010107.

Speaker: Giordon Holtsberg Stark (University of Chicago)

11:15 SysVar: Consistent and scalable treatment of systematic uncertainties in template-based analyses

SysVar is a Python package offering a complete workflow for handling and propagating systematic uncertainties in high-energy physics analyses that rely on templates built from simulated data.

Propagating systematic uncertainties from correction weights into templates, while keeping the correlations in the signal extraction variables intact, requires increased bookkeeping as analyses grow in complexity. SysVar addresses this with a lightweight API that abstracts away most of the bookkeeping involved in applying correction weights to simulated events, building systematic variations, and histogramming both nominal and varied templates. This simplifies and automates essential workflows for high-precision measurements. Uncertainties are propagated consistently by building the full covariance matrix over the multidimensional analysis space and using Principal Component Analysis to obtain orthogonal eigenvariations for the subsequent statistical fits. Correlations are retained through a controlled compression of the systematic space, which reduces the number of nuisance parameters. Analyses are specified entirely through a declarative configuration, supporting reproducibility and scalability, and built-in visualization tools aid diagnostics, debugging, and studies of uncertainty models.

SysVar is intended as a drop-in layer between reconstruction ntuples and model-building frameworks, with native output for cabinetry and pyhf. In this talk we illustrate its use by combining two orthogonal and independent pseudo-measurements that profit from the consistent treatment of systematic correlations encoded in the template shapes. While it was developed in the Belle II context, SysVar is open-source, pip-installable, and experiment-agnostic.

Speaker: Agrim Aggarwal

11:30 evermore - Differentiable Binned Likelihood Functions with JAX

evermore is a python package that provides building blocks for building and evaluating (binned) likelihood functions in high-energy physics using JAX. The key concepts of evermore are performance and differentiability. JAX provides automatic differentiation, just-in-time (jit) compilation, and vectorization capabilities, which can be leveraged to improve the performance of statistical analyses. Jit-compilation and vectorization can be used for parallelizing fits on GPUs which is especially advantageous for likelihood scans and toy based upper limits.

We present the concepts of evermore and show its features.

Speaker: Felix Philipp Zinn (RWTH Aachen University)

11:45 everwillow: model-agnostic statistical inference in pure JAX

everwillow is a statistical inference library for high energy physics built on JAX pytrees. It is deliberately open-world: there is no model specification format and no assumption about where the likelihood comes from. Any negative log-likelihood written as a JAX function works, whether it comes from a binned template fit, an unbinned parametric model, or another JAX-based package, and everwillow supplies the inference around it: fitting with optimistix, bounded parameters via transformations, profiling, and hypothesis testing. Test statistics, sampling distributions, and calculators are independent components, so asymptotic and toy-based inference go through the same interface and the full pipeline is JIT-compilable and differentiable. The talk presents the library's design, benefits of a pure-JAX engine, and demonstrates typical inference workflows. It will also touch on ongoing work towards efficient exploration of multi-dimensional parameter spaces.

Speaker: Mohamed Aly (Princeton University)

12:00 → 13:30 Lunch

13:30 → 14:30 Talks

13:30 Packaging ROOT for Python: Progress and Challenges

ROOT is distributed through a range of channels suited to different users and workflows, from CVMFS-based releases to pre-built binaries and specialized packaging systems.
Given the growing adoption of Python in the HEP community, it becomes increasingly important that ROOT is easy to obtain and depend on as a Python package with pip and conda offering reliable ways to install it.

This contribution discusses ongoing efforts to modernize and expand ROOT's packaging infrastructure. We report on the continued development of ROOT's PyPI wheels, experimentation with new wheel configurations aimed at improving adoption across platforms, and the broader work of validating ROOT as a dependency in real-world Python workflows.

Speaker: Silia Taider (EP-SFT)

13:50 HEP Packaging Coordination: Distributing the HEP software ecosystem on conda-forge

The packaging of high energy physics software with robust, yet flexible, distribution methods is a complicated problem that has been met with multiple approaches by the community. The HEP Packaging Coordination community project expands packaging of the HEP software ecosystem through building and distributing language-agnostic conda packages on the conda-forge package index. Through use of the conda-forge community build cyberinfrastructure, computing platform specific optimized builds of packages can be created for selections of Linux, macOS, and Windows across x86-64, AArch64/ARM64, and ppc64le architectures. In addition to supporting builds of ROOT, this work provides multi-platform packaging of a wide array of low-level-language phenomenology tools, the broader simulation stack, end-user-analysis tools and statistical frameworks, and the reinterpretation ecosystem. Ongoing work is also supporting builds of LHCb experiment software and distributions of community software with experiment-specific patches applied for use in LHC physics analyses.

This process significantly lowers technical barriers across tool development by providing automatic packaging systems with source code, distribution through secure and transparent build cyberinfrastructure, and enables use through multi-platform optimized binary builds. When combined with next generation scientific package management and manifest tools, the creation of fully specified, portable, and trivially reproducible multi-language software environments becomes easy and fast, even with the use of development platforms for hardware accelerators (e.g. CUDA on NVIDIA GPUs). This talk provides an overview of the work, gives practical recommendations for adoption and best practices for both software maintainers and end-user analysts, and demonstrates examples of new distribution methods that are complementary to existing community technologies, such as CernVM-FS.

Speaker: Matthew Feickert (University of Wisconsin–Madison)

14:10 Scipy, Numpy, Xarray and Python all have a pixi.toml. Why?

After 3 years, Pixi is widely adopted in the scientific Python ecosystem, let me show you why!

Scientific Python has specific challenges that Pixi can solve well; a lot of our beloved packages contain C, C++, Rust, CUDA or even Fortran code. With Pixi, a single tool can install compilers, different Python versions and other build tools in one go, thanks to piggybacking on the years of development the Conda ecosystem has seen.

Thanks to Pixi’s task system and native multi-platform capabilities, the contributor experience is also enhanced. Daunting tasks like running CMake, installing the correct Rust version or C++ compilers are all hidden away behind a magical: pixi run foobar.

Are you interested to see how you could improve your own workflow and learn from what these big open-source projects are doing? Then you should join this talk! You'll be amazed by what is possible these days.

Speaker: Ruben Arts (prefix.dev GmbH)

14:30 → 18:00 Discussion / Coding

Wednesday, 9 September

09:30 → 10:30 Discussion / Coding

10:30 → 11:00 Coffee Break

11:00 → 12:00 Summary Discussion