Speaker
Yana Holoborodko
(IRIS-HEP, Princeton University (US))
Description
This talk aims to present coffea-workflow package - workflow manager for coffea HEP analyses. This HEP-specific extension for coffea analyses does not replace existing workflow managers (Snakemake, LAW, …) - instead it focuses on three things coffea alone does not provide out of the box:
- Partial results - split your fileset into independently cached chunks; if some fail you keep the rest.
- Facility factories - one-line switching between local execution, coffea-casa, and CERN lxplus (HTCondor) without changing your analysis code.
- Execution control - choose between sequential and parallel chunk submission, tune executor type and worker count per facility
These three aspects, as well as some of the key concepts underlying the implementation of coffea-workflow, will be presented and discussed.
Primary author
Yana Holoborodko
(IRIS-HEP, Princeton University (US))
