One of the most challenging tasks in High Energy Physics is to reconstruct the particles entering the detector from the low-level detector response data. From the Deep Learning point of view, this is a set-to-set prediction task requiring multiple features and their correlations in the input data. We deploy a set-to-set neural network architecture to reconstruct particles in events containing a single jet in a fully-simulated calorimeter. The performance comparison favors a novel architecture based on learning hypergraph structure, which benefits from a physically-interpretable approach to particle reconstruction.