Author: Billich, H.R.
Paper Title Page
Data Streaming - Efficient Handling of Large and Small (Detector) Data at the Paul Scherrer Institute  
  • S.G. Ebner
    PSI, Villigen, Villigen, Switzerland
  • H.R. Billich, H. Brands, E.H. Panepucci, L. Sala
    PSI, Villigen PSI, Switzerland
  For the latest generation of detectors transmission, persistence and reading of data becomes a bottleneck. Following the traditional pattern acquisition-persistence-analysis leads to a massive delay before information on the data is available. This prevents the efficient use of beamtime for users. Also, sometimes, single nodes cannot keep up in receiving and persisting data. PSI is breaking up with the traditional data acquisition paradigm for its detectors and is focusing on data streaming, to address these issues. Data is immediately streamed out directly after acquisition. The resulting stream is either retrieved by a node next to the storage to persist the data, or split up to enable parallel persistence, as well as online processing and monitoring. The concepts, designs, and software involved in the current implementation for the Pilatus, Eiger , PCO Edge and Gigafrost detectors at SLS, as well as what we are going to use for the Jungfrau detector and the whole beam synchronous data acquisition system at SwissFEL, will be shown. It will be shown how load-balancing, scalability, extensibility and immediate feedback are achieved, while reducing overall software complexity.  
slides icon Slides WED3O06 [2.264 MB]  
WEPGF063 Developing HDF5 for the Synchrotron Community 1
  • N.P. Rees
    DLS, Oxfordshire, United Kingdom
  • H.R. Billich
    PSI, Villigen PSI, Switzerland
  • A. Götz
    ESRF, Grenoble, France
  • Q. Koziol, E. Pourmal
    The HDF Group, Champaign, Illinois, USA
  • M. Rissi
    DECTRIS Ltd., Baden, Switzerland
  • E. Wintersberger
    DESY, Hamburg, Germany
  HDF5 and NeXus (which normally uses HDF5 as its underlying format) have been widely touted as a standard for storing Photon and Neutron data. They offer many advantages to other common formats and are widely used at many facilities. However, it has been found that the existing implementations of these standards have limited the performance of some recent detector systems. This paper describes how the synchrotron light source community has worked closely with The HDF Group to drive changes to the HDF5 software to make it more suitable for their environment. This includes developments managed by a detector manufacturer (Dectris - for direct chunk writes) as well as synchrotrons (DESY, ESRF and Diamond - for pluggable filters, Single Writer/Multiple Reader and Virtual Data Sets).  
poster icon Poster WEPGF063 [0.718 MB]