Keyword: data-management
Paper Title Other Keywords Page
WED3O02 Databroker: An Interface for NSLS-II Data Management System experiment, interface, detector, framework 1
 
  • A. Arkilic, D.B. Allan, D. Chabot, L.R. Dalesio, W.K. Lewis
    BNL, Upton, Long Island, New York, USA
 
  Funding: Brookhaven National Lab, U.S. Department of Energy
A typical experiment involves not only the raw data from a detector, but also requires additional data from the beamline. This information is largely kept separated and manipulated individually, to date. A much more effective approach is to integrate these different data sources, and make these easily accessible to data analysis clients. NSLS-II data flow system contains multiple backends with varying data types. Leveraging the features of these (metadatastore, filestore, channel archiver, and Olog), this library provides users with the ability to access experimental data. This service acts as a single interface for time series, data attribute, frame data access and other experiment related information.
 
slides icon Slides WED3O02 [2.940 MB]  
 
WEPGF059 The Australian Store. Synchrotron Data Management Service for Macromolecular Crystallography synchrotron, experiment, interface, operation 1
 
  • G.R. Meyer, S. Androulakis, P.J. Bertling, A.M. Buckle, W.J. Goscinski, D. Groenewegen, C. Hines, A. Kannan, S. McGowan, S.M. Quenette, J. Rigby, P. Splawa-Neyman, J.M. Wettenhall
    Monash University, Clayton, Australia
  • D. Aragao, T. Caradoc-Davies, N. Mudie
    SLSA, Clayton, Australia
  • C.S. Bond
    University of Western Australia, Crawley, Australia
 
  Store. Synchrotron is a service for management and publication of diffraction data from the macromolecular crystallography (MX) beamlines of the Australian Synchrotron. Since the start of the development, in 2013, the service has handled over 51.8 TB of raw data (~ 4.1 million files). Raw data and autoprocessing results are made available securely via the web and SFTP so experimenters can sync it to their labs for further analysis. With the goal of becoming a large public repository of raw diffraction data, a guided publishing workflow which optionally captures discipline specific information was built. The MX-specific workflow links PDB coordinates from the PDB to raw data. An optionally embargoed DOI is created for convenient citation. This repository will be a valuable tool for crystallography software developers. To support complex projects, integration of other instruments such as microscopes is underway. We developed an application that captures any data from instrument computers, enabling centralised data management without the need for custom ingestion workflows. The next step is to integrate the hosted data with interactive processing and analysis tools on virtual desktops.  
poster icon Poster WEPGF059 [2.109 MB]  
 
WEPGF060 A Data Management Infrastructure for Neutron Scattering Experiments in J-PARC/MLF neutron, database, experiment, operation 1
 
  • K. Moriyama, T. Nakatani
    JAEA/J-PARC, Tokai-Mura, Naka-Gun, Ibaraki-Ken, Japan
 
  The role of data management is one of the greatest contributions in the research workflow for scientific experiments such as neutron scattering. The facility is required to safely and efficiently manage a huge amount of data over the long duration, and provide an effective data access for facility users promoting the creation of scientific results. In order to meet these requirements, we are operating and updating a data management infrastructure in J-PAPC/MLF, which consists of the web-based integrated data management system called the MLF Experimental Database (MLF EXP-DB), the hierarchical raw data repository composed of distributed storages, and the integrated authentication system. The MLF EXP-DB creates experimental data catalogues in which raw data, measurement logs, and other contextual information on sample, experimental proposal, investigator, etc. are interrelated. This system conducts the reposition, archive and on-demand retrieve of raw data in the repository. Facility users are able to access the experimental data via a web portal. This contribution presents the overview of our data management infrastructure, and the recent updated features for high availability, scaling-out, and flexible data retrieval in the MLF EXP-DB.  
poster icon Poster WEPGF060 [1.017 MB]