Managing the data flow of the MHONGOOSE survey with open-access tools
2021-10-28, 11:00–11:15, Grand Ballroom

One of the challenges facing the SKA Precursor/Pathfinder Large Survey teams, is how to document and keep track of the data reduction process of the many observation tracks that make up the surveys. I will present an overview of the workflow used by the MHONGOOSE team to manage our data flow from submission of observation scripts to the assessment of the data reduction quality and creation of the science data products. We make use of the GitHub project management tools which allow us to link the status updates of each observation to the CARAcal configuration scripts used for the data calibration and imaging, and the reports created to assess the quality of the data. The reports, created by scripts developed by members of the MeerKAT Fornax Survey team, are an essential part of recording the data reduction process and assessing the quality of the data at each step in the pipeline. We have automated the uploading of the reports to GitHub as issues which makes it easy to monitor and track the progress of the data reduction.


Solutions for workflow management and reproducibility