Astronomical Image Processing at Scale With Pegasus and Montage.
2021-10-26, 08:30–09:00, Grand Ballroom

Image processing at scale is a powerful tool for creating new datasets and integrating them with existing data sets, and performing analysis and quality assurance investigations. Workflow managers offer advantages in this type of processing, which involve multiple data access and processing steps. Generally, they enable automation of the workflow by locating data and resources, recovery from failures, and monitoring of performance, We demonstrate in this focus demo the use of the Python API of the Pegasus Workflow Manager ( to manage processing of images to create mosaics with the Montage Image Mosaic engine ( Since 2001, Pegasus has been developed and maintained at ISI, USC. Montage was in fact one of the first applications used to design Pegasus and optimize its performance. Pegasus has since found application in many areas of science. LIGO exploited it in making discoveries of black holes ( . The Vera Rubin Observatory used it to compare the cost and performance of processing images on cloud platforms ( While these are examples of projects at large scale, investigations on local clusters of machines by small team can benefit from Pegasus as well. We describe how astronomers can use the Pegasus Python API to plan and execute workflows to create visualizations of the sky that comply with the Hierarchical Progressive Survey (HiPS) standards, on local and on cloud platforms.


Solutions for workflow management and reproducibility, Building accessible and friendly user interfaces, Image processing for the public and scientists