2021-10-25, 10:45–11:00, Grand Ballroom
The Legacy Survey of Space and Time (LSST, www.lsst.org), operated by the Vera C. Rubin Observatory, is a 10-year astronomical survey due to start operations in 2022 that will image half the sky every three nights. Rubin estimates that the total amount of data that will be collected during operations to be about 60 petabytes (PB) from which a 20PB large catalog will be produced. At these data volumes reprocessing even a relatively small subset of data requires significant resources. We wanted to leverage cloud technologies to enable and accelerate data (re)processing. In this talk we describe how we adapted Rubin's image analysis software to run on AWS and the, often surprising, lessons learned while doing so.
Image processing for the public and scientists, Big data: How to deal with the 5 Vs (volume, velocity, variety, veracity, value)