My colleagues at the Information Sciences Institute, University of Southern California have recently released version 3.1.0 of the Pegasus Workflow Management System. I have been working with the Pegasus team for many years now, and in a future post, I will write specifically about how it supports astronomy data processing.
Pegasus is a highly fault tolerant workflow management system that runs workflow applications in many different environments including desktops, campus clusters, grids, and now clouds. In a workflow application, the output from one component becomes the input to another component, as in a pipeline application. Pegasus enables scientists to construct workflows in abstract terms without worrying about the details of the underlying execution environment. This is what makes it so powerful as a science tool, and why it has found applicability in many fields, including astronomy, bioinformatics, earthquake science, climate modeling and others.
Shown below is a sample workflow from the Laser Interferometer Gravitational-Wave Observatory (LIGO). Pegasus has become an integral part of LIGO’s science data processing system: