Scientific Workflows for Science and the Science of Scientific Workflows

In this era of compute intensive astronomy, tools for managing complex workflows are becoming a crucial part of our science infrastructure. This video offers a fine introduction to the Pegasus workflow manager by the team leader, Ewa Deelman (ISI, USC).

The abstract for Ewa’s talk follows::

“Modern science often requires the processing and analysis of vast amounts of data in search of postulated phenomena, and the validation of core principles through the simulation of complex system behaviours and interactions. This is the case in fields such as astronomy, bioinformatics, physics, and climate and ocean modelling, and others.

In order to support the computational and data needs of today’s science, new knowledge must be gained on how to deliver the growing high-performance and distributed computing resources to the scientist’s desktop in an accessible, reliable and scalable way.

In over a decade of working with domain scientists, the Pegasus project has developed tools and techniques that automate the computational processes used in data- and compute-intensive research. Among them is the scientific workflow management system, Pegasus, which is being used by researchers to discover gravitational waves, model seismic wave propagation, to discover new celestial objects, to study RNA critical to human brain development, and to investigate other important research questions.

This talk will review the conception and evolution of the Pegasus research program. It will touch upon the role of scientific workflow systems in advancing science, and will give specific examples of how the Pegasus Workflow Management System has done so. It will describe how the Pegasus project has adapted to changes in application needs and to advances in high performance and distributed computing systems. It will discuss the interleaving of Computer Science research and software development and how each benefits from the other while providing value to other science domains.”

If you want a more general overview of workflow managers, see this talk at NCSA by Scott Callaghan: