The Department of Energy (DOE) just released its final report on the Magellan project. Magellan’s remit was “..to investigate the potential role of cloud computing in addressing the computing needs for the DOE Office of Science (SC), particularly related to serving the needs of mid- range computing and future data-intensive computing workload.” A testbed infrastructure at Argonne National Lab was set up to probe questions such as: Are the open source cloud software stacks ready for DOE HPC science? How usable are cloud environments for scientific applications?
This 169-page document may be long, but I think it is essential reading for anyone interested in applying cloud technology to scientific applications, especially data driven workflows. I will write about these issues in more detail in future posts. Here, I will summarize the key findings:
Finding 1. Scientific applications have special requirements that require solutions that are tailored to these needs.
Finding 2. Scientific applications with minimal communication and I/O are best suited for clouds.
Finding 3. Clouds require significant programming and system administration support.
Finding 4. Significant gaps and challenges exist in current open-source virtualized cloud soft- ware stacks for production science use.
Finding 5. Clouds expose a different risk model requiring different security practices and policies.
Finding 6. MapReduce shows promise in addressing scientific needs, but current implementa- tions have gaps and challenges.
Finding 7. Public clouds can be more expensive than in-house large systems.
Finding 8. DOE supercomputing centers already approach energy efficiency levels achieved in commercial cloud centers.
Finding 9. Cloud is a business model and can be applied at DOE supercomputing centers.
There are many recommendations made, but here I will simply summarize those for developing science applictions:
- Science groups need to carefully benchmark applications with the different options to find the best performance-cost ratio.
- Scientists should work with tool developers to ensure that their requirements and workflows are sufficiently captured and understood.
- Application developers should consider the potential for variability and failures in their design and implementation.
- Science groups should attempt to use standardized secure images to prevent security and other configuration problems with their images. Science groups will also need to have an action plan on how to secure the images and keep them up to date with security patches.
- Scientific users should evaluate technologies such as message queues, tabular storage, and object storage during application design phase.