I am attending the SPIE Astronomical Telescopes + Instrumentation conference in Amsterdam, Netherlands (July 1-6, 2012). One of the most interesting talks I have attended so far, in the Software and Cyberinfrastructure for Astronomy conference, was given jointly by Rebecca Parsons (ThoughtWorks, Inc) and Andreas Wicenic (Univ of Western Australia) , on the topic of how agile development techniques are applicable to the Square Kilometer Array, an example of a project requiring massive distributed processing.
Debates have raged over the value of agile development in scientific computing, often considered a model best applied to massive distributed commercial efforts. The talk emphasized the need to focus on the principles of agile development:
- Rapid Feedback
- Frequent peer review
- Transparency into progress
- Early and often user involvement
- Test automation
Transparency, rapid feedback, frequent peer review and test automation were identified as the key issues in the SKA, where science functionality and user needs are likely to change rapidly in such a massive and complex project.
The importance of test driven development was emphasized. Given the importance of frequent releases and changing and granular requirements, an automated testbed is requisite. Perhaps most crucial of all is is the need for continuous integration of components that are developed in distributed environments. Thus, there is a need for strict integration requirements and the capability to perform fast and reliable builds.