Approaches to Technology Evaluation in the Era of Data Intensive Astronomy

This was the title of my talk at the “Innovations in Data Intensive Astronomy” Workshop, held at NRAO, Green Bank, WV from May 3-5 2011. I posted a short article about it last week. This week,  I thought I would outline some of the ideas in the talk; some of them are subjects of  blog posts, so I will supply links where appropriate.

Expected growth in astronomy data volume

Expected growth in astronomy data volume

Astronomy is a data intensive science. There are already 1 PB of public science data accessible from electronic astronomy archives, and this is growing at an accelerating rate of 0.5 PB/yr.  Querying and analyzing these data will require a new business model of astronomical computing, as was recognized by many attendees at the meeting. Datasets will be so large that large scale transfer of data to astronomers machines will not be possible, and the processing will have to move to do the data.

So how do we go about evaluating technologies that will be of value to astronomers in this new era? There are, I think, several approaches needed. One of the most important will be to do careful cost and performance evaluations of different technologies to understand their strengths and  weaknesses.  I cited the examples of cloud computing and the use of GPU’s, both covered in this blog. We will also have to develop tools  that help astronomers evaluate the best technology for their purposes; these don’t exist at the moment.

Having to move code to the data will have profound implications for code development. Code will have to consist of portable, scalable, environment agnostic components that can be plugged together to build new applications. I used the Montage mosaic engine as an example of how such code might look.

Design of the Montage mosaic engine

Design of the Montage mosaic engine

Finally, the culture of astronomy will change dramatically. Among the changes that I think need to take place are:

  • Greater recognition of the role of software engineering: provide career-paths for IT professionals; and next generation software skills should be a mandatory part of graduate education.
  • An on-line journal devoted  to computational techniques in astronomy.
  • Share computational knowledge from different fields and take advantage of it, through an organization such as the U.K. Software Sustainability Institute, a brain trust for software knowledge, if you will.
This entry was posted in astroinformatics, Astronomy, Cloud computing, cyberinfrastructure, Data Management, High performance computing, image mosaics, information sharing, Parallelization, programming, software engineering, software maintenance, software sustainability and tagged , , , , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s