How Should Astronomy Pay For Software?

So asked Frossie Enonomou at ADASS XXIV in Calgary, October 2014.  She gave her view of the answer to this question. It’s not of course about paying for software, but about paying people to write software.  Generally speaking, it’s not faculty who write the software, but, to quote Frossie:

  • People drawn from long-lived and/or multi-mission institutions – Funding in decline
    – Limited longevity even within such institutions
  • People coming from the private sector
    – We can’t compete in pay
    – We increasingly can’t compete on culture
  •  Postdocs (and similar) on soft money

Software is invariably written on soft money, and Frossie points out this model has all kinds of consequences:

  • No sustained, strategic development
  • Software is an afterthought on bigger grants
  • No technical assessment = no reward for success
  • No post-grant evaluation = no penalty for failure
  • Soft money has displaced other funding models

So how do we write top-class, sustainable software on soft money? Making code Open Source is one part of the solution – it encourages community participation, for one thing. Frossie suggested two new approaches as well: exploiting the AAS- WGAS and adopting the LSST model.

Let’s look at WGAS first. WGAS is the newly revitalized  AAS Working Group on Astronomical Software, whose goals are “to promote the interests of AAS members developing astronomical software, and to support the advancement of astronomical research through open-source software collaboration.” Frossie is the new chair, and she is organizing its efforts around a set of special interests groups. There are two proposed so far: Publication, to promote software sharing, publishing and preservation; and Common Technologies, to support the development and dissemination of APIs, libraries, formats and standards undertaken within the WGAS community. If you are a software engineer and are concerned about proper recognition for software, sign-up on GitHub at AAS-WGAS and get involved. I have joined up.

LSST has been described as a massive computing technology project with a telescope attached.  Frossie makes these points as to why a project like this is a good model for software development:

  • Software a primary mission component
  • Long horizon
  • Public software products
  • Open Source development model
  • Software re-use internally and externally
  • High level of external oversight from NSF

Frossie ended by re-iteratng the importance of Open Source, the need for overall ability to trump specific skills, the need for mentoring, a commitment to sustainability, all underpinned by an evolving set of best practices.

I wish to thank Ms. Frossie Economou for generously making her presentation available.

Posted in Astronomy, High performance computing, astroinformatics, software maintenance, software sustainability, cyberinfrastructure, social networking, careers, jobs, social media, information sharing, LSST, software engineering, programming, user communities, History of Computing!, Open Access, Career Advice, agile development, informatics, Scientific computing, Computing, Open Source | Tagged , , , , , , , , , , , | Leave a comment

From The Front Lines of ADASS 2014 – Visualization

Visualization technology is undergoing a transformation these days, and the October 7 morning session was devoted to advances in visualization. Here I will give a synopsis of some of the talks.

Chris Beaumont described Glue as a “Hackable User Interface,” by which he means interfaces that support not just visualization but the data analysis cycle. Perhaps the best way to get the flavor of his talk is to watch this youTube video of his excellent talk at SciPy2013:

and you can see some more videos on the Glue demo page at

Erik Rosolowsky spoke about the Cube Analysis and Rendering Tool for Astronomy (CARTA), a cube visualization tool under development to meet the needs of the Atacama Large Millimetre/submillimetre Array (ALMA).  When complete, it is intended in the long term as a replacement for the current CASA viewer. Its goal is to navigate and analyze large data cubes by leveraging the capabilities of the CASA viewer (full featured, well developed analytics, and  an expert user-base)  and the CyberSKA viewer (scalable among other things). It can be used as a stand-alone application or as part of an archive architecture. The project uses the PureWeb commercial middleware to allow server side processing, and supports plug-ins from python, yt and glue: plug-ins are a powerful way of extending capabilities and allowing users to customize the visualizer.

Andre Schaaf described how technology intended primarily for gaming can be adapted for use in astronomy, specifically the SDK Oculus Rift.

Pierre Fernique decribed how CDS has been developing and validating new methods to generate, publish and display huge astronomical image data cubes based on the Hierarchical Progressive Survey (HiPS) framework. Their goal is to allow astronomers to interact with massive cube survey data on their desktops with common clients and visualizers, primarily Aladin. Data cubes with two spatial dimensions and an additional spectral or temporal dimension are mapped onto HEALPix grids at different resolutions, and this enables zooming and panning across the sky and in the third dimension.  CDS has  demonstrated the approach on various flavors of cube data, and surveys of cube data, including wide sky coverage data Canadian Galactic Plane Survey, the pointed data cube observations of the CALIFA v500 survey of SDSS galaxies, and composite cubes constructed from multi-band mission imaging, including a 4TB WISE-cube (WISE 3.4, 4.8, 12 and 22 um bands) and a HST-cube built from HST imaging data in 13 bands (F110W, F160W…,F850LP).

I was looking forward to Slava Kitaeff’s talk on “Large astronomy imaging with JPEG2000.” but he was unable to attend. For interested parties, I am including his abstract below:

“The sheer volume of data anticipated to be captured by future radio telescopes, such as, The Square Kilometer Array (SKA) and its precursors present new data challenges, including the cost and technical feasibility of data transport and storage. Servicing such data as images to the end-user in a traditional manner and formats is going to encounter significant performance fallbacks. Thus, image and data compression are going to be important techniques to reduce the data size. We discuss the requirements for extremely large radio spectral-imaging data-cubes, and in this light we analyse the applicability of the approach taken in the JPEG2000 (ISO/IEC 15444) standards. We provide the quantitative analysis of the effects of JPEG2000’s lossy wavelet image compression algorithm on the quality of the radio astronomy imagery data. This analysis is completed by evaluating the completeness, soundness and source parameterisation of the Duchamp source finder using compressed data. We report that the JPEG2000 image compression can be effectively used for noise filtering. “

Posted in astroinformatics, Astronomy, Computing, Data mining, High performance computing, informatics, information sharing, programming, publishing, Scientific computing, software engineering, software maintenance, visualization | Tagged , , , , , , | Leave a comment

Valuing Software and Other Research Outputs – Talk by Dan Katz.

This is the title of a paper presented by Dan Katz at the Altmetrics Conference in London. Dan is the Program Director, NSF Division of Advanced Cyberinfrastructure, and one of the founders of the Working Towards Sustainable Software for Science: Practice and Experience workshops.

Software has become so essential to research that sustaining it requires that it is treated as infrastructure. This is one the drivers behind the organization of the Agency’s Advanced Cyberinfrastructure Division. Dan gives a fine summary in only a few slides on the NSF Software Infrastructure Projects and how funding of projects is carried out, along with links to resources. Briefly, there are three types of projects funded – software elements (e.g. apps, libraries), software frameworks, and Software Institutes. Once a proposal has a successful review, Dan works with divisional Program Officers to get them funded. If you are thinking of proposing, these slides are worth viewing for this information alone.

You can watch the talk here (starting at 45:31)

or view the slides on SlideShare at, or step through them here:


The crux of the talk is that is that the impact of software needs to be measured and credit needs to be given to software provider. The NSF is exploring ways of measuring impact, and has recently issued a dear colleague letter to seek new metrics for measuring impact. See CISE/ACI & SBE/SES Dear Colleague Letter: Supporting Scientific Discovery through Norms and Practices for Software and Data Citation and Attribution (NSF 14-059,  (

A desktop science app would not be measured the same way as a math library, for example. The NSF’s hypothesis is that “better measurement of contributions can lead to rewards (incentives), leading to career paths, willingness to join communities, leading to more sustainable software.” Dan points out, for example, that a software registry may be a powerful tool in assigning  credit.

Posted in astroinformatics, computer videos, Computing, computing videos, cyberinfrastructure, High performance computing, informatics, information sharing, metrics, Open Access, programming, publishing, Scientific computing, social media, social networking, software engineering, software maintenance, software sustainability, user communities | Tagged , , , , , , , , , , , , , , | Leave a comment

Enhancements to the Astronomy Source Code Library (ASCL)

The Astronomy Source Code Library (ASCL) is a rapidly growing, free online registry for astronomical source codes that have been used in research that have appeared in, or been submitted to, peer-reviewed publication. There are now over 900 codes indexed by the ASCL,  and this reflects a growing understanding in the community of the need to expose code in verifying science results.


The ASCL has been active in the community, in organizing workshops and panels on code sharing. It is indexed by the SAO/NASA Astrophysics Data System (ADS) and is citable by using the unique ascl ID assigned to each code. The ascl ID can be used to link to the code entry by prefacing the number with (

In response to its recent growth,  the Library has made a number of enhancements to its infrastructure and interfaces to improve services for users.  For example, it now uses a MySQL database for code records, and WordPress and phpbb discussion forums are fully integrated into the site.

The web page now offers a simpler submissions form; shows the the ten newest codes on the home page; provides much improved browsing, with abstract and compact views, and ordering of codes by alpha, reverse alpha, and newest or oldest additions; offers a one-click author search that can be expanded; as well as bibcode and discussion thread links in each record.

Section of the new browse page:


Future capabilities will include a dashboard for statistics and information,; a “See also” field for related codes; and Previous and Next links for paging through code entries.

The Astrophysics Source Code Library (ASCL) was founded in 1999 by Robert Nemiroff and John Wallin. The Editor is Alice Allen. Disclosure: I am a member of the advisory board

Posted in astroinformatics, Astronomy, cyberinfrastructure, Data Management, document management, informatics, information sharing, Internet, Open Access, Scientific computing, social media, social networking, software engineering, text processing, Web 2.0 | Tagged , , , , , , , , | Leave a comment