Automating OSIRIS Data Reduction for the Keck Observatory Archive

The Keck Observatory Archive (KOA) will by the end of this calendar year curate all observations – previously acquired as well as newly acquired data – made with all seven active instruments at the W. M. Keck Observatory. One of the major challenges for the archive team has been to create scientifically useful, reduced browse products over many nights of observations, thousands in some cases. There are two reasons why this is a difficult task:

  • The Keck Observatory is a classically scheduled observatory, and data are usually not obtained in a uniform fashion from night to night, such that a particular frame cannot be readily identified as a sky, science, calibration or dark frame.
  • Data reduction pipelines are often designed for interactive use.

Now these issues are general problems faced by archives of ground-based observatories, and are an aspect of a larger problem of creating self-describing data sets. Solutions are generally custom to a particular instrument, and I describe here the solution for one instrument, the OH-Suppressing Infra-Red Imaging Spectrograph (OSIRIS), an integral field spectrograph that also has a separate diffraction-limited imaging camera. It uses an array of tiny lenslets to simultaneously produce spectra at up to 4096 locations. It covers the spectral range of 1.0 to 2.4 microns and has been used for over 1,000 nights of observations.

The OSIRIS instrument team released an interactive pipeline in IDL that is normally accessed through a GUI.  The program processes an OSISRIS “dataset,” a collection of dithered frames obtained sequentially in time with the same filter, scale, and exposure time combination, to produce a data cube used for analysis. It does more iterations than a quick-look version used at the telescope to give a better construction of the data cubes. The spectral extraction process requires maps of the instrumental PSF of each lenslet at all possible wavelengths. These maps are called “rectification matrices” which are created periodically for each filter and plate scale combination. The entire data reduction package, including the GUIs to fill in the XML input files required by the pipeline can be  found at the following page:

The KOA team has automated all the reduction parameters to process the 200,000 science files that it will archive. The reduced products will be served along with the raw data, and they will be made available to PI’s starting in December 2013 and to  the public starting in December 2014.   The automation involves two steps, which produce an XML structure for input to the pipeline:

  • Performing zeroth order KOA header validation and  image typing of each frame – is this a flat, dark, object? Is this from the spectroscopic or imaging detector?  KOA only performs level 1 processing for spectroscopic object frames. The algorithm is summarized below:
  • imtypDetermining the best files for background subtraction for spectroscopic files. Observers can set which of the frames in the dataset to be a sky frame, using the OSIRIS planning GUI during observing. This process would set the header keyword ISSKY to “true,” but is often not performed: the algorithm then uses other characteristics to identify sky frames. Here is the algorithm:


Finally, KOA has added a quality metric keyword SKYTYPE to each reduced FITS file header. It shows in which path the algorithm found the sky file. JPEG browse products will be produced for each reduced data cube, along with all raw imaging data. Included in the browse products are a JPEG image of the reduced cube that has been median stacked along the wavelength axis and a JPEG of the ”depth plot” or sum of all spectra at each pixel will be also included in the archive.

A poster describing this work is available here: OSIRIS_poster.

This work was described in a poster presented at ADASS XXIII by J. Holt, H. D. Tran, R. Goodrich, J. Lyke, G. B. Berriman, C .R. Gelino, and the KOA Team. I wish to thank Jennifer Holt for permission to present this material. Disclosure: I am the project manager for KOA.

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s