A New Python Pipeline for Reducing NIRSPEC Data Acquired at the Keck Observatory

The Keck Observatory Archive (KOA) (https://koa.ipac.caltech.edu) has just released a new data reduction pipeline (NSDRP) for data acquired with the Keck Observatory NIRSPEC cross-dispersed infrared echelle spectrograph. The pipeline is intended to reduce high-resolution (0.947 – 2.63 um) object spectra. All phases of reduction – including order location and tracing, flat fielding, background subtraction, spectral extraction, wavelength calibration and data product generation – proceed automatically without user intervention. You can read full details on the project web page, and the code is freely available for download from GitHub. NSDRP is optimized for point sources, but extended objects are accommodated provided there is a discernible peak in the spatial intensity profile. It can reduce a wide range of echelle and cross disperser angles and all of the high resolution slits. NSDRP works best on data that have:

  • A single point source in the slit
  • Reasonably bright targets with detectable continuum
  • NIRSPEC-1 through NIRSPEC-7 filters
  • Well-separated orders without overlapping
  • Sufficient exposure times (~> 30s) with detectable sky lines

Single frames can be reduced,  in which case sky windows adjacent to the spatial peak are used for background subtraction. “AB nod pairs” can be reduced in which case sky background is removed by pair subtraction. OH sky emission lines are used for wavelength calibration. For short exposures with weak sky lines, NSDRP uses the grating equation to approximate the wavelength scale. A  night of data can be reduced together with wavelength calibration determined from long exposure frames can be applied to short exposure frames taken in the same temporarily contiguous instrument configuration. Low-resolution mode and wavelength calibration using arc lamps have been identified for future development.

The NSDRP is written in the Python programming language and makes extensive use of common libraries including numpy, astropy, scipy and matplotlib. The software is written mainly in the procedural style, though the main application-specific data structures are implemented as Python classes.

Data products generated from the reduction results include flux and noise spectra and spatial profiles for each order and wavelength calibration line identification tables in ASCII and FITS formats as well as preview plots in PNG format. Two types of log files are produced: one to record per-night summary data and another to record details of data reduction on a per object frame basis.

KOA has used the pipeline to extract and serve browse-quality spectra from 1,821 nights of NIRSPEC high-dispersion data. The data products comprise a total of over 4,400,000 files altogether; see one sample below


 Rectified image, spatial profile and flux spectrum of the asteroid 511 Davida, extracted with the KOA NIRSPEC Data Reduction Pipeline from observations acquired on April 29, 2010 (H. Roe, PI).

The NSDRP  is described in detail in the software design document and the the data products document.

The high level design is as follows:



Below are some figures illustrating steps in the reduction process, taken from the design document:






Disclosure: I am the manager of the Keck Observatory Archive.

This entry was posted in archives, astroinformatics, Astronomy, Computing, cyberinfrastructure, Data formats, FITS, information sharing, Observatories, Pipelines, programming, publishing, Python, Scientific computing, software engineering, softwarte sustainability, user communities, W. M. Keck Observatory, workflows and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s