Metadata and Data Management for the Keck Observatory Archive

This is the title of a poster (spie14_koa_poster) by Hien Tran et al. (the et al. are the Keck Archive team), presented at the SPIE  Software and Cyberinfrastructure conference, part of
“Astronomical Telescopes + Instrumentation” in Montréal, Quebec, Canada.

Data and metadata management are difficult problems, but solving them are essential in making data self-describing and allowing long-term scientific usefulness of the data. Tran’s poster described the particular problems faced by the Keck Observatory Archive in archiving data from the eight instruments in active use at the W. M. Keck Observatory (WMKO).WMKO began operations in 1993, and most of the data produced were not designed with modern archiving in mind or intended to be released publicly. Observers have complete control over their observations, and the instruments were built by different teams. These circumstances have led to collections of metadata that are heterogenous across instruments and often do not encode the type of data contained within each file. For example, it is difficult to determine if an observation is of a science target or a sky frame required for proper data reduction.  The correct owners of observing programs are often hard to identify for time-domain and target-of-opportunity observations, or on split nights, shared by two or more principle investigators.

The KOA team developed methodologies and tools to address unsatisfactory state of affairs, and summarized below.

 Managing Heterogenous Metadata

Determining the File Type

This is invariably instrument dependent. The chart below shows the process for OSIRIS, an integral field spectrograph:

The calibration acquisition tool (CAT):

• A simple interface, the “calibration acquisition tool” (CAT) makes it easier for observers to take calibration data in a uniform manner and identify the file type in the FITS header.
• Allows observers to easily select and acquire the necessary calibrations for the science data all at once.
• With one press of a button, observers can obtain a sequence of calibrations, rather than manually reconfiguring the instrument for each type of calibration.
• Options to automatically set exposure times and to shutdown the instrument after the calibration acquisition sequence is finished are available.

Program Identification GUI

• Used while observing, the “program identification GUI” (PIG) sets certain keywords (i.e., OBSERVER for the observers’ names, and OUTDIR for the output data directory) in the header that facilitates the assignment of data to the proper owners.
• Supports “targets of opportunity” (ToOs) programs not scheduled on the telescopes.
• The active program is indicated in green. To switch programs, observers click on the appropriate button, which sets the OBSERVER and OUTDIR keywords.

Disclosure: I am the Project Manager for the Keck Observatory Archive