Archive

The LCO science archive is the repository for all science data (raw, processed, and ancillary) and data products (source tables, final calibration files) produced by the LCO network. There are two ways to access data stored in the archive: the API (application programming interface) and its web frontend UI (user interface). The archive UI is a user-friendly interface to the API. To use the UI, users must login with the same username and password required for the observatory Portal. The archive API is a web service that provides methods to query the metadata associated with data products stored in the archive. To use the API, users must submit their Portal username and password to the API's authentication endpoint. Query results are returned in JSON, so they can be easily parsed by a wide variety of libraries and programs. When the desired data products are found, they can be downloaded using the urls in the metadata.

The use of the LCO archive and/or data implies acceptance of LCO's terms of use policy.

All image FITS files are Rice-compressed ("fpacked") to reduce file-size and increase transmission speed, and these bear fz file extensions when downloaded. These can be decompressed with CFITSIO funpack, although many common astronomy software applications (e.g. ds9) can interpret the compressed format.

The FITS header keywords and their meanings are given in this table.

Raw frames from science imagers should appear in the archive within a few minutes of being taken (subject to internet latency and occasional outages). Raw science frames have e00 suffixes in their basenames. The generation of final reduced frames is triggered as soon as the raw frames arrive at LCO headquarters. Reduced frames are processed using the best/most-current calibration files (bias/dark/flat) and are typically available in the archive a few minutes after the raw frames. The format of the final data products depends on the pipeline that is used to reduce the data (see below).

In the UI, the calibration frames (bias, dark, flat, etc) used in the processing of a science frame are revealed by clicking on the '+' symbol in the left-most column. Conversely, clicking on the '+' symbol left of a calibration frame will unfurl a list of all other frames to which that calibration frame was applied. In the API, the related calibration frames can be retrieved by using the 'related_frames' key in the API payload.

BANZAI pipeline

The BANZAI pipeline processes frames from LCO imagers: the Spectral cameras on the 2m telescopes, the Sinistro cameras on the 1m telescopes, and the SBIG cameras on the 0.4m telescopes. Processed images have e91 suffixes in their basenames. The images are multi-extension FITS files. Each file includes four extensions: the reduced image, the bad pixel mask (BPM), an array of pixel uncertainties (ERR), and the catalog of sources (CAT) detected by SEP in a FITS binary table. The catalog lists the pixel positions (X, Y), semi-major and semi-minor axes (A, B), positions angles (THETA), fluxes and errors (FLUX, FLUXERR) of each source.

The BANZAI pipeline began running on 1 April 2016.

NRES pipeline

High resolution spectra acquired by one of the nodes of the Network of Robotic Echelle Spectrographs are processed by a dedicated NRES pipeline. The data products are gzipped tar files that contain an fpacked main data (FITS) file and a set of pdf diagnostic plots.

FLOYDS pipeline

The Python/Pyraf FLOYDS pipeline processes data taken with the FLOYDS low-resolution spectrograph. The data product, including intermediate and final reduced files, as well as guider images, comes packaged in a .tar.gz file. This package will be associated with the data products that were used to create it, and vice versa.

ORAC-DR pipeline

Reduced data processed with the ORAC-DR pipeline are available for the period May 1, 2014 through May 31, 2016. These are files with the suffix e90 (final reduced) and e10 (quicklook reduced). Catalogs generated by SExtractor are archived as separate files with suffixes such as _e90_cat.fits_.

Data acquired for science projects sponsored by LCO Science Collaboration institutions and other contracted institutions is subject to a 12-month proprietary period. During this period, which begins on the date the data are acquired, only PIs and registered Co-Is of a science project have access to that project's data (and associated metadata).

Data acquired for educational projects and public outreach events are made public immediately. In addition, calibration observations (bias frames, dark frames, flat fields, etc.) and observations of photometric standard fields are also available to all users.

For imaging observations, the file name structure is: (site)(tel)-(instr)-YYYYMMDD-(frame)-(type)(red.level).fits . A representative frame name might be lsc1m005-fa03-20200108-0666-e91.fits.

Site

Telescope

Instrument

Date

YYYYMMDD: local date at start of night

Frame Number

Frame number: increases sequentially, per night

Observation Type

Reduction Level

By signing in to the Science Archive UI, you will be able to search for data acquired for projects for which you are the PI or a Co-I. Use the same username and password that you do to access the observatory portal. Public data is available without logging in. On the left side of the user interface are a set of fields which can be used to narrow down the results in the main table. Starting from the top, these fields are:

Date (calendar)

This widget specifies the time period from which results will appear. There are a few pre defined date ranges to choose from, including this semester, last semester and the last month. If you need a more specific time range, choose "Custom Range"

Proposal

This is the proposal from which the frames were taken. If you are logged in, your proposals will appear near the top. Public proposals are listed as well.

Point

The points fields allow you to search for data by position on the sky. Entering an object designation (eg. M42) in the Lookup field will use an online service to search for the object, return the RA/Dec and populate the RA and Dec search fields. You can also manually enter position in decimal or sexagesimal.

Object

This field directly searches the data's OBJECT fits keyword. It does not do any kind of spatial search.

Obstype

This is the type of observation. See the Filename elements section above.

Reduction level

The level of processing the data has gone through. See the Filename elements section above.

Site

The site at which the data were acquired.

Telescope

The telescope class with which the data were acquired.

Instrument

The instrument (camera) with which the data were acquired.

Filter

The filter used during the observation.

Exposure Time

The exposure time of the observation.

Other User Interface Elements

On the top right of the table there is a row of buttons.

The refresh button will reload the data in the table, preserving filters.

The columns button will allow you to pick which columns to show in the table. Some columns are hidden by default.

The Export table data button will export the displayed data in the table in a variety of formats, including csv and excel.

Search Example

Search for all reduced frames acquired for a particular project in the last 30 days.

1. In the upper left corner of the UI, click on the calendar box.
2. In the drop-down menu that appears, click on "Last 30 days".
3. Just below that calendar box, click on the "Proposal" field (where it says "All").
4. From the drop-down menu, select your proposal ID. For example, CON2016A-007.
5. Further down in the column of search filters, click on the "Reduction Level" field (where it says "All").
6. From the drop-down menu, select "Reduced (experimental)" to select frames processed through the BANZAI data reduction pipeline.
7. To select all of the frames displayed in the table, click on the check box at the top of the column of check boxes (left of the "Basename" column).
8. To download the selected frames, click on the blue "Download ##" button (just above the column of checkboxes), where "##" will be number of frames you selected.

To download data, select the row that contains the data you wish do download using the checkbox. Multiple rows can be selected at once. Rows will stay selected if you change the search filters, so you can gather as much data as you'd like.

Once you have selected the date you wish to download, click the download button. You will be prompted to download a .zip file containing your data. Alternatively, if you wish you can download a wget script instead that can be run as an executable on your machine. To download this script, select "wget script" from the dropdown menu that is activated by pressing the down caret next to the download button.

There is a daily request limit set on the archive. All logged in users, regardless if using the archive UI or API, are limited to 5000 requests a day. Non-authenticated users are limited to 3500 requests a day. Actions that count as a request include: performing a new search, changing pages, and downloading a zip file. While using the API, a single request is any time your code performs a http request against the api.

The API documentation is located on the LCO developers page . Changes to the archive API and authentication protocols were made in March 2022. Users may need to modify their login behavior or their scripts that interact with the archive.

Apr 6, 2020 - Release of BANZAI with major architectural changes that will enable the development of future pipelines, e.g. BANZAI-NRES. The extensions of the FITS data products are ['SCI','BPM','ERR','CAT']. The ERR extension contains an array of uncertainties associated with each image pixel.

Jan 31, 2020 - All pipelines (BANZAI, FLOYDS, NRES) were modified to retrieve raw data from LCO's S3 (cloud) archive.

Feb 6, 2019 - Release of the "real-time" BANZAI imaging pipeline. Quicklook data products are no longer available, but the pipeline creates final data products on the quicklook timescale, as soon as possible after raw images are received at LCO headquarters.

May 10, 2018 -Release of version 0.8 of the NRES pipeline. All FITS files, including the extracted spectrum, are included in a single multi-extension FITS file.

May 16, 2017 - The fluxes of catalog sources in BANZAI-processed images are now being correctly calculated in apertures with diameters in arcseconds. Prior to this change, the fluxes were calculated in apertures with diameters in pixels. The pipeline version number (PIPEVER keyword) was incremented to 0.7.9dev1197.

Aug 8, 2016 - Added ability to export displayed table data as well as view headers directly in the archive interface.

Aug 8, 2016 - Added ability to export displayed table data as well as view headers directly in the archive interface.

April 12, 2016 - Inclusion of all guide images in FLOYDS data products.

April 1, 2016 Initial release of LCOGT Science Archive.

May 1, 2014 - Start of LCOGT V1.0 Operations