Processing TRES Spectrograph Data
Jessica Mink, 2023-May-08 2018-October-22

2. Run TRES reduction and classification IDL pipeline

• treduce runs last night's data through reduce_tres, Lars Buchhave's TRES IDL pipeline reduce_tres, in the /home/tres/idl/tres directory.
  For other nights, treduce yyyy-mm-dd reduces the data in /home/tres/traw/[yyyy]/[yyyy.mmdd] and runs reduce_tres yyyy.mmdd in /home/tres/idl/tres/ for any one date or for a list of dates, one per line, in a file invoked as @[file].

  When a subset of a night's data needs to be reduced, or it needs to be processed in other than the normal default order, see this page.

  Here are all of the programs which it runs:

  • reduce_tres1 is called to run the IDL pipeline by invoking tres_reducex.pro . Raw data is read from ~/traw/[yyyy]/[yyyy.mmdd]/ and reduced data is written to ~/tred/[yyyy-mm-dd]/ in the IRAF .ec format, that is, with a nonlinear wavelength solution in the header. Spectra are cross-correlated against a library of template spectra, and the results with the best template for each spectrum are written to the directory ~/tred/[yyyy-mm-dd]/classifications/ . There are three files for each spectrum. Log files for the date are listed in ~/tred/[yyyy-mm-dd]/log.list

  • tresdb which calls readlog to extract information from ~/tred/[yyyy-mm-dd]/classification/*.log files and recreates the names of the FITS files which match them, which is a bit tricky because the time in the two file names is in totally different formats and units. The output is piped into a local tres.db file which is then appended to the master TRES ~/tred/tres.db file. A local tresps.db file is also created for the Postscript graphs of the results and appended to the master TRES ~/tred/tresps.db file.

  • The barycentric velocity correction is extracted from the Postscript file containing a plot of the RV31 order and appended to the FITS spectrum header as BCV. The midtime of the observation as Solar System Barycentric Julian Date, extracted from the corresponding log file for tres.db is added to the FITS header as HJDN. HJDN is converted to Geocentric Julian Date and added to the header as GJDN, and as a modified Julian Date of the midtime, MJD-MID. bcvcorr invokes the IRAF rvsao.bcvcorr program to compute the barycentric velocity corrections and adds them to each FITS spectrum header as BCV, along with the observation mid-time Solar System Barycentric Julian Dates as HJDN.

  • tresblaze removes the blaze function order by order from each of the spectrum files and place the results in ~/tred/[yyyy-mm-dd]/b/ with the same names.

  • reduce_sky1 reduces any twilight sky spectra from each night, and places them in a separate directory ~/tred/Sky/[yyyy-mm-dd]/

  • reduce_tres emails staff that pipeline has been run for this date.

  • treslist runs the following summarizing tasks:
    • tresklist extracts Kepler objects from the database into a local tresk.db file. Results are reformatted into a local tres.kfop tables of the results.
    • treskslist extracts Kepler standards from the database into a local Ii>tresks.db file. Results are reformatted into a local tres.ksfop table for the date, and a message is mailed to the local staff.
    • tresk2list extracts Kepler 2 objects from the database into a local tresk2.db file. Results are reformatted into a local tres.k2fop table for the date, and a message is mailed to the local staff.
    • trestlist extracts TESS objects from the database into a local trest.db file. Results are reformatted into a local tres.tfop table for the date, and a message is mailed to the local staff.

  • tresnexp finds the number of raw files used for each reduced file and creates the following two list files of reduced TRES FITS spectra:
    • file1.list contains a list of spectrum files which were reduced from only one raw image, so there are likely to be cosmic rays in the spectra
    • filen.list contains a list of spectrum files which were reduced from multiple images, so there are less likely to be cosmic rays to be removed from the spectra.

  • tresback backs up the uncorrected reduced spectra by copying the entire reduced directory to /home/tres/tred/Back/[yyyy-mm-dd]back.

• reduce_tres takes a long time--over an hour on a long night--to run. If you run it in the background, its progress can be monitored with the following tasks:
  • treslog [yyyy-mm-dd] runs tail -f on the most recent tres_reducex_yyyymmdd-hh:mm:ss.log file in the directory /home/tres/tred/[yyyy-mm-dd].
    twl with no argument runs treslog on last night's data

  • tressort [yyyy-mm-dd] sorts the existing FITS files in the directory /home/tres/tred/[yyyy-mm-dd] in sequence number order so you can find out how far the extraction process has gone. Extracted files contain the sequence number of their first component image in their header.
    tws with no argument runs tressort on last night's data

  • tresps [yyyy-mm-dd] lists the existing Postscript files in the /home/tres/tred/[yyyy-mm-dd]/classification/ directory alphabetically. It first gives [number of Postscript files] / [number of FITS files] if all of the night's data has been extracted. If not, it gives the number of FITS files processed so far and lists them alphabetically.
    twp with no argument runs tresps on last night's data

  • tresfits [yyyy-mm-dd] lists the existing FITS files in /home/tres/tred/[yyyy-mm-dd] and the number of raw files from which each of them is reduced.
    twf with no argument runs tresfits on last night's data

1. Create and catalog TRES raw data archive

2. Run TRES reduction and classification IDL pipeline

3. Check TRES spectra and clean cosmic rays and hot pixels

4. Re-run TRES classification IDL pipeline

5. Run multi-order cross-correlations of red TRES spectra

6. Run multi-order cross-correlations of selected TRES spectra

> treduce
Reducing TRES data for night of 2023.0405
/home/tres/idl/tres
[1] 196412
fleming.tres 24> ps
   PID TTY          TIME CMD
165821 pts/0    00:00:00 tcsh
196412 pts/0    00:00:00 reduce_tres
196453 pts/0    00:00:00 reduce_tres1
196495 pts/0    00:00:00 idl
196517 pts/0    00:00:00 ps

> ps
   PID TTY          TIME CMD
165821 pts/0    00:00:00 tcsh
196412 pts/0    00:00:00 reduce_tres
196453 pts/0    00:00:00 reduce_tres1
196495 pts/0    00:11:21 idl
196748 pts/0    00:00:00 ps

> tws
Listing 1 spectrum FITS files in /home/tres/tred/2023-04-05 by RFN
3 150 20230405.0044

> tws 2023-04-05
Listing 24 spectrum FITS files in /home/tres/tred/2023-04-05 by RFN
20230405.0044 3 150 HIP035253_2023-04-06_02h21m20s_cb.spec.fits
20230405.0047 3 135 HIP024130_2023-04-06_02h26m20s_cb.spec.fits
20230405.0050 3 270 HIP027269_2023-04-06_02h30m34s_cb.spec.fits
20230405.0054 3 3 Sirius_2023-04-06_02h42m32s_cb.spec.fits
20230405.0058 3 2890 T0011286229_2023-04-06_02h47m57s_cb.spec.fits
20230405.0062 6 30 HD39801_2023-04-06_03h46m45s_cb.spec.fits
20230405.0069 3 180 H38230_2023-04-06_03h58m10s_cb.spec.fits
20230405.0073 3 3060 T0090853136_2023-04-06_04h07m10s_cb.spec.fits
20230405.0077 3 1980 T0220144363_2023-04-06_05h02m52s_cb.spec.fits
20230405.0081 3 270 T0456260074_2023-04-06_05h42m58s_cb.spec.fits
20230405.0085 3 3240 T0094823134_2023-04-06_05h53m23s_cb.spec.fits
20230405.0089 3 600 T0265010875_2023-04-06_06h55m12s_cb.spec.fits
20230405.0093 3 180 H65583_2023-04-06_07h15m41s_cb.spec.fits
20230405.0097 3 300 S0027066428_2023-04-06_08h08m40s_cb.spec.fits
20230405.0101 3 400 S0224292761_2023-04-06_08h19m05s_cb.spec.fits
20230405.0105 3 330 S0068121645_2023-04-06_08h31m52s_cb.spec.fits
20230405.0109 3 540 S0351370842_2023-04-06_08h44m20s_cb.spec.fits
20230405.0113 3 720 S0321553459_2023-04-06_08h59m29s_cb.spec.fits
20230405.0117 3 120 H127334_2023-04-06_09h20m20s_cb.spec.fits
20230405.0121 3 3600 BA219012316_2023-04-06_09h28m25s_cb.spec.fits
20230405.0126 3 12 Vega_2023-04-06_11h05m17s_cb.spec.fits
20230405.0130 3 800 S0388310112_2023-04-06_11h13m08s_cb.spec.fits
20230405.0134 3 2940 T0366846982_2023-04-06_11h34m34s_cb.spec.fits
20230405.0138 3 129 HD185144_2023-04-06_12h28m40s_cb.spec.fits
>

> twp 2023-04-05
nps = 24
TWP: Listing 24/24 TRES Postscript files in /home/tres/tred/2023-04-05/classifications
-rw-r--r-- 1 tres tresgrp 303631 Apr  6 14:37 BA219012316.classify.TR.60040.91700.ps
-rw-r--r-- 1 tres tresgrp 297240 Apr  6 14:37 H127334.classify.TR.60040.89359.ps
-rw-r--r-- 1 tres tresgrp 301019 Apr  6 14:38 H38230.classify.TR.60040.66485.ps
-rw-r--r-- 1 tres tresgrp 296665 Apr  6 14:38 H65583.classify.TR.60040.80486.ps
-rw-r--r-- 1 tres tresgrp 300114 Apr  6 14:38 HD185144.classify.TR.60041.01998.ps
-rw-r--r-- 1 tres tresgrp 301131 Apr  6 14:39 HD39801.classify.TR.60040.65654.ps
-rw-r--r-- 1 tres tresgrp 298808 Apr  6 14:39 HIP024130.classify.TR.60040.60004.ps
-rw-r--r-- 1 tres tresgrp 304893 Apr  6 14:39 HIP027269.classify.TR.60040.60461.ps
-rw-r--r-- 1 tres tresgrp 295547 Apr  6 14:40 HIP035253.classify.TR.60040.59938.ps
-rw-r--r-- 1 tres tresgrp 303381 Apr  6 14:40 S0027066428.classify.TR.60040.84513.ps
-rw-r--r-- 1 tres tresgrp 304584 Apr  6 14:41 S0068121645.classify.TR.60040.86207.ps
-rw-r--r-- 1 tres tresgrp 302190 Apr  6 14:41 S0224292761.classify.TR.60040.85241.ps
-rw-r--r-- 1 tres tresgrp 303111 Apr  6 14:42 S0321553459.classify.TR.60040.88234.ps
-rw-r--r-- 1 tres tresgrp 300859 Apr  6 14:42 S0351370842.classify.TR.60040.87172.ps
-rw-r--r-- 1 tres tresgrp 304087 Apr  6 14:42 S0388310112.classify.TR.60040.97082.ps
-rw-r--r-- 1 tres tresgrp 278990 Apr  6 14:44 Sirius.classify.TR.60040.61297.ps
-rw-r--r-- 1 tres tresgrp 304273 Apr  6 14:44 T0011286229.classify.TR.60040.63186.ps
-rw-r--r-- 1 tres tresgrp 301385 Apr  6 14:44 T0090853136.classify.TR.60040.68901.ps
-rw-r--r-- 1 tres tresgrp 304989 Apr  6 14:45 T0094823134.classify.TR.60040.76553.ps
-rw-r--r-- 1 tres tresgrp 302277 Apr  6 14:45 T0220144363.classify.TR.60040.72193.ps
-rw-r--r-- 1 tres tresgrp 300274 Apr  6 14:45 T0265010875.classify.TR.60040.79515.ps
-rw-r--r-- 1 tres tresgrp 308733 Apr  6 14:45 T0366846982.classify.TR.60040.99736.ps
-rw-r--r-- 1 tres tresgrp 306504 Apr  6 14:46 T0456260074.classify.TR.60040.73924.ps
-rw-r--r-- 1 tres tresgrp 280132 Apr  6 14:46 Vega.classify.TR.60040.96232.ps
> 

> wc: /home/tres/tred/2023-04-05/tres.db: No such file or directory
mv: cannot stat '/home/tres/tred/2023-04-05/tres.db': No such file or directory
wc: /home/tres/tred/2023-04-05/tresps.db: No such file or directory
mv: cannot stat '/home/tres/tred/2023-04-05/tresps.db': No such file or directory
ls: No match.
arg1="2023.0405" idate=2 date=2023-04-05 input=2023.0405
Copying /home/tres/tred/2023-04-05 to /home/tres/tred/Back/2023-04-05back
REDUCE_TRES: Reduction of TRES data for 2023.0405 is complete

Click for larger image



Click for larger image