TRES spectra

Processing TRES Spectrograph Data
Jessica Mink, 2024-March-19 2018-October-22

Telescope Data Center 		TRES ThAr Image
[TDC Home] [TDC Search] [OIR Home]

[Observation] [IDL Pipeline)] (Software) [Archive]

[TRES Data Directories] [Raw Data Keywords] [Reduced Data Keywords]

All TRES data since July 6, 2020 has been reduced this way

Task

Commands to Reduce Last Night's Data

1. Create and catalog TRES raw data archive

Unless otherwise noted, commands can be run from any directory; I run them from the tres home directory. All commands starting with t can be run for nights other than the night before by adding the date as yyyy-mm-dd or yyyy.mmdd as an argument. Always use the date if running after 6:00 pm Eastern Time.

Use tall to check on the state of last night's data in the transfer directory. If is not all there, use tsync to transfer last night's data from Mt. Hopkins.

Use trcp to transfer the raw data for one night from the unmodifiable raw directory to a modifiable raw directory. If no date is given, last night's data is assumed. In this second online copy of raw data files, object and file naming errors can be corrected and bad data files removed. Keeping this second copy started early in TRES' career, when its Cambridge raw data disk crashed and it took a full month to recover using backups.

This command also assigns a reduced file number (RFN=[yyyymmdd].[nnn]) to each FITS file and adds it to a tab-separated database of all raw files and one of all raw object files, if appropriate. The raw object file database can be searched using tresearch -r.

For last night's data: 

> date
Wed Apr 21 11:29:20 EDT 2021
> tsync
-rw-r----- 1 tres tresgrp 5057280 Apr 21 07:54 0164.HIP093984.fits
-rw-r----- 1 tres tresgrp 5057280 Apr 21 07:56 0165.HIP093984.fits
-rw-r----- 1 tres tresgrp 5057280 Apr 21 07:57 0166.HIP093984.fits
-rw-r----- 1 tres tresgrp 5057280 Apr 21 07:59 0167.HIP099841.fits
-rw-r----- 1 tres tresgrp 5057280 Apr 21 08:02 0168.COMP.fits
-rw-r----- 1 tres tresgrp 5057280 Apr 21 08:04 0169.Vega.fits
-rw-r----- 1 tres tresgrp 5057280 Apr 21 08:04 0170.Vega.fits
-rw-r----- 1 tres tresgrp 5057280 Apr 21 08:05 0171.Vega.fits
-rw-r----- 1 tres tresgrp 5057280 Apr 21 08:06 0172.COMP.fits
-rw-r----- 1 tres tresgrp   13310 Apr 21 08:07 tres.log

> trcp
CPRAW: copy 198 files from /home/tres/araw/2021.0420 to /home/tres/traw/2021/2021.0420
20210420.0001 2021.0420/0001.COMP.fits,0
. . .

20210420.0198 2021.0420/0198.BIAS.fits,0

FILENAME             	OBJECT     
---------------------	-----------
0002.sky.fits        	sky        
. . .

0160.Vega.fits       	Vega

2. Run TRES reduction and classification IDL pipeline

Type treduce to run Lars Buchhave's TRES pipeline from the TRES IDL directory, /home/tres/idl/tres. It runs in the background to extract and calibrate 51 orders of each TRES object spectrum from one night. The night sky spectra in each file is not extracted. The pipeline also cross-correlates the RV31 order of the object spectra to a library of templates and saves the results in Postscript graphs, text files, and FITS header keywords. Information is extracted from various resulting files to create two databases, one including the FITS file path and best template and the other including the file path of the Postscript summary graph. Blaze-removed copies of the FITS spectra are also computed and saved in a b/ directory for the observing date. All three of these can be searched using tresearch with no, "-p", and "-b" arguments. When working remotely, check progres with tws and twp so that a time/interaction-limited Internet connection doesn't cause problems with the pipeline.

When a subset of a night's data needs to be reduced, or it needs to be processed in other than the normal default order, see this page.

reduce_tres takes a long time, so the twl, tws, twp, and twf commands have been added to check reduction progress.

For last night's data:
> treduce

> tws
Listing 27 spectrum FITS files in /home/tres/tred/2021-04-20 by RFN
20210420.0002 sky_2021-04-21_00h28m29s_cb.spec.fits
 . . .

20210420.0158 Vega_2021-04-21_12h09m31s_cb.spec.fits

> twp
nps = 27
TWP: Listing 27/27 TRES Postscript files in /home/tres/tred/2021-04-20/classifications
 . . .

sky.classify.TR.59325.52047.ps

> twf
Reading from /home/tres/tred/2021-04-20
Listing 27 spectrum FITS files in /home/tres/tred/2021-04-20
FILENAME                                      WS_ORG_N
...
Vega_2021-04-21_12h09m31s_cb.spec.fits        3
...
sky_2021-04-21_00h28m29s_cb.spec.fits         15

REDUCE_TRES: Reduction of TRES data for 2021-04-20 is complete
[1]    Done                          reduce_tres 2021.0420


The spectra are listed in files.list, which is split into
file1.list containing single exposures, which are likely to have more cosmic
rays, and filen.list those with multiple exposures, which dilute or remove most
cosmic ray exposures.

3. Check TRES spectra

Plot all of the TRES spectra using shell scripts which through tres.tresplot call the RVSAO spectrum plotting program rvsao.qplot. It is automatically in editing mode and waits after displaying each order of the spectrum after removing the hot pixels and bright night sky lines which are listed in the file badtreslines.dat in the directory [IRAF home directory]/extern/tres/lib/badtreslines.dat. The user must then interactively remove cosmic rays.
tplot1 plots the spectra in file1.list, containing single exposures, which are likely to have more cosmic rays, which will require more work to remove. The longer the exposure, the more likely the spectrum is to have cosmic rays.
tplotn plots the spectra in filen.list, those made up of multiple exposures, which dilute or remove most cosmic ray exposures. Files with on the order of 100 counts and exposures longer than 2000 seconds are most likely to have significant cosmic rays. A log file, tresplot.log is written noting the pixels which have been changed. 		For last night's data: 
 
> tplot1
/home/tres/tred/2021-04-20
Plotting 1 single-exposture TRES spectrum on 2021-04-20 from order 1
. . .

> tplotn
/home/tres/tred/2021-04-20
Plotting 26 multiple-exposture TRES spectra on 2021-04-20 from order 1
. . .

Or for a specific night: 

> tplot1 2021-06-01
/home/tres/tred/2021-06-01
Plotting 3 single-exposture TRES spectra on 2021-06-01 from order 1
. . .

> tplotn 2021-06-01
/home/tres/tred/2021-06-01
Plotting 19 multiple-exposture TRES spectra on 2021-06-01 from order 1
. . .

4. Re-run TRES classification IDL pipeline

Re-run the classification part of the pipeline in the TRES IDL directory /home/tres/idl/tres, where all of the needed programs exis, and update the database files, sending messages of pipeline completion to colleagues, plus separate messages containing formatted summaries of results from Kepler and TRES follow-up spectra. This task can be started from any directory. It runs in in the background because no interaction is necessary.

tcp lists the resulting Postscript files with the times so you can see how many have been recreated so far.

For last night's data: 
 
> treclass
[1] 25607
> /bin/rm: No match.
/bin/rm: No match.
/bin/rm: No match.
classify_tres1 2021-04-20
inputDir is /home/tres/tred/2021-04-20
Spectrum directory is /home/tres/tred/2021-04-20
classify processing log to /home/tres/tred/2021-04-20/tres_classifyx_20210421-11:45:02.log

> tcp
TCP: Listing 27/27 TRES Postscript files in /home/tres/tred/2021-04-20/classifications
-rw-r--r-- 1 tres tresgrp 296965 Apr 21 11:55 /home/tres/tred/2021-04-20/classifications/sky.classify.TR.59325.52047.ps
-rw-r--r-- 1 tres tresgrp 308079 Apr 21 11:55 /home/tres/tred/2021-04-20/classifications/W62019.classify.TR.59325.76690.ps
-rw-r--r-- 1 tres tresgrp 299676 Apr 21 11:54 /home/tres/tred/2021-04-20/classifications/W61064.classify.TR.59325.71140.ps
. . .
-rw-r--r-- 1 tres tresgrp 313423 Apr 21 11:45 /home/tres/tred/2021-04-20/classifications/67-364.classify.TR.59325.67945.ps
>

RECLASS_TRES: Reclassification of TRES data for 2021-04-20 is complete
[1]    Done                          reclass_tres 2021-04-20

5. Run multi-order cross-correlations of red TRES spectra

Run tsm to use Jonathan's Python supermulti program, modified by Emily Pass, to run multi-order cross-correlations on TRES spectra of objects cooler than 4500 degrees. Because it is multi-tasked, this task is best run on a compute server. It can be started from any directory. It runs in in the background because no interaction is necessary.

When it is complete, run tressm1 to copy results to the TRES data archive in yyyy-mm-dd/SM/.txt.

If any TESS objects have been observed, also run tsmJK which will check their spectral types and occasionally pick up some spectra that tsm missed. If any new TESS specra are processed, run tressm1 again to transfer all of the new results to yyyy-mm-dd/SM/. Run tscd to go to that directory and edit duplicate lines out of to_upload.txt, and cd back to the tres home directory.

Run tesssm1 to upload TESS object information to the TESS web server. It takes a while, but it prints what it is doing and returns to the shell when it is done.

For last night's data: 
> tsm
/data/tres/python/forAllyson
2023-03-29 1
TSM: Running supermulti on TRES data for night of 2023-03-29
[1] 61025
/data/tres/python/forAllyson
fleming.tres 113> /home/tres/tred/2023-03-29/HIP063510_2023-03-30_06h44m48s_cb.spec.fits
/data/tres/python/forAllyson/jonathan_code/tres/read_spec.py:201: UserWarning: object HIP063510 not found in srclist
  warnings.warn("object " + objname + " not found in srclist")
#Filename                                                              BJD-TDB       vrad     e_vrad     h     exptime vsini 
/home/tres/tred/2023-03-29/HIP063510_2023-03-30_06h44m48s_cb.spec.fits 2460033.7926  -12.8291 0.073739   0.793  800.00 10.06
/home/tres/tred/2023-03-29/S0236741269_2023-03-30_10h33m17s_cb.spec.fits
/data/tres/python/forAllyson/jonathan_code/tres/read_spec.py:201: UserWarning: object S0236741269 not found in srclist
  warnings.warn("object " + objname + " not found in srclist")
#Filename                                                                BJD-TDB       vrad     e_vrad     h     exptime vsini 
/home/tres/tred/2023-03-29/S0236741269_2023-03-30_10h33m17s_cb.spec.fits 2460033.9500 -122.6830 0.078199   0.593 1200.00  4.29
After making sure that tsm has processed its last spectrum by typing 
> ps
   PID TTY          TIME CMD
 12031 pts/0    00:00:00 tcsh
 67101 pts/0    00:00:00 ps
If python tasks are no longer running, copy results to the TRES archive 
> tressm1
TRESSM1: 2 /home/tres/tred/2023-03-29/SM/to_upload.txt super-multi results copied
from /data/tres/python/forAllyson/results to /home/tres/tred/2023-03-29/SM
Copying from list /home/tres/tred/2023-03-29/SM/pdf.list to directory /home/tres/tred/2023-03-29/SM

>

6. Run multi-order cross-correlations of TRES spectra

Run tlsd to use George Zhou's Python tres_lsdanalysis program to run multi-order cross-correlations on TRES spectra of objects warmer than 4500 degrees. Because it is multi-tasked, this task is best run on a compute server. It can be started from any directory. It runs in in the background because no interaction is necessary, but it doesn't let you know when it is done. Use ps and tlll to find out whether it is finished. When output.exoplanetdk appears somewhere in the directory listing given by tlll, it will be complete.

When it is complete, run tesslsd1 to copy results to the TRES data archive and create a file with the TESS results in yyyy-mm-/LSD/tesslsd.txt.

For date on night of 2022-08-13: 
> tlsd 2022-08-13
Running TRES lsdanalysis Python progam for night of 2022-08-13
> tlll 2022-08-13
Files for 5 / 5 objects in /data/tfop/tres_lsd_reduced/2022-08-13:
total 636
-rw-r--r-- 1 tres tfopgrp 58360 Aug 14 12:54 H4628_2022-08-14_11h13m23s_cb.spec.fits.ccf
-rw-r--r-- 1 tres tfopgrp   243 Aug 14 13:26 H4628_2022-08-14_11h13m23s_cb.spec.fits.rv
-rw-r--r-- 1 tres tfopgrp 49915 Aug 14 13:26 H4628_2022-08-14_11h13m23s_cb.spec.fits_LSD.png
-rw-r--r-- 1 tres tfopgrp 58470 Aug 14 12:54 HIP005586_2022-08-14_11h08m48s_cb.spec.fits.ccf
-rw-r--r-- 1 tres tfopgrp   247 Aug 14 13:26 HIP005586_2022-08-14_11h08m48s_cb.spec.fits.rv
-rw-r--r-- 1 tres tfopgrp 50084 Aug 14 13:26 HIP005586_2022-08-14_11h08m48s_cb.spec.fits_LSD.png
-rw-r--r-- 1 tres tfopgrp 58398 Aug 14 12:54 HIP010661_2022-08-14_11h42m11s_cb.spec.fits.ccf
-rw-r--r-- 1 tres tfopgrp   255 Aug 14 13:27 HIP010661_2022-08-14_11h42m11s_cb.spec.fits.rv
-rw-r--r-- 1 tres tfopgrp 57337 Aug 14 13:27 HIP010661_2022-08-14_11h42m11s_cb.spec.fits_LSD.png
-rw-r--r-- 1 tres tfopgrp 58442 Aug 14 12:54 HIP014328_2022-08-14_11h51m29s_cb.spec.fits.ccf
-rw-r--r-- 1 tres tfopgrp   251 Aug 14 13:27 HIP014328_2022-08-14_11h51m29s_cb.spec.fits.rv
-rw-r--r-- 1 tres tfopgrp 49521 Aug 14 13:27 HIP014328_2022-08-14_11h51m29s_cb.spec.fits_LSD.png
-rw-r--r-- 1 tres tfopgrp 58263 Aug 14 12:54 T0416739490_2022-08-14_11h18m58s_cb.spec.fits.ccf
-rw-r--r-- 1 tres tfopgrp   255 Aug 14 13:26 T0416739490_2022-08-14_11h18m58s_cb.spec.fits.rv
-rw-r--r-- 1 tres tfopgrp 61415 Aug 14 13:26 T0416739490_2022-08-14_11h18m58s_cb.spec.fits_LSD.png
-rw-r--r-- 1 tres tfopgrp  1326 Aug 14 17:29 output.exoplanetdk
After making sure that tlsd has finished processing its last spectrum by typing 
> ps
   PID TTY          TIME CMD
 12031 pts/0    00:00:00 tcsh
 67101 pts/0    00:00:00 ps
If no other tasks are no longer running in this window, copy results to the TRES archive 
> tesslsd1 2022-08-13
TESSLSD1: lsd analysis results copied from /data/tfop/tres_lsd_reduced/2022-08-13 to /home/tres/tred/2022-08-13/LSD
TESSLSD1: 1 lines copied from /home/tres/tred/2022-08-13/LSD/output.exoplanetdk to /home/tres/tred/2022-08-13/LSD/tesslsd.txt
>