External SPECROAD: What Is It?

The Backstory

External SPECROAD or E-SPECROAD came about when my colleagues and I were awarded time on the Hectospec multi-fiber spectrometer on the MMT on Mount Hopkins in the summer of 2007. The MMTO is a "joint venture" between the University of Arizona and the Smithsonian Institution (via the Smithsonian Astrophysical Observatory aka SAO). We were awarded these nights in part because one of my colleagues is at the University of Minnesota and since the U of M is a partner on the Large Binocular Telescope, they have bought into a certain number of nights at the U of Arizona facilities and so they can apply for nights.

Image of the Hectospec Focal Plane showing many of the 300 optical fibers deployed as well as the two fiber positioning robots in the upper left and lower right. This image is from the Hectospec Photo Gallery which also contains movies of the instrument in action.

Unlike most projects utilizing the Hectospec, our project was a stellar spectroscopy project which requires relatively high resolution spectra at short wavelengths (as opposed to the low resolution red-focused spectra typical of extragalactic work). As such we used the 600 gpm grating centered at 4800 Ångstroms instead of the more commonly used 270 gpm grating. Furthermore, we were NOT SAO personnel, so we faced a problem of how to reduce our Hectospec data at our own home institutions without access to SAO computers.

After investigating the use of HSRED, a series of IDL scripts developed by Richard Cool at the U of Arizona (now at Princeton), I rejected that approach because the scripts were designed to work with the 270gpm grating and frankly, I am not an IDL expert. So next I turned to the Hectospec IRAF package. This Hectospec package has everything in it necessary for dealing with the Hectospec data, although you then need to know what to do with it. Since this was the first time I was reducing Hectospec data, I was hoping for a more 'cookbook' approach. I knew from talking to folks at the MMT that there was a set of shell scripts called SPECROAD used by the SAO folks to automate much of the process of reducing Hectospec data. It turned out they even provided a version of it the Hectospec IRAF package (in the Shell/ subdirectory). So rather than re-invent the wheel, I decided to try to use SPECROAD to reduce our data.

The Problem with SPECROAD

The SPECROAD scripts seemed like the safest approach to reducing the Hectospec data. All the steps for the user were laid out and the scripts could even take advantage of multiple processors on modern computers to run several IRAF processes in parallel, thus speeding up data reduction. However, I quickly discovered the existing SPECROAD scripts had some major issues:

1. Nonportability: The SPECROAD scripts were only designed to run on a Solaris computer at the SAO, with files in specific hard-coded locations using a specific IRAF environment with all the IRAF parameters set before the scripts were run. Along with the issues of simply porting the code to run on a new computer, I was faced with not knowing which IRAF parameters needed to be set to particular values ahead of time.
   
   
2. Lack of Documentation: The documentation for SPECROAD is frankly inadequate. It was an in-house tool for the SAO, so the publicly available documentation consists mostly of one webpage describing what the scripts do.
   
   
3. No Installation Instructions: There were several IRAF packages to install (available here). The shell scripts required not only the installation of several IRAF packages and the compilation of several C programs, they used korn shell (ksh) which I was unfamiliar with. And, as I would soon discover, the ksh installed on the Mac >had serious bugs in its list handling.
   
   
4. Undocumented Bugs: The available SPECROAD scripts had major bugs whose workarounds were undocumented and only available when I would email someone at the SAO to inquire about them. In one case, you had to know to actually break out of the script with Control-C, then run some of the reduction in IRAF, and then restart the script!
   
   
5. Assumed Availability of Spectral Templates: The SPECROAD scripts typically relied on pre-set spectral line databases for different grating/central wavelength configurations instead of having the user exploit the IRAF identify command to build a database. This was not a bug, but since we were using an 'atypical' configuration, it proved annoying.
   
   
6. Assumed Use of Low Resolution Spectra: The SPECROAD scripts were mostly pre-configured for working with 270gpm gratings and required some tweaking for 600gpm.
   
   
7. Inconvenient Backups As Data Reduction Progresses: If one of the scripts executed by SPECROAD crashed, you had to dig around subdirectories to "backtrack" the changes before restarting the script. With disk space not an issue, I prefer monolithic backups of the entire data directory at certain key points in the data reduction pipeline.
   

Reworking SPECROAD into E-SPECROAD

Given these problems with the SPECROAD scripts, in fall of 2007 I set about to port SPECROAD to the Mac OS X platform, which is where I do most of my data reduction. As it turned out, this required extensive minor changes to many of the SPECROAD scripts to address issues I discovered along the way. Given the extensive changes, I have called this version of SPECROAD, "External SPECROAD" or "E-SPECROAD" because its goal was to make it possible to run the automated SPECROAD scripts external to SAO. Although Jessica Mink at the SAO helped quite a bit, the SAO is not responsible for this creation and it is I who should be held accountable for its flaws or features.

1. Nonportability: With the goal of making E-SPECROAD scripts run on any computer (but my Mac in particular), I set about removing as much of the site-specific code as possible. There is some minimal editing that still needs to be done to make the scripts run on a particular computer, but it is much less extensive than before. I was able to resolve the problem of unknown "preset" IRAF parameters in June 2008 when Bill Wyatt at the SAO granted me access to their computers, which allowed me to examine their setup and configure E-SPECROAD to try to match it.
   
   I also want to acknowledge that E-SPECROAD would probably not have gotten anywhere without the efforts of Jessica Mink. He is an awesome programmer and the person responsible for some commonly used astronomical packages such as RVSAO and WCSTools. He was dedicated to the portability of the Hectospec package for IRAF and so he worked quite hard to port the portions of the package that relied on in-house, non-IRAF code to IRAF native code. Without those changes, running E-SPECROAD on a non-Solaris machine would still be extremely difficult.
   
   
2. Lack of Documentation: I have tried to provide a more extended User's Guide to reducing Hectospec data with External SPECROAD which tells you a little bit of what it does and what to expect it to do. This isn't quite "Hectospec for Dummies", but at least it gets you to "Hectospec for Graduate Students".
   
   
3. No Installation Instructions: I have documented the system requirements and installation instructions for E-SPECROAD here.
   
   
4. Undocumented Bugs: In E-SPECROAD I have tried to fix as many of the undocumented bugs as I found and dealt with the requirement of running the IRAF hcal process external to the script by having it pause and request the user run IRAF separately. This allows the user to process the data without requiring the user to interrupt the E-SPECROAD scripts.
   
   
5. Assumed Availability of Spectral Templates: The E-SPECROAD script starts by copying the HeNeAr spectral line list from the Hectospec IRAF scripts library and then allows you to build your own spectral line identifications database of the HeNeAr spectra for your particular set of fibers (through the identify IRAF procedure). This was possible with the original SPECROAD scripts, they just were not really configured to default to this. Another annoyance related this is that the HeNeAr lamps we used at the Hectospec leaked out much of their helium some time ago (as of Summer 2008 it does look like this might have been remedied with a new lamp), so I also allow download of a more recent spectral atlas for the lamp originally provided to me by Susan Tokarz of the Harvard Smithsonian Center for Astrophysics (CfA) here.
   
   
6. Assumed Use of Low Resolution Spectra: The E-SPECROAD scripts now automatically determine the spectral range you are working with. This allows the E-SPECROAD scripts to work with both 270gpm or 600gpm data without modifications.
   
   
7. Inconvienent Backups As Data Reduction Progresses: The E-SPECROAD scripts now prompt you to make backups of the data directory in its "current state" at two points. This allows you to recover fairly easily in the event of some crashes. The only major annoyance being you need to have a significant amount of free space available to make the backups, enough to clone the directory. These backups are optional, but highly recommended.
   

Funding: The creation of E-SPECROAD was supported by National Science Foundation Grant AST-0729989 ("RUI: Collaborative Research to Map the Asymmetric Thick Disk").

Papers Published Using External SPECROAD

1. "Kinematics and Metallicities in the Bootes III Stellar Overdensity: a Disrupted Dwarf Galaxy?"
   Carlin, J.L., Grillmair, C.J., Munoz, R.R., Nidever, D.L., and Majewski, S.R. 2009, ApJL, 702, L9-L13 (ADS abstract page) (ArXiv) (PDF)
2. "A Plethora of AGN Among Lyman Alpha Galaxies at Low Redshift"
   Finkelstein, S.L., Cohen, S.H., Malhotra, S., Rhoads, J.E., Papovich, C. 2009, ApJ, 703, L162 (ADS abstract page) (ArXiv) (PDF)
3. "Dust Extinction and Metallicities of Star-Forming Lyman Alpha Emitting Galaxies at Low Redshift"
   Finkelstein, S.L., Cohen, S.H., Moustakas, J., Malhotra, S., Rhoads, J.E., Papovich, C. 2011, ApJ, 733, 117. (ADS abstract page) (ArXiv) (PDF)
4. "Mapping the Asymmetric Thick Disk: III. The Kinematics and Interaction with the Galactic Bar"
   Humphreys, R.M., Beers, T.C., Cabanela, J.E., Grammer, S., Davidson K., Lee, Y.S., and Larsen, J.A..2011 AJ, 141, 131. (ADS abstract page) (ArXiv) (PDF)
5. "First spectroscopic measurements of [OIII] emission from Lyman-alpha selected field galaxies at z ~ 3.1"
   McLinden, E.M. , Finkelstein, S.L., Rhoads, J.E., Malhotra, S., Hibon, P., Richardson, M.L.A., Cresci,G., Quirrenbach, A., Pasquali, A., Bian, F., Fan,X., Woodward C.E. 2011,ApJ, 730, 136. (ADS abstract page) (ArXiv) (PDF)