Q Scan

Introduction

QScan is a tool to perform an detailed study of the LIGO data stream around a specific time of interest (such as a candidate gravitational-wave event, a detector glitch, or a hardware injection). In addition to displaying the time series and time-frequency spectrograms of gravitational-wave channel data, QScan can also efficiently search a large number of auxiliary and environmental channels for statistically significant signal content.

The following images are taken from an example scan of an inspiral hardware injection and are typical of time series, time-frequency spectrograms, and time-frequency eventgrams produced by QScan.

Quick Start

The quickest way to get started is to use one of the existing QScan installations at one of the LIGO Laboratory computing clusters. For example, the following commands will perform an example QScan on the CIT computing cluster (assuming you already have access).

  grid-proxy-init
  ssh ldas-pcdev1.ligo.caltech.edu
  ~qonline/q/bin/qscan.sh 816335770.0 &

You can monitor the scan status by watching the log file.

  tail --pid=$! -F ~/public_html/qscans/816335770.0/log.txt

The full form of qscan.sh is

  qscan.sh gpstime <configurationfile> <framecachefile> <outputdirectory>.

A number of predefined configuration and frame cache files are distributed with the QScan utility. These predefined files are specified by the syntax @name, where name is the name of the desired configuration or framecache file. Predefined configuration and frame cache files include @default, @online, @gw, @full, and the various interferometer combinations @H1, @H2, @L1, @H1H2, @H1L1, @H2L1, and @H1H2L1. Alternatively, the user can specify the path to a custom configuration or frame cache file. Other predefined configuration and framecache files are available and can be found by looking in the configurations/ and framecaches/ sub-directories of the qscan/ installation directory. If the specified frame cache file is a directory instead of a file, QScan generates a temporary frame cache file for all frame files under the specified directory.

By default, QScan will use the predefined @default configuration and frame cache files based on the S5 RDS_R_L1 data available at each site, and write its output to the the directory ~/public_html/qscans. On the LHO and LLO computing clusters, QScan defaults to the @online configuration and frame cache files if the requested time is within the past 72 hours.

The scan should take ~25 minutes and the result will be available on-line at the following URL.

  http://ldas-jobs.ligo.caltech.edu/~username/qscans/816335770.0/

Note that since QScan only plots images for channels with statistically significant content, the run time of a scan depends somewhat on the nature of the event. Each statistically significant channel requires the production of 27 images. As a result, if many channels have statistically significant content, the analysis time may increase noticeably. There may be additional latencies if the data needs to be recovered from the tape archive.

It is also simple to launch QScans on LIGO Laboratory computing clusters using the Condor scheduling system. In this case, you need to submit the jobs from the ldas-grid machine on the desired cluster. For example, the following commands will submit our example QScan on the LHO computing cluster (assuming you already have access).

  grid-proxy-init
  ssh ldas-grid.ligo-wa.caltech.edu
  condor_run ~qonline/q/bin/qscan.sh 816335770.0 &

Finally, since data quality information changes over time, the qupdate.sh utility is provided to update this information without rerunning the entire scan. For example, the following commands will update the example QScan created above.

  grid-proxy-init
  ssh ldas-pcdev1.ligo.caltech.edu
  ~qonline/q/bin/qupdate.sh ~/public_html/qscans/816335770.0 &

This will update the scan based on the most recent online summary of segment and data quality information. These summaries are performed once per day at 18:00 UTC and reflect information through 16:00 UTC.

Installations

QScan and QPipeline are installed on most LSC computing clusters and LIGO Laboratory general computing machines. The installation location varies some between sites, and is summarized in the following tables.

Please do not run QScan directly on the ldas-grid machines. These machines run the condor scheduler and should not be subjected to heavy computational load. For now, either run QScan on the ldas-pcdev1 machines or submit jobs to the cluster via the condor scheduler as described above.

While QScan is installed on the PSU and UWM computing clusters, these clusters do not currently provide a web server to view the resulting web pages. Also, up to date predefined of automatic frame cache files are only supported on the CIT, LHO, LLO, and EGO computing clusters.

Configuration

The QSCan configuration file is an ASCII text file that specifies each channel to scan as well as the desired scan parameters. What follows is a typical entry for the H1 gravitational-wave channel.

  {
    channelName:                 H1:LSC-DARM_ERR
    frameType:                   RDS_R_L4
    sampleFrequency:             4096
    searchTimeRange:             32
    searchFrequencyRange:        [32 Inf]
    searchQRange:                [4 64]
    searchMaximumEnergyLoss:     0.2
    whiteNoiseFalseRate:         1e-3
    searchWindowDuration:        0.25
    plotTimeRanges:              [1 4 16]
    plotFrequencyRange:          []
    plotMaximumEnergyLoss:       0.2
    plotNormalizedEnergyRange:   [0 25.5]
    alwaysPlotFlag:              1
  }

The qconfigure.sh utility is provided to automatically generate typical QScan configuration files from the channels avaiable in a sample frame file. For example, the following command creates a configuration file for RDS_R_L1 data from LHO.

  ~qonline/q/bin/qconfigure.sh configurations/H0H1H2_RDS_R_L1.txt /archive/frames/S5/L1/LHO/H-RDS_R_L1-8163/H-RDS_R_L1-816335680-64.gwf

This file can then be edited to remove unwanted channels or to modify the search parameters. Configurations generated from different frame files can also be merged into a single QScan configuration file.

Configuring for the full raw data stream is discouraged due to the very large number of channels and higher sample frequencies.

A frame cache file, which caches the location of frame file data, can be generated using the createframecache.pl utility. This utility is now distributed with LIGOTools, and is also copied into the QScan installation directory. For example, the following command caches the location of all available S5 RDS_R_L1 LHO data at CIT.

  ~qonline/q/bin/createframecache.pl framecaches/H0H1H2_RDS_R_L1.txt /archive/frames/S5/L1/LHO

It is also possible to convert a LAL formatted frame cache file to the form used by Q Scan using the convertlalcache.pl utility.

  ~qonline/q/bin/convertlalcache.pl lalframecache qscanframecache

Interpretation

Care must be taken when interpreting QScan results. The resulting images can be both informative and misleading. For this reason, a variety of images types are produced:

• time series
• raw
• high pass filtered
• whitened
• time-frequency spectrograms
• raw
• whitened
• autoscaled
• time-frequency eventgrams
• raw
• whitened
• autoscaled

Each of these image types is also produced for multiple time scales as specified in the configuration file.

Whitening filters are useful for enhancing weak signals in the presence of colored noise. At the same time, the whitening filters also introduce artifacts into the resulting images. Although these artifacts are typically weak compared with the central signal, they are particularly noticable for very high SNR events, where they stand out above the background noise. To avoid misintrepretation, time series, spectrograms, and eventgrams are shown for both the raw and whitened data streams.

In order to observe the full time-frequency structure of events, it is helpful to fix the colormap scale to saturate at a normalized energy of around 25.5 (corresponding to a matched filter SNR of 7 for minimum uncertainty signals). This permits the study of significant structure that is nonetheless weak compared to the central signal. At the same time, it is also useful to autoscale the colormap to the most significant content in the spectrogram.

Finally, QScan provides both time-frequency spectrograms and time-frequency eventgrams. While the spectrograms are displayed for the single Q that yielded the most significant minimum uncertainty tile, eventgrams display significant tiles from all Q planes simultaneously. Again, both image types are useful when interpreting events.

Code

The source code for QScan is provided along with the QPipeline as part of the Q transform code base. The most recent version is available from the LSC MatApps CVS repository along with a nightly build of documentation.

The following commands will check out the most recent version of the Q transform code base from CVS and into a subdirectory named q in the current working directory.

cvs -d ":pserver:anonymous@gravity.phys.uwm.edu:2402/usr/local/cvs/lscsoft" checkout q

For more information, or to obtain a CVS account, see the document "How to use the LSCSoft CVS archive".

Requirements

QScan has been built and tested on 32 bit Linux Fedora Core 3, 32 bit and 64 bit Fedora Core 4, and Solaris 9.

In addition to LIGOTools and MATLAB version R13, QScan also requires the ImageMagick suite of image conversion utilities in order to produce thumbnail images. Future version of QScan will also require the LSCsegFind utility to query segment and data quality information.

Building QScan requires the optional Signal Processing and Statistics toolboxes for Matlab, as well as the Matlab R13 compiler.

While the build and luanch scripts are provided for MATLAB version R13, it should also be possible to produce standalone executables using version R14. While this has not been attempted, the resulting executable will probably run somewhat slower.

Building and Testing

Building a stand-along QScan executable requires that both LIGOTools and MATLAB version R13 be installed and properly configured. This includes setting the $LIGOTOOLS and $MATLAB environment variables to point to the respective top level installation directories. It is then possible to build QScan by performing the following commands.

  cd ~
  cvs -d ":pserver:anonymous@gravity.phys.uwm.edu:2402/usr/local/cvs/lscsoft" checkout -P q
  cd q/src
  ./build.sh

It may be necessary to edit the qsetup.sh shell script to point to the location of MATLAB R13, LIGOTools, and the native C compiler.

The resulting executable and supporting scripts will be installed in the ~/q/bin directory.

Once the executable is built, the following command will perform a simple test scans and analyses of sample data bundled with the source code.

  cd ~/q
  ./test.sh

It may be necessary to edit the qsetup.sh shell script to point to the location of MATLAB R13 runtime libraries, ImageMagick convert utility, LIGOTools tconvert utility, and the LSCsegFind utility.

The results of the test scan will be available in the directory ~/q/test/qscan/822191381.5/ and should look like this

The distribution can be moved to other locations and will still function normally.

Future Improvements

QScan is currently under active development and a number of improvements are pending.

• Update configuration files to reflect current channel lists, the most recent h(t) version number, and suggestions from others.
• Provide option to locate data with LSCdataFind.