S5 Data Quality Review

Data Quality Activities

Data Quality Monitor (DataQual)

A data quality monitor based on the PSLmon program will run during the entire S5 run. This monitor made trends of band limited RMS and glitch rates on several channels. The DataQual configuration files can be viewed by following these links for the LLO configuration and for the LHO configuration. The current status of the monitors at LHO and LLO  are also available online.
 

3-day Summary trends

Trends of band limited RMS and Glitch rates are available as pdf files for the last 3 days.
 

AS_Q bands
 H1 H2 L1
MC_F bands
 H1 H2 L1
POB_Q bands
 H1 H2 L1
REFL_I bands
 H1 H2 L1
Glitch Rates H1 H2 L1

Segment Data Base

An online database is stores segment times that define certain data quality conditions. Online documentation is available here. The data base may be queried with the LSCsegFind command included in the lscsoft glue package. Segments in the database have several sources:

Online Segment Generation

The DMT monitor SegGener runs online and continuously defines several segment types. These segments are inserted into the segment database within about 5 minutes of their generation. The current status of the SegGener processes can be seen at LHO and LLO. At present the segments defined on line by SegGener are:

Condition
 IFOs
 Description
H1_Not_Locked
 H2
 Unlocked H1 can interfere with H2 data by optical coupling.
H2_Not_Locked H1 Unlocked H2 can interfere with H1 data by optical coupling.
PD_Overflow
 H1, H2, L1
 LSC photodide saturations.
Wind_Over_30MPH
 H1, H2
 High winds (>35MPH at end stations or >30MPH at LVEA) measured by the Hanford wether stations.
ASI_CORR_OVERFLOW
 H1, H2, L1
 Saturation in the ASnI_CORR signals

Also planned for online generation are:

2007.12.03 - Wind_Over_30MPH error segments removed

The LHO weather stations had several glitches (readout errors?) resulting in unphysical wind speed measurements, generally greater than 400MPH. I have scanned the minute trends to find segments where these erroneous measurements were made (listed here for H1 and H2) and then removed these segments from the online segments. The new segment definitions are here for H1 and H2.

Calibration Line Dropouts

Calibration lines are injected throughout the S5 run to monitor the IFO gain. To verify the stability of the calibration lines, the injected signals are monitored by the ZGlitch DMT monitor. ZGiltch runs a 6-sigma glitch finder on the excitation channels, Xn:LSC-DARM_CTRL_EXC_DAQ, after notching out the three calibration lines. Once the notch filters have settled, the resulting signal should be just the difference between the measured signal and a perfect sine wave. If the only error is numeric round-off from the single precision float used to store the data, the signal should be A/224 ~10-8.

Dropout rates and classification

I have seen four classes of calibration line errors:
Type
 Duration
 t Offset
 Significance
Single sample drop-outs
 62 µs
 875ms
 ~255
Phase shift
 62ms
 N×62.5ms >20
1 second drop-outs
 1s 875ms
 >20
Awg stuck
 indefinite (>4s)
 N×62.5ms
 >6

Glitch segments

The segment times are listed here for H1, H2 and L1 updated 2007.10.15 for glitches through the end of S5.  The ZGlitch configuration did not include calibration line monitoring for  L1 segments before GPS-817600000 so these are listed separately (updated 2007.12.01). Each segment in these list files contains a Start time, end time, ns offset of found glitch, amplitude and significance of the found glitches and a type code. The type codes are:

Scripts

The segments listed above were produced using scripts and unpublished utilities that are contained in directories here and in a tar file here. There is also some installation documentation here.

OSEM Sensor Glitches

The OSEM shadow sensors make a direct measurement of a mirror's motion relative to its suspension frame and are used for local damping of several optics. The Sensor signals have been seen to glitch and feed indirectly (via the local damping?) into the GW channel (especially on H2). This problem was probably due to the lighter filtering of the signals in H2 than was used on H1 and L1. The H2 filtering was replaced by more agressive (L1-like) filters on Feb 9 (see here) in an attempt to fix this problem.

An example glitch in H2:SUS-ITMX_SENSOR_UL is shown here:



Shourov has also prepared a Q-Scan showing the effect on the IFO of an example OSEM glitch event.

I have tabulated glitches by looking at the sigmas of the sensor channel second trends and looked for correlation with the ZGlitch Xn:LSC-AS_Q triggers. The segments produced in this way seem to have two components:
  1. Readout errors (as above)
  2. Noisy periods (presumably driven by wind/seismic activity)
Only a few of the sensor channels seem to correlate to AS_Q glitches. Presumably the large readout glitches will be fairly likely to produce a glitch in AS_Q, whereas any glitches in the general noisy periods will be much less correlated. To eliminate vetos cause by the general noise I required the segments to be exactly 1 second long, i.e. that the excess noise is isolated.  The following table gives statistics for various  H2 sensor channels, sigma thresholds and cuts in cluster width for data from Nov 4, 2005 to Jan 31, 2006.

Channel
 Cluster
width
 Sigma
threhold
 Segments
 Glitches
vetoed
H2:SUS-ITMX_SENSOR_LL
 >0
 20
 153
 10

1
 20
 50
 5

1
 25
 13
 3

1
 30
 4
 2
H2:SUS-ITMX_SENSOR_UL
 >0
 20
 124
 79

1
 20
 97
 77
H2:SUS-ITMY_SENSOR_LL >0
 20
 215
 33

1
 20
 92
 24

1
 25
 28
 18
H2:SUS-ITMY_SENSOR_UL
 1
 20
 80
 18

1
 25
 25
 15

1
 30
 18
 15

Segment lists are linked to from the channel names. The "Glitches vetoed" column reports the number of ZGlitch AS_Q glitches found in the segments defined by the sensor sigma. Links from the numbers in these rows give the segment list interspersed with with correlated trigger information. The highlighted rows give the selection criteria chosen for the segment lists. I looked for correlations between AS_Q glitches and  glitches/noisy sections in all other H1 and H2 sensors but I didn't find any other significant correlations.

Optical Lever Glitches

The optical levers are also used for local damping. There is a known failure mode of the optical lever Lasers where they start to glitch (they stop lasing for an instant?) as they age. One such Laser (H2-??) was very glitchy and was replaced in Feb. 2006. Several channels have been set up at LHO to monitor the Laser output by summing the output of the 4 quadrants of each of the optical lever photodiodes. Malik Rakhmanov has started to set up a monitor for these Lasers.

Power Line Glitches

The ZGlitch program continually watches the power line monitor channels for glitches. This is done by notching out the fundamental power line frequency (60Hz) and watching for a changes in the remaining digitization noise. Any large (>5 sigma) glitch is recorded as a GDS_TRIGGER in the LDAS database. I have summarized all S5 triggers up to June 15, 2006  in the following files during science mode running in H1, H2 and L1.


Transition to Triggered (ASPD5) mode

All interferometers enter a robust backup running mode when one or more photodiodes saturate. In this mode, the AS port signals are provided by a single photodiode (ASPD5) rather than an average over four photodiodes ASPD1-4. This allows the IFOs to remin in lock with much reduced sensitivity until whatever sitation (usually seismic activity) subsides enough to switch back to low noise mode for science data acquisition. Since the triggered mode running is incompatible with science data taking, interlock should be in place to prevent science mode running while in triggered mode. There have been a couple issues found with this interlock:
  1. When the triggered mode transition takes place, the interferometer is supposed to be knocked out of science mode by the conlog mechanism (Xn:LSC-AS_TRIGL is set to 1). During the first few months of S5 (before GPS 82xxxxxxx?), this mechanism wasn't working at LLO and the triggered mode transition did not  knock L1 out of science mode.
  2. It is possible to return an IFO to science mode even if it is running in triggered mode. (This should be interlocked against.) Although this goes against standard operation procedures there have been a few slip-ups where the triggered mode was reset after restarting.
Gaby et al [ref?] have generated a list of segments that didn't have AS_TRIGL set and this has been added to the DQ segment list (segment type?). I have since scanned the second trends for segments of type(2) where AS_TRIGL was set during science mode. The lists of segments are now (2008.01.14) complete for all of S5 GPS 815000000-876000000 for H1, H2 and L1.

Environmental Channels

Studies of several environmental channels are underway.

LSC and ASC overflows

Overflow of the channels in the LSC and ASC loops are monitored by the from end processes and produce counts in slow channels called e.g. LSC-AS1Q_OVERFLOW. These channels are all summed together into a single master overflow channel: LSC-MASTER_OVERFLOW through GPS. Segments are produced online from these channels as summarized below. Note that the segment names link to a text file containing the S5 segments 851900000.
Segment Name
 Channels
 Meaning
H1:SEVERE_LSC_OVERFLOW
 H1:LSC-AS1_I, H1:LSC-AS1_Q, H1:LSC-AS2_I, H1:LSC-AS2_Q, H1:LSC-AS3_I, H1:LSC-AS3_Q, H1:LSC-AS4_I, H1:LSC-AS4_Q, H1:LSC-POB2_I, H1:LSC-POB2_Q, H1:LSC-REFL1_I, H1:LSC-REFL1_Q H1 in-loop overflows
H2:SEVERE_LSC_OVERFLOW H2:LSC-AS1_I, H2:LSC-AS1_Q, H2:LSC-AS2_I, H2:LSC-AS2_Q, H2:LSC-AS3_I, H2:LSC-AS3_Q, H2:LSC-AS4_I, H2:LSC-AS4_Q, H2:LSC-POB1_I, H2:LSC-POB1_Q, H2:LSC-REFL1_I, H2:LSC-REFL1_Q H2 in-loop overflows
L1:SEVERE_LSC_OVERFLOW L1:LSC-AS1_I, L1:LSC-AS1_Q, L1:LSC-AS2_I, L1:LSC-AS2_Q, L1:LSC-AS3_I, L1:LSC-AS3_Q, L1:LSC-AS4_I, L1:LSC-AS4_Q, L1:LSC-POB1_I, L1:LSC-POB1_Q, L1:LSC-REFL1_I, L1:LSC-REFL1_Q L1 in-loop overflows
H1:LSC_OVERFLOW H1:LSC-SPOB H1 out-of-loop overflows
H2:LSC_OVERFLOW H2:LSC-SPOB H2 out-of-loop overflows
L1:LSC_OVERFLOW L1:LSC-SPOB, L1:LSC-NSPOB, L1:LSC-NPTR L1 out-of-loop overflows
H1:PD_Overflow
 H1:LSC-MASTER_OVERFLOW
 All H1 overflows
H2:PD_Overflow H2:LSC-MASTER_OVERFLOW All H2 overflows
L1:PD_Overflow L1:LSC-MASTER_OVERFLOW All L1 overflows


ASnI_CORR saturations

The ASnI_CORR signals were found to scre up when the go above 20000 adc counts. To prevent this, Rana put a software limit on these signals at LLO. Although similar non-linearities are expected to be present in the LHO interferometers, not software limit was placed on the LHO signals. When the ASnI_CORR signal hits the software limit, it makes a kink in the signal which leaks into the AS_Q (mechanism?). I have therefore added a new DQ flag (ASI_CORR_OVERFLOW) for each interferometer. As described above, L1 may be the only instrument to have AS_Q glitches cause by this saturation, but the LHO instruments' sensitivity and noise properties may be affected when these flags are set.

For times before the ASI_CORR_OVERFLOW flag was calculated in SegGener, I have tablulated the  H1,  H2 and L1 saturations using the min and max values in the Xn:ASnI_CORR_OUT_DAQ second trends. These segments were clustered to join segments that were at most 10 seconds apart.

Data Acquisition (dataValid) Errors

Errors occasionally occur when some part of the data aquisition (DAQ) system is unable to keep up with the data stream. These are generally detected by the DAQ system and flagged by setting the FrAdcData:dataValid field of the affected channel to a non-zero value. Unfortunately, this field is generally ignored by the analysis software although they are often noticed by the Glitch investgations (e.g. L1 event at 842053582: event, Qscan).

Scan of L3 RDS frames

I have scanned the channels in the L3 RDS frames for such errors from the start of S5 through GPS 847400000 and created segments. The use of the L3 RDS frames makes the procedure very quick, but it limits the scan sensitivity to the DARM_ERR channels. The segment data are tabulated for H1, H2 and L1. These segments have a start-time, a stop-time, the dataValid code and the channel name for each segment. The procedure used to generate these segment tables is documented here.

Scan of L1 RDS frames

Because the L1 RDS frames have most of the channels and dataValid auxillary vectors, they can be used for a much more  complete scan for DAQ errors. I have run the dvTest monitor on all S5 RDS_L1 frames on the CIT cluster. This required about 1 hour for the LLO frames and about 2.8 hours for the LHO frames (all S5 data). The results of these runs were ~340 trigger files for each site. I have written out flat ascii files containing the following fields for each trigger:
  1. Channel Name
  2. Start time (GPS seconds)
  3. Start time offset (nanoseconds)
  4. Duration (seconds)
There are ~397k triggers (one per  line) in the LHO file and ~344k triggers in the LLO file. These data need to be further reduced for segment lists.

Seismic Up-Conversion

Some evidence indicates that the best measure of potential for upconversion is made by looking at out-of-band (5-7 Hz) activity in the DARM_ERR and AS_Q channels. Sudden increases of RMS in the 5-7Hz band are being recorded as triggers by the RmsBands monitor (LHO reconfigured Aug 29, 2006, first H1 trigger: 840919900, first H2 trigger: 840914795). I have since rerun the triggers on times from the start on S5 (~GPS 815000000) through GPS 839000000. These triggers are available in 1Ms GPS chunks here.

BSC Stack resonances

One specific problem for H1 has been the BSC stack resonances near 6Hz coupling to side motion of the optic. The side OSEM sensor is fed back to the side coil which make things worse. (Sam has volunteered to tune the side servo filters to improve this). I have calculated the band-limited rms in the bans from 5.8-6.2Hz for the side sensor channels on all large optics + BS and looked for correlations with high amplitude glitches found by kleineWelle. The channel that has the biggest correlation with DARM_ERR glitches is H1:SUS-ETMY_SENSOR_SIDE channel. This can be expected since the H1 y-arm is affected most by highway traffic (especially gravel trucks). The following figures show the maximum kleineWelle DARM_ERR glitch amplitude(?) in a one minute stride versus the largest 5-second RMS squared for the 5.8-6.2Hz band during the same minute for various optics.


 H1
 H2
 L1
BS


ETMX

ETMY


ITMX


ITMY


RM





The axes are the decimal logarithms of the plotted values. There is a strong correlation between high amplitude kw triggers and H1-ETMY RMS² values greater than ~50 (1.7 on the plot). The segments where the maximum value is greater than 50 are listed here for h1-etmy and h1-etmx. Note that each plot above links to a list of minutes with RMS² > 50. The columns contain the trend statistics (i.e. start GPS, avg, rms, min, max, N) for the band limited RMS² in the 6Hz band in the specified minute. Points with kw significance < ymin are plotted at (x, ymin) and points with RMS² > 1000 are plotted at (x, 3).

2007-Dec-12:
I have rerun PSLmon on the 5.8-6.2 Hz bands of all side coil sensors. The RMSs were measured by taking a DFT and summin the squares of tha approptiate bins. DFTs were performed on 5-second intervals, windowed with a Hann windowing and 50% overlap. Trigger with RMS > 50 were used to generate intervals and overlapping intervals were merged. The resulting intervals are here for H1:ETMX and H1:ETMY. The total time in these segments are 1472s for ETMX and 24882s for ETMY, which is about seven time less than the dead-tmes for the segments derived from the trends (168365 for ETMY). Instead of looking at the trends, I generated triggers for these bans. Each

Last modified: December 3, 2007 11:30 PST by  John Zweizig