DMT Maintenance

Upgrades/improvements in progress:

1. DMT hardware infrastructure upgrades
• GDS network upgrade at LLO
• Develop Dell switch settings
  
• Install switch
• fiber to decatur/bermuda - daisy chain to second cp?
  
• LHO server hardware upgrade?
• Replace stone/sand with V240s. Dave looking into pricing.
  
• bermuda
• receive broacast frames.
  
• marble
• matlab?
  
2. DMT infrastructure s/w upgrades/fixes
• EasyCalibrate: Calibration of DARM_ERR
  
3. New Monitors
• OpticalLever: Configuration test of new monitor.
• InspiralMon_BBH
• Processes added to run-list
• Needs trigger files in /cluster.
• LigoLwMon: New generic version of InspiralMon
• LineMonitor_<ifo>_mir: Monitor high frequency lines?
4. Monitor bug-fixes/upgrades/reconfigure
• IRIG-B: Fix occasional 262144s offset (-ve integer passed as unsigned?)
  
• PulsarMon: History storage inmemory
• LineMon: NaN problem
  

Plans for next maintenance period (Sep 4, 2007 15UTC)

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• webview:
  
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• -- none --
  

Results of previous maintenance periods

August 28, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• libtclient.so: fix Process ID format. Restart TrigMgr, all monitors generating triggers.
  
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• -- none --
  

August 21, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• All online nodes: rebooted with gds-2.11.2.
  
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• -- none --

August 14, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• obsidian: rebooted with gds-2.11.2.
  
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• PulsarMon:
  
• Fix mismatched deletes (Giovanni).
  
• Fix exponential averaging after lost/unlocked data (Keith).
  
• Fix OSC condition (Keith).
  

August 5, 2007

• DMT h/w reboots
  
• stone: Crashed with memory error. rebooted

June 26, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• webview: restart to release log files, leaked memory.
• New monitors
  
• PDnMon: Moved to production list.
  
• Updated/Reconfigured monitors
• PulsarMon: Updates from Giovanni to fix granularity. Add output directories.
• PlaneMon: Restart old version by hand on delaronde.
  

June 19, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• webview: restart to release log files, leaked memory.
• libdmtsigp: installed libdmtsigp.so.0.1.1 - minor fixes to memory management in FIRFilter and Slope
• NDS trend reconfiguration: [Keith, Dave] Add PDnMon, remove obsolete StochMon dirs.
  
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• -- none --
  

June 12, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• webview: restart to release log files, leaked memory.
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• SegGener: add COIL_OVERFLOW segments.
  

May 23, 2007

•  Stone failed/restared
  

May 22, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• webview: new version with (partially) fixed memory leak.
  
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• PlaneMon: Fixed
  

May 15, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• libhtml: Fix memory leaks (textAttr destructor).
  
• liblmsg: Pad with zeroes in TransOutput.
• webview: Recompile and restart.
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• DuoTone: New matlab libraries. Run under procmgr.
  
• PulsarMon: Fix minor errors.
• BurstMon: Generate standard trend file names.
  

May 8, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• webview: restart to release leaked memory.
  
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• -- none --
  

May 1, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• webview: restart to release leaked memory.
  
• New monitors
  
• PDnMon: Run in test mode on stone/delaronde.
  
• Updated/Reconfigured monitors
• -- none --
  

April 24, 2007

• DMT infrastructure h/w upgrades
• Restart LHO: restarted all nodes to fix log problems after disk full error
  
• DMT infrastructure s/w upgrades/fixes
• webview: restart to release leaked memory.
  
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• SegGener: Version 0. 

April 17, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• webview: restart to release leaked memory.
  
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• -- none --

April 10, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• webview: restart to release leaked memory.
  
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• SixtyHertzMon: Generate unique trend file names
• ZGlitch: Add TCS channels to configuration.

April 3, 2007

• DMT infrastructure h/w upgrades
• Trend directories: rename seismic and StochMon trend frames
• DMT infrastructure s/w upgrades/fixes
• webview: restart to release leaked memory.
  
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• -- none --
  

March 13, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• webview: restart to release leaked memory.
  
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• PulsarMon: New version from giovanni
  

March 6, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• Channel class: fix delete[] bug in decimation code.
  
• gds-2.11.1: Install new version on poplar and all LHO compute servers.
  
• New monitors
  
• SixtyHertzMon: Production version on all machines.
• Updated/Reconfigured monitors
• -- none --
  

February 27, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• Solaris DST patch: Reboot all online machines LLO+LHO.
• gabbro: rebuilt after patch failed
  
• gds-2.11.1: Install new version on all LLO sparc machines, gabbro (after rebuild)
  
• New monitors
  
• SixtyHertzMon: Started by hand on delaronde (LLO) production version at LHO.
• Updated/Reconfigured monitors
• SenseMonitor_H1: moved to gypsum while gabbro offline

February 13, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• gds-2.11.1: Running on stone, delaronde.
  
• New monitors
  
• -- none --
• Updated/Reconfigured monitors
• SixtyHertzMon: New  version run in test mode on  staone, delaronde.

January 30, 2007

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• -- none --
  
• New monitors
  
• -- none --
• Updated/Reconfigured monitors
• New_Seis_Blrms: Add to 30-100mHz band to all LHO seismometers.

January 30, 2007

• DMT infrastructure h/w upgrades
• clearwater: Replace LLO server with V240 (2×1.5GHz Sparc III)
  
• DMT infrastructure s/w upgrades/fixes
• -- none --
  
• New monitors
  
• -- none --
• Updated/Reconfigured monitors
• -- none --

January 9, 2007

• DMT infrastructure h/w upgrades
• ldas switch: Ldas temporarily unavailable while switch was moved
• DMT infrastructure s/w upgrades/fixes
• webviewer: Restart to release leaked memory (LLO)
  
• New monitors
  
• -- none --
• Updated/Reconfigured monitors
• -- none --
  

December 19, 2006

• DMT infrastructure h/w upgrades
• GC Network: Test configuration with stone behind NAT router (failed).
• Home disk (sand): Run fsck to verify fs integrity - global reboot.
  
• DMT infrastructure s/w upgrades/fixes
• webviewer: Restart to release leaked memory (LLO & LHO)
  
• New monitors
  
• dvTest: Add to flint production list.
• Updated/Reconfigured monitors
• -- none --
  

December 12, 2006

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• webviewer: Restart to release leaked memory (LLO & LHO)
  
• New monitors
  
• -- none --
• Updated/Reconfigured monitors
• -- none --
  

December 5, 2006

• DMT infrastructure h/w upgrades
• -- none --
  
• DMT infrastructure s/w upgrades/fixes
• webviewer: Restart to release leaked memory (LLO & LHO)
  
• New monitors
  
• dvTest: Check datavalid on L1-RDS channels.
• SixtyHertzMon: Run test versions on stone & delaronde.
• NoiseFloorMon: switch new code to production.
  
• Updated/Reconfigured monitors
• -- none --
  

November 28, 2006

• DMT infrastructure h/w upgrades
• -- none --
• DMT infrastructure s/w upgrades
• Spi: Run SPI.pl on clearwater, poplar. Sleep 10s before preparing LLO page to avoid race condition.
• webviewer: Restart to release leaked memory (LLO & LHO)
• Updated/Reconfigured monitors
• -- none --

November 14, 2006

• DMT infrastructure h/w upgrades
• -- none --
• DMT infrastructure s/w upgrades
• webviewer: Restart to release leaked memory (LLO & LHO)
  
• Updated/Reconfigured monitors
• -- none --

October 24, 2006

• DMT infrastructure h/w upgrades
• Reconfigure GDS network interface on pickerel
• DMT infrastructure s/w upgrades
• webviewer: Restart to release leaked memory (LLO & LHO)
  
• Updated/Reconfigured monitors
• TimeMon: Fix negative sqrt error, restore LHO configuration.
• SegGener: Install new LSC_OVERFLOW and LSC_SEVERE_OVERFLOW segments
  

October 5, 2006

• DMT infrastructure h/w maintenance/upgrades
• Hot-swap user home backup disk on sand
  

October 3, 2006

• DMT infrastructure h/w upgrades
• Reconfigure GDS network interface on pickerel
• DMT infrastructure s/w upgrades
• webviewer: Fixed minor memory leaks (more to come?)
  
• Updated/Reconfigured monitors
• CheckDataValid: Add veto list (remove channels always seen at LLO)

September 26, 2006

• DMT infrastructure s/w upgrades
• -- none --
  
• Updated/Reconfigured monitors
• PulsarMon: New version run as production on LLO, LHO.
• RmsBands: 5-7 Hz band AS_Q and DARM_ERR trigger thresholds raised
• StrainBandsMon: Notch lines as in DataQual
  

September 19, 2006

• DMT infrastructure s/w upgrades
• webviewer Restarted at LLO to keep ahead of memory leak
  
• Updated/Reconfigured monitors
• PulsarMon: launched on delaronde in test mode.

September 12, 2006

• DMT infrastructure s/w upgrades
• webview: Restarted at LLO to keep ahead of memory leak.
• sshd: SSH damon added for LHO GC network to disable shared accounts.
  
• Updated/Reconfigured monitors
• -- none --

September 5, 2006

• DMT infrastructure s/w upgrades
• webview: Restarted at LLO to keep ahead of memory leak
• sshd: SSH damon added for LLO GC network to disable shared accounts.
• Updated/Reconfigured monitors
• PlaneMon: New version running as production at LLO and LHO.

August 29, 2006

• DMT infrastructure s/w upgrades
• SPI.pl: New version installed, running under cron on stone.
  
• Updated/Reconfigured monitors
• PhotonCal: launched on stone (H1) and delaronde (L1) in test mode.
• RmsBands_LHO: 5-7 Hz bands in AS_Q and DARM_ERR  added.
  

August 22, 2006

• Updated/Reconfigured monitors
• Check_online: Moved to production lineup (LLO, LHO).
• Check_archive: Moved to production lineup (LLO, LHO).

August 15, 2006

• DMT infrastructure s/w upgrades
• CheckDataValid: New script to look for dataValid!=0 errors.
  
• Updated/Reconfigured monitors
• Check_online: Check for dataValid errors in online frames (test mode LLO, LHO).
• Check_archive: Check for dataValid errors in archive frames (test mode LLO, LHO).
• InspiralMon: Restart after starting script to remove old links.
  

July 11, 2006

• DMT infrastructure h/w upgrades
• Replace delaronde with V240
  
• Updated/Reconfigured monitors
• Disable PSD file writing in GainMon, PulsarMon.
  

July 4, 2006

• No maintenance - Independence day
  

June 27, 2006

• Updated/Reconfigured monitors
• SegGener: Add ASI_CORR_OVERFLOW flags

June 20, 2006

• DMT infrastructure h/w upgrades
• delaronde replacement V240 mounted, cabled for test
  
• Updated/Reconfigured monitors
• LockLoss: Fix alarm URL (restarted at LLO,  and LHO).
• StrainBandsMon: New version with notches, better html reports (restarted at LLO,  and LHO).
  

May 30, 2006

• DMT infrastructure s/w upgrades/fixes
• -- none --
• New monitors
  
• -- none --
  
• Updated/Reconfigured monitors
• SenseMonitor: Restart after filled disk on london.
  

May 23, 2006

• DMT infrastructure s/w upgrades/fixes
• -- none --
• New monitors
  
• StrainBandsMon: Added to production monitor list.
  
• Updated/Reconfigured monitors
• -- none --

May 16, 2006

• DMT infrastructure s/w upgrades/fixes
• -- none --
  
• Update/Restart modified monitors
  
• PSLmon: Add minimum size requirement to the Glitch tool
• Reconfigure monitors
• ZGlitch: Add size>1e-4 requirement for Xn:LSC-DARM_CTRL_EXC_DAQ glitch tools
  

May 9, 2006

• DMT infrastructure s/w upgrades/fixes
• -- none --
  
• Update/Restart modified monitors
  
• -- none --
  

May 2, 2006

• DMT infrastructure s/w upgrades/fixes
• FSeries: Bugs in += and -= fixed (installed libgdscntr.so.0.0.2)
  
• Update/Restart modified monitors
  
• Slice2: Added to production monitor list
  
• Reboot all LHO nodes after power outage.
  

Apr 25, 2006

• DMT infrastructure s/w upgrades/fixes
• WatchDawg: Check that device is mounted in check-space
  
• findProc: Look for Process_List_* in $DMTPROCDIR if specified
  
• Update/Restart modified monitors
  
• Slice2: Keep running r/o error total

Apr 18, 2006

• Install S5-V2 calibration for H2 (ReferenceCalibration_H2.xml)
  
• Restart all monitors that use the H2 Calibration
• burstMon_H2
• burstMon_H2_256Hz
• burstMon_H2_2kHz
• GainMon_H2
• PulsarMon_H2
• SenseMonitor_H2
• StochMon_1L_H2H1
• StochMon_2L_H1H2
  

• Install new StochMon configuration and reference PSD files. Restart all StochMon instances:
• StochMon_1L_H1H2
• StochMon_1L_H1L1
• StochMon_1L_L1H1 (LLO)
  

Apr 11, 2006

• Restart all monitors after power outage (LLO)

Apr 10, 2006

• Restart all monitors after disk failure (LLO)
• New LockLoss conditions ("Common Mode" excludes AS_Triggered state) included used by all monitors

Apr 9, 2006

• Restart all monitors after power failure (LHO)
• New LockLoss conditions ("Common Mode" excludes AS_Triggered state) included used by all monitors
  

Apr 4, 2006

• Reconfigure monitors
• LockLoss: AS_Trigger state reclassified to state 2
  

Mar 28, 2006

• -- No Changes --
  

Mar 21, 2006

• Update/Restart modified monitors
  
• BicoMon: bugs fixed
• PlaneMon: general upgrades
  
• Reconfigure monitors
• ZGlitch_LHO: Remove discontinued H0:PEM_*_V2 channels.
• WaveMon_L1:  Remove verbose printout.
  

Mar 16, 2006

• DMT hardware infrastructure upgrades
• marble
• Install DMT software
  
• Install/configure in GDSnet
  

Mar 14, 2006

• Restart monitors using H1 calibration:
• StochMon_xxx, GainMon_H2 and SenseMonitor_H2  (who restarted these?)
  

Mar 7, 2006

• Restart monitors using H1 calibration:
• PulsarMon_H1 (LHO).
• BurstMon_H1 instances (LHO).
• Mods to DMT shared objects (LHO & LLO)
  
• libgdscntr.so
• Fix CWVec bug (SEGV when cloneing uninitialized vector)
• libxsil.so
• Fix bug in MetaIO class - files not closed.
• Link to new libmetaio.so
  
• Update/Restart modified monitors (LHO & LLO)
  
• LockLoss: remove averaging of trended values
• InspiralMon: MetaIO fixes
  
• Restart reconfigured monitors (LHO)
  
• SegGener:  H<i>:H<j>NOTLOCKED with strict lock condition
• Test version of PlaneMon on stone (LHO)

Mar 1, 2006 (H1 calibration update)

• Install S5V2 calibration for H1
• Update ReferenceCalibration_H1 in cvs, install it in ~ops/pars
• Restart SenseMon_H1
  

Feb 28, 2006

• Installed BicoViewer (sparc only)
  
• No action taken at LLO - Mardi Gras
  

Feb 21, 2006

• Restart SegGener with new configuration (LHO and LLO)
• Installed fix to BicoMon (not run under process manager)
• Installed MatLab libraries.