Weekly Report for Week Ending August 17, 2006

Past Weekly Reports

There will be no LIGO Executive Committee scheduled Monday, August 21, 2006 due to the LIGO Staffing Committee meeting scheduled that morning.  The next Executive Committee Meeting will be scheduled Monday, August 28, 2006.

Special Announcements

Weekly Report Highlights

This is a particularly brief status report due to the involvement of a large number of LIGO personnel in the LSC Meeting held this week in Louisiana.

LSC Issues (Saulson)

This week we held a very successful LSC meeting at the Lod Cook Conference Center on the LSU campus. Attendance was around 170. In addition to the many formal talks and informal conversations, several actions were taken. They include:

  • The LSC Executive Committee approved the S4 Stochastic all-sky paper.
  • The LSC Council approved, by large majorities, the applications for membership from new groups at Andrews University, Rochester Institute of Technology, and San Jose State University.
  • The LSC Council enthusiastically endorsed the plans for Enhanced LIGO.
  • The LSC Council endorsed, by a large majority, the idea of close collaboration with Virgo along the basic lines expressed in the current draft MOU. Many good suggestions were made for ways to improve the MOU's language.

At this meeting, we declared the official transition from the LSC's old structures to the new committees called for in the Bylaws. In other words, we have now completed the LSC's restructuring.

LIGO Laboratory Administration (Lindquist)

STATUS OF LSC MOUs (Lloyd)

  • No report.

SITE TELECONFERENCE (Lindquist)

  • There was no site teleconference scheduled Thursday, August 17, 2006.
  • There are currently no open action items.

PROPERTY MANAGEMENT (Luna)

>From: Rod Luna <rluna@ligo.caltech.edu>

  • Created Property records for one server @ LHO and three servers @ LLO.
  • Coordinated the payment for taxes and duties to US Customs for the Gingin equipment in Australia.  Account Number LIGO.TEC-1.3-NSFLIGOFY02ON  $9,814.80.
  • Preparing a Memorandum to clear old items from LHO's inventory (eight Gateway computers).

DOCUMENT CONTROL CENTER (Turner, Mak)

>From: Linda Turner - turner@ligo.caltech.edu>

  • No report.

>From: Cleveland Mak <mak_c@ligo.caltech.edu>

  • No special projects to report.
  • Processed two sets of drawings.
  • Processing presentations for the LSC Meeting this week at LSU.
  • Scanning - Progress continues on scanning of contract closeout files:

Week Ending

08/17/2006

In

Out

Packages

28

5

Faxes

23

17

FINANCIAL SYSTEMS (Funaro, Brambila, Kaufman)

>From: "Funaro, Catherine" <Catherine.Funaro@caltech.edu>

>From: "Brambila, Ruth" <Ruth.Brambila@caltech.edu>

  • No report (vacation).

>From: Florence Kaufman <fkaufman>

  • Attended a Sponsored Research Forum on August 8th.
  • Noted than incorrect expenditure types were for stipends paid to students traveling to Italy, which resulted in the imposition of $9000 in Indirect Cost.  Corrections have now been made and the Indirect Cost has been reversed.
  • At Ed Jasnow's request, started to work on a process for a "Roll Up Signature Mechanism" for the Payroll Distribution Confirmation (PDC) reports that are required to be signed by Principal Investigators on a semi-annual basis.
  • Discovered that we apparently have not received the Payroll Distribution Confirmation report for the period from October 1, 2005 to March 31, 2006.  These reports were supposed to have been distributed on July 28th, and have to be turned in with signatures by August 28th.  Have requested a copy of the report from Project Accounting.
  • Financial reports can be found at: http://docuserv.ligo.caltech.edu/~fireport. (For passwords contact Florence).

SUBCONTRACTS MANAGEMENT (Jasnow, Salone)

>From: Gina Salone <gsalone@ligo.caltech.edu>

  • Nothing significant to report.

>From: Ed Jasnow <jasnow@ligo.caltech.edu>

  • Four bids were received for the machined parts for the LLO SEC outdoor wall exhibit (kinetic facade). The bids ranged from $29,250 to $65,830.  The low bidder, Scientific Fabrication Service, Inc., of Plaquemine, LA, was visited by Allen Sibley and Janeen Romie, and was found to be fully qualified to perform on this effort.  A purchase Order will be issued to this company this week.
  • The LLO Science Education Center is nearing completion, with beneficial occupancy scheduled for the end of this month.  At that point, an inspection will be conducted and a punch list developed.

PROPOSALS and REPORTS (Lindquist)

No activity this week.  We will soon be modifying the budget model for FY 2007 to reflect organizational changes.

CHANGE CONTROL/CONTINGENCY MANAGEMENT (Lindquist)

  • There are no open change requests.

HUMAN RESOURCES (Akutagawa)

>From: Cindy Akutagawa <cindy@ligo.caltech.edu>

  • A meeting of the LIGO Staffing Committee is scheduled Monday, August 21, 2006.  The agenda has been posted for Monday's meeting.  All files are posted and up-to-date on the web page.

Quality/Safety (Tyler)

>From: Bill Tyler tyler@ligo.caltech.edu

Nothing significant to report this week.

The LHO safety audit is planned for next week (Monday and Tuesday).  The results of this audit will be reported next week.

LIGO Hanford Observatory (LHO) and Interferometer Operations (Raab)

Summary of S5 Activities at LIGO Hanford Observatory (compiled by K. Kawabe)

Another very good week, with solid performance from both of the interferometers.

Duty cycles this week (Tue Aug/08 to Mon Aug/014) were: H1 89.53 percent, H2 91.49 percent. Most notably, both H1 and H2 marked 100% duty cycle all day long on Saturday 12/Aug.

As always, inspiral range of H1 was 13-14 Mpc at night, somewhat lower during the day. For H2, it was 7+ at night and 6.5-7 during the daytime.

There was no commissioning effort this week, although some maintenance activities were done on Tuesday.

S5 highlights from the LHO elog are bulleted below:

LIGO Livingston Observatory (LLO) and Interferometer Operations (Zucker)

No report.

Mechanical and Optical Systems (Coyne)

No report.

Controls and Data Systems (Bork)

No report,

40-Meter Interferometer (Weinstein)

No report.

Thermal Noise Interferometer (Black)

No report.

LASTI (Ottaway)

No report.

CIT Science Group (Lazzarini)

No report.

Laboratory Computing (Anderson)

LDAS Software Systems (Maros)

Work has begun on sunopteron and linuxopteron to get a fully functional /ldcg for building LDAS. The process is slower than expected as changes are required for 64 bit support for the opteron chip set. For instance, sunopteron required gcc 4.1.1 to be installed to get a working C++ compiler.

Efforts to get TclGlobus 64bit working on tandem-ii failed. After modifying the configuration scripts to be able to build, it now appears that the code base itself is not 64bit clean.

Modifications to the cmonClient resulted in both UWM and PSU have been removed from the server list (PR#3048) and an issue of cmonClient generating an error when displaying resource differences on a platform without tkdiff (PR#3050) being resolved.

The cntmonAPI was also modified. Database graphs no longer incorrectly exclude non-X509 user connections (PR#3039). Also, the abortJob command for the "ALL USER COMMAND" test is once again available under the username/password mode (PR#3038).

Some memory leaks for the getMetaData command have been addressed (PR#3033).

Finally, the manager no longer leaks manager key into the log file (PR#3043).

System testing of the LDAS software was done using version 1.8.270. Lsync continues to see RDSVerify and some /scratch directories. Also ran Greg's createRDS loop test on ldas-dev and got intermittent failures due to frame thread callback sometimes failing to find .tmp file. A new problem report was files describing this problem (PR#3040).

Additionally, changed password authority to emaros by updating the cntlmon resource variable at all sites and tandems (PR#3037).

LDAS System Administration (Anderson)

Caltech

(Dan Kozak)

  • Did RLS cleanup for S5 Burst-MDC files.
  • Did some work with my new SAM dump to du script to identify deletable data.
  • Did a throttled rm of some Laura Cadonati's 'trashcan' directories.
  • Started working on script to identify directories with lots of files in them.
  • Worked on LLO disk full condition (/archive).  While there were several problems, mostly human error (/archive/backup/fb0_frames was marked archive -n from when we were rsync'ing to it, a couple of directories were pinned to disk that shouldn't have been), there were some L0 files had not gone to tape when they should have.  I'll start monitoring the logs from the nightly samfsdump to try to catch this.
  • Working on repacking tapes at LLO from long ago (data strewn across many tapes problem).

(Phil Ehrens)

  • Upgraded ldas-suntest5 and ldas-suntest4 to Solaris 10 upgrade 2.  Documented upgrade glitches and required workarounds. In general, it is inimical to the upgrade process to have more than one submirror configured for a metadevice during the upgrade process, as the upgrade process attempts to obtain exclusive lock on ALL existing local partitions, as does the metadevice manager, so that they are always in conflict when multiple submirrors are defined.
  • Reconfigured filesystem on ldas-suntest2 to a mirroring configuration.  This required copying 12Gb of files to a temporary location and then copying them onto new mirrored partitions. I used the mv command to do this since it preserves links and file ownership.
  • Performed extensive investigation of x509 certificate disposition on ldas gateway machines and began preparing scripts and documentation for managing the critical certificates required for data transfer and analysis.
  • Cleaned up a number of expired certs that were no longer associated with any known activity. 54 certs or links to certs were identified on ldas machines exclusive of user certs.

(Erik Espinoza)

  • Deployed kernel option "iommu=off" to first rack (node1-45).
  • Deployed kernel option "iommu=off" to last 7 (node321-327).
  • Attempted to reproduce notification bug on frog (desktop), was actually a sendmail crash on ldas-grid.
  • Re-enabled Condor on node301-308.
  • Created home dirs for users on sunopteron.
  • Enabled snmp on Cisco 6509 switch.
  • Crashed nodes:

-         node246: 8/13

-         node143: 8/16

(Stuart Anderson)

  • Successfuly tested a Linux kernel option to prevent MCE GART errors on the LDAS-CIT Opteron cluster.
  • Continued testing of possible replacement memory for the LDAS-CIT cluster with mixed results.
  • Copied the S4 Burst MDC frames into the LDAS-CIT cluster scratch disk space.
  • Re-enabled DedicatedScheduler in Condor pool to support Parallel Universe jobs currently being used for numerical relativity MPI jobs.

Livingston

(Dwayne Giardina)

  • Ejected tapes for shipment to CIT, imported and labeled new tapes.
  • Failed disk in T3-4, replaced with disk from T3-16.
  • Sorted 'failed' disks in T3-13 and T3-16, found 9 that were no longer reporting as failed.  placed them all in T3-16 and built a new volume on those disks.
  • /archive reached capacity, disk2disk process were dead.  Dan cleared up disk space, I restarted disk2disk scripts.
  • Organized LDAS room, repackaged replacement parts, labeled boxes, removed most cardboard containers from room, etc.  Need to make room for new tape storage cabinet.
  • TempAlarm work, still trying to get email alerts functioning properly.

Hanford

(Ben Johnson)

  • Received 2160 tapes from disk-o-tape. We received the proper number, and they all appear to be labelled correctly. I've only checked the labels of ~1200 of them, but I assume the rest are ok.
  • Discovered the "ipfs" command, which should save the ipfilter state table; good for reboots, and reloading the ipf kernel module.  Unfortunately, it does not work properly on Solaris 10 U2 (or U1), due to what appears to be a Solaris bug (Sun doc ID  5040248).
  • DMT monitors are now writing to the fb0_frames filesystem. Work is still needed to move the backup script and LDAS+GC NFS mounts to the new filesystem+dir.
  • /frames@LHO is close to maximum capacity. Dave will reduce the second-trend look-back to give us some breathing room.
  • Assisted MIT with getting the DiskCache dataFindSerer, and supporting processes, working.

General Computing (Wallace)

MIT

(Larry for Keith)

  • Worked documentation and getting things ready for the new sysadmin.

Livingston

(Dwayne)

  • Created GC accounts for a few more SciMon camp attendees
  • Continued training our spam filter
  • One WAP required power cycle
  • Treating a computer for spyware, scan is running now
  • Discussed new network setup with Shannon
  • Applied Windows updates to several computers, including security server
  • Other usual user requests and support

(Shannon)

  • Set up Solaris 9 & applied all current patches on the new V490.
  • Installed a substantial amount of software for the SciMon camp this week.  Installed this in such a way that it should make setup of future DMT machines easier.  This really needs to be done because the DMT machines have never really been standardized at LLO.
  • Updated the IDS rulesets to catch the new microsoft vulnerabilities.
  • Updated the LHO CDS IDS as above.
  • Worked all day tuesday with the fiber subcontractor for Charter.
  • Did some troubleshooting on the auditorium lighting with a Leviton tech.  Spent a couple of hours trying to determine what we thought the problem was.  We think we know where the problem is and Allen is working on getting a replacement part and/or a tech out to repair it.  Lighting is done by bypassing the unit for now.
  • Poured through the IDS logs.
  • Put together a rough drawing of the future network topology as part of the planning for the new backup connection.  Spent some time going over the planned changes with Dwayne.  Monday I will have to meet with Boyd at LSU to discuss the changes.
  • Investigating switch software on the foundry switches.  Need to find out just what kind of MAC filtering they can do, or perhaps 802.1x.  Hopefully they can do auto-assignment of VLANs based on MAC address.
  • pcard & travel reports, etc.

Hanford

(Christine)

  • PNNL will have a power outage to upgrade their cooling and power distribution systems which will take down the LHO primary GigE network. The primary network will be down from 3 p.m. 8/18 through 10 a.m. 8/21. The backup FastEthernet network is now working and will become the primary network during the above mentioned times.
  • Ran some more tests on the backup network over the weekend.  I was able to determine that with the FastEthernet interface card on its own controller on the router the backup network worked for all outbound traffic, but not for incoming traffic.  Further tests showed that ESnet did not automatically route our IP addresses to the backup network.
  • Worked with ESnet on Tues. 8/15 to manually re-configure the routers on both ends to route traffic on the backup network.  This worked with out errors.  The plan is to manually re-configure the routers to use the backup network at 3 p.m. on Fri 8/18 and then about 10 a.m. Mon. 8/21 manually re-configure the routers back to using the primary GigE network.
  • Meanwhile, PNNL was able to find an alternate fiber path for the backup network that does not go through the same building and switches as the primary network.  The existing media converters worked on the new fiber path.  This is a permanent change.  So now our two networks take separate paths from PNNL to Seattle.
  • Notified Dave that his CDS intrusion detection system is sending emails to the GC root account.  Dave is working with Shannon to fix this.
  • Straightened up the server room and computer user room for the upcoming safety audit.
  • Still working on the vulnerability report and the Linux NIS+ client.

CIT

(Mike)

  • Continued work on DCC's webserver trying to get SSL to work.  This is an on going project. This has something to do with the way the webserver & SQL server comunicate with each other. I have put in a call to File Hold's tech support regarding this issue.
  • Still working on a Sun workstation for the W/B control room.
  • Continued work on maintaining our mail servers with Larry W.
  • Swapped out Bruce Sears with a new Sun workstation & monitor.
  • Working the spam filers searching for false positives.
  • Surplus old equipment, working with Larry and Rod on this.
  • Other misc. user/surf student support.

(Veronica)

  • LIGO:  Website updates.  Roster database updates.  High-resolution images for an online project of NSF-supported observatories.  User support.
  • LSC:  Website updates.  Posting the talks of the August meeting.  Keep working on the LAAC website.  Updates of the LSC-related mailing lists.

(Christian)

  • Installed new 24 inch LCD monitor for Julie Hiroto and Cindy Akutagawa.
  • Irene Baldon - Replaced toner and drum kit on Irene's local printer.
  • 3flr W/B - Replaced toner cartridges on HP 5500 printer.
  • Working on creating and configuring Linux and Windows XP image on a Macbook Pro with Parallels software. Also, trying to run Linux on a new Dell workstation with a 30 inch monitor.
  • Worked on the Spam Filters with Mike and Larry.
  • Other misc.: Continued onsite software/phone support.

(Larry)

  • Worked a number of procurement issues.  Sent in the list of items for the Foundry maint. contract renewal.  Purchased a couple of notebook computers. Have a few more to order and at least one more workstation to order. Working on a SUN disk system order but this is still some time away.  Purchased a number of misc. items for people. Also, purchase a hard case for notebook computers, will purchase a couple more, these are for those that will be traveling to areas that will not allow the notebook as a carry-on.  Resolved the Primavera payment issue.  Helped Gina with the new copier procurements.  Spent a number of hours doing the P-card reconciliation. The biggest problem this time was getting invoice information from Dell.
  • Assisted Rod and Mike with some of the GC equipment removal. Mostly, getting a lot of the old stuff put into a central location.  Also, working with Rod on getting equipment to foreign locations.
  • Assisted/Assisting Mike on a couple of OS installations.
  • Working with Christian on getting the 30inch monitor setup working in a LINUX environment. The difficulty is getting h/w that will work in the space of the display case and will drive the monitor.
  • Assisted the DCC with a number of issues. Most dealing with the new system.
  • Checking out the SURF students as they leave to their home institutions. This is taking a little more time in-that many of them are getting extensions for their accounts.
  • Worked a couple of small GC issues for the MIT group.
  • Assisted a number of users with different issues. Including plans for new setups and logistical issues covering travel.
  • Worked the mail servers. The spam storms and people sending large files are still causing problems with the system to where we have to monitor it on a regular basis.

Mail Statistics for August 10-16, 06

Mail Statistics

8/17/2006

Rejected Messages

30,254

Virus Messages

1,683

Accepted Messages

21,356

Total Messages

51,610

Advanced LIGO and Supporting R&D (Shoemaker)

No report this week.

For additional information about this report, contact Stan Whitcomb or Phil Lindquist