Advanced LIGO - Data Acquisition

The differences between the initial LIGO and Advanced LIGO Data Acquisition, Network & Supervisory Control (DAQ) requirements derive from the improved sensitivity and performance of the Advanced LIGO interferometers. We specify an increased ADC dynamic range to more easily accommodate the great disparity between narrowband features and lower broadband noise, and a greater number of channels to monitor a greater number of active control systems.

The principal Advanced LIGO reference design parameters that will drive the data acquisition subsystem requirements are summarized in Table 1.

Table 1 Principal impacts of the Advanced LIGO reference design on data acquisition and data analysis system. The number of degrees of freedom (DOF) is indicated for the main interferometer to give a sense of the scaling.

Parameterization	AdvLIGO reference design	Initial LIGO implementation	Comment
Whitened h[t] dynamic range	>=121 dB (20 bits)	96 dB (16 bit ADC)	Range of h[t] is determined by narrowband feature amplitude and broadband noise floor
Acquisition System Maximum Sample Rate, s/s	16384	16384	Effective shot noise frequency cutoff is well below f Nyquist (8192 Hz)
Active cavity mirrors, per interferometer	7	6	Signal recycling mirror will be added
Active seismic isolation system servos	11 chambers per interferometer, 18 DOF per chamber, total	2 end chambers per interferometer, total, 12 DOF	Initial LIGO uses passive isolation with an external 6 DOF pre-isolator on end test masses; Advanced LIGO uses active multistage 6 DOF stabilisation of each seismic isolation platform
Axial and angular alignment & control, per interferometer (4km/2km)	SUS DOF: 42 L DOF: 5 () DOF: 12	SUS DOF: 36 L DOF: 4 () DOF: 10	Advanced LIGO has 2 additional cavities. Each actively controlled mirror requires 6 DOF control of suspension points plus (,L) control of the bottom mirror
Total controlled DOFs	257	62	Relative comparison of servo loop number for maintaining resonance in the main cavities (PSL and IO not included)

The reference Advanced LIGO design will have a broadband noise floor between narrowband features that is limited by radiation pressure noise at a level h[f]~2-3 10^-241/Hz^1/2, ~10x lower than the initial LIGO design. Our present best estimate is that the Advanced LIGO dynamic range requirement for whitened signals at the interferometer output port will be ~10x greater than the initial LIGO baseline, leading to a working requirement for ADC resolution of 20 bits.

Advanced LIGO will require monitoring and control of many more degrees of freedom (DOF) than exist in the initial LIGO design. The additional DOFs arise primarily from the active seismic isolation, with a smaller contribution from the move to multiple pendulum suspensions and the additional suspended mirror. Table 1 summarizes these modifications. Both the suspension and the seismic isolation systems will be realized digitally (except for the sensors and actuators) and the DAQ will need to capture a suitable number of the internal test points for diagnostics and state control (as is presently done for the initial LIGO digital suspension controllers).

Referring to Table 1, the number of loops per interferometer that are required for Advanced LIGO is seen to be ~250. This is to be compared to ~60 for initial LIGO. The number of channels that the DAQ will accommodate from the interferometer channels for Advanced LIGO will reflect this 4X increase in channel number.

Table 2 presents approximate channel counts classified by sample bandwidth for Advanced LIGO and compares these to initial LIGO values. These represent the total volume of data that is generated by the DAQS + GDS; a significant fraction of these data are not permanently acquired. Nonetheless, the ability to acquire all available channels must be provided.

Table 2 DAQ acquisition data channel count and rates

System	AdvLIGO reference design	Initial LIGO	Comment
Channels, LHO + LLO Total (Total: 3xIFO + 2x PEM)	5464 + 3092 8556	1224 + 714 1938	LIGO II will have ~4.5X greater number of channels
Acquisition rates, MB/s	29.7 + 16.3 46	11.3 + 6.1 17.4	DAQS II has ~3X total data acquisition
Recorded framed data rates, MB/s LHO + LLO Total	12.9 + 7.7 20.6	6.3 + 3.5 9.8	DAQS II has ~2X total framed data recording rate

The driving features of the Advanced LIGO hardware design are the increase in channel count and increase in data word length for the main sensing channels. The initial LIGO 16 bit ADCs will be exchanged for newer 32 bit ADCs (note: 20 bits are actually specified). Not all DAQS channels require the greater dynamic range. Moreover, the increase in acquisition bandwidth with double data-word size dictates that only those channels requiring the increased dynamic range should be upgraded.

The additional data channels required for the newer seismic isolation and compound suspension systems will require additional ADCs distributed throughout the experimental hall CDS racks. Additional racks will be required and can be placed alongside the present CDS racks within the experimental halls. In those cases where there is interference with existing hardware, racks will need to be located further away, at places previously set aside for LIGO expansion. Additional cable harnesses for new channels will be accommodated within the existing cable trays.

The initial LIGO data acquisition processors do not have excess capacity sufficient to accommodate the increase in acquisition rate and will need to be upgraded. The upgrade will be a combination of updating the hardware technology and using a greater number of processors. The existing fiber optic infrastructure will accommodate the Advanced LIGO DAQS changes without requiring an upgrade. The DAQ framebuilder and on-line mass storage systems will be upgraded to accommodate the greater data and frame size. The Global Diagnostic System (GDS) will be upgraded to handle ~3X as much real time data as the initial LIGO GDS.

At present, ADC technology is not capable of providing full 20-bit ADC precision at output rates of 16384 samples per second. Our experience indicates that the principal limitation is likely the ADC board design that uses the 24-bit ADC chip, and we may need to develop in-house or collaborative solutions with industry to meet our stringent requirements. Additional performance limitations may also come from the VME format of the boards that initial LIGO uses. The VME bus is a very noisy environment that may limit ADC performance, and we will study alternatives such as VXI for sensitive parts of the design.

This will require new solutions to be identified and prototyped to determine performance of candidate hardware solutions. Using the 40 Meter Interferometer at Caltech, which is designed to exercise the hardware and software environment for Advanced LIGO, we will perform much of this type of work.

Similarly, the GDS hardware will need to be scaled for the greater processing and throughput requirements. Parallelization techniques that are being used in the LDAS I design (e.g., passing messages across Beowulf clusters) can be introduced to solve compute-bound (but not I/O bound) data processing problems.

It is plausible that hardware technology trends will continue over the next 5 years. Thus, it is likely that the solutions required to support the ~3X increased acquisition rates and data volumes would become commercially available by the time they are needed. We have taken as the point of departure that "Moore’s law" will be a reasonable predictor of the growth in available performance.

The first phase will develop a detailed set of requirements for the DAQ upgrade. These will proceed with the development of a Design Requirements Document and a Conceptual Design. Activities that begin in this phase include the development and refinement of an Advanced LIGO model. This will produce a curve of strain sensitivity goal with sufficient details so that issues of dynamic range, etc. can be addressed with simulation to guide the hardware design. As refined design information for new SEI, SUS, and ISC subsystems becomes available, the channel count estimate and their sampling rates will be improved.

The second phase will incorporate results from prototyping. Preliminary board layouts for custom components will be developed as part of this stage. The procedures by which the existing plant will be de-integrated and the newer components introduced will be identified. Software development associated with DAQ II modifications of the DAQ I plant and infrastructure will begin.

The third phase will culminate in a detailed set of drawings, specifications, and procurement or fabrication plans for the DAQ II equipment. Fabrication will follow, and it is anticipated that primarily the LIGO Laboratory staff will carry out this phase as it was during initial LIGO construction.

This element includes all R&D, design, prototype testing, and hardware for the analog and digital signal conditioning electronics, computers, programmable items, networking, software, sensors, actuators and excitation devices for reading Advanced LIGO data and diagnostic data and operating diagnostic systems. Common elements of the supervisory control and human interface for subsystems, and the infrastructure (cable plant, servers, etc.) are also in this subsystem. The element includes all additions and modifications to the LIGO Global Diagnostics System (GDS) and the Physics Environmental Monitor (PEM) system.