Disaster planning
From CSL Wiki
This is a collection of notes about Disaster Planning for CENS computing and network resources, with a focus on the 3551 Boelter Hall facility.
Threats
Earthquake, major (~7.8M)
When the "Big One" strikes, extensive physical damage to 3551 Boelter is expected. In particular, the Math Sciences Wing above 3551 can be expected to sustain sufficient damage that access to 3551 will be prohibited following the event, due to either structural failure potential or release of asbestos normally safely encased in stucco-wrapped beams found throughout the western end of 3551.
In this event, systems restoration must rely on archives stored "off site", which at least means elsewhere in Boelter Hall (eg, 3420 Beolter CSD DCF area), or other Engineering buildings with a modern seismic safety rating (eg, CNSI or E-IV or E-V), or elsewhere on the UCLA campus (eg, Melnitz), or off-campus (eg, staff homes in the San Fernando Valley or a commercial service such as Iron Mountain). Also note that a temporary facility must be constructed elsewhere to host such restoration, using other hardware; such hardware does not exist at present, but could be acquired rapidly and installed in a "friendly" facility elsewhere on the UCLA campus or in off-campus leased space.
Earthquake, moderate (~6.8M)
In this event, building structural failure does not occur to any significant extent.
However, severe shaking of 3551, especially within the 3551B server closet, is expected. Server racks, especially heavy 4-post racks with > 200 kilos of equipment, are likely to be severely stressed by rocking.
Racks should be organized with heaviest equipment at the very bottom (eg, UPS with batteries or high-density 48-drive RAID system), and lightest equipment (older 1U servers, bulky tape drives) near the top, for increased stability during a seismic event.
The server racks need to be secured to floor and both side and read walls, to minimize movement potential that might cause additional damage to equipment and cabling.
The in-rack equipment needs to be secured with locking rails.
DEFICIENCIES:
- four-post racks not secured to floors or walls; some may still be on rollers rather than supported by stationary feet.
- some equipment (primarily Tripp-Lite UPS) needs suitable rails for four-post rack mounts.
Earthquake, minor (~6.0M)
3551 Electrical failure, complete
3551 power is fed from Math Sciences (NOT Boelter). A complete 3551 power failure might only affect 3551, but it is more likely to have also affected the primary Math Sciences building.
3551 Electrical failure, partial
In this event, one or two of the three 220V service legs feeding 3551 fails, taking down a portion of the equipment both in the server closet and in 3551 at large. This will also affect the HVAC system, which relies on three-phase power to operate the fans.
Boelter Hall electrical failure
All of the network traffic exiting CENS travels through one or more CSD network switches, all of which are located in various wings of Boelter Hall and dependent on Boelter Hall power. Note that DCF and one switch are in E-II, while the CENS admin offices and the primary switch connection are in E-III. SEASnet (2nd floor Boelter) is also in E-III.
Fire sprinkler water dispersal into 3551B Server Closet
Occurred in December 2007 from joint leak in sprinkler pipe above server racks; repairs completed without damage to equipment. Note: failure occurred less than two years after occupancy.
Sprinkler water supply is from ???; is separate system from Boelter Hall.
Water intrusion into 3551 generally
Occurred in January 2008, due to leak on 7th floor roof.
HVAC failure
3551 has its own internal, monolithic HVAC system, dependent on chilled water fed from Boelter Hall. Need to install temp sensor/notification hardware/software into server closet.
Loss of key personnel
How/where to archive institutional knowledge in 'safe' places? RG knows too much that isn't documented in accessible places
