>From Peter Gutmann
The city of Auckland has its power provided by Mercury Energy, who have four
110kV cables feeding the central business district. Two of these cables have
copper conductors inside a pressurized nitrogen jacket (apparently we're one
of the few countries which use these) and were installed in 1975. The other
two are oil-filled and were installed in 1947, so they're over 50 years old
although they are supposed to have a very long life expectancy (they were
only used as backups anyway). The suspicion is that the El Nino summer has
dried out and heated the ground so that vibration and ground movement
(shrinkage) have damaged the cables. With daytime temperatures as high as
30C (86F) and humidity up to 75%, air conditioning loads have soared (at one
point it wasn't possible to get fans in the city). Ground temperatures have
also been very high, accelerating the ageing of the cables. It's also
possible that variations in load have caused enormous dynamic magnetic field
changes, so the electromagnetic forces between the three conductors and the
steel pipe (more on this below) caused the core to move around inside the
pipe (this may have been the reason why the later cable failures occured,
they would have been severaly overloaded when the first cables went). Also,
due to very slight diametric temperature gradients, the differential thermal
expansion of a cable that big causes bending and warping forces. The exact
cause of the failures won't be known for certain until the damaged cables
have been subjected to lab analysis.
Because of one or more of these reasons, all the cables have failed, leaving
the central city without power. So far this has affected (at various times)
a number of banking data centres (the first day the power went out was on the
Thursday when everyones pay is supposed to be processed - the data centres
themselves have generators, but the sources feeding them information don't),
the stock exchange, some (unidentified) central city post office buildings,
customs and immigration, inland revenue, internal affairs, social welfare,
the Auckland City Council, the central police station, Aucklands main
hospital and medical school complex (they have generators, but one of them
failed, leaving the childrens hospital without power for awhile), the city
campus of the university and technical institute (affecting 30,000 students
in the middle of enrolment), several TV and radio stations, and God knows
what else (the government departments have tentacles all over the city, so
it's not so bad for them). Although many of these places have generators,
there were various glitches in switching over and one or two breakdowns which
have caused problems, and most of the generators can't handle anywhere near
the load being placed on them but were designed to power only essential
services. It's possible that the power company may not survive the lawsuits
which follow this (taking out some suburb is serious enough, but taking out
the central business district with its cluster of multinational accounting
and legal firms, banks, government departments, and whatnot is really bad).
There's a web site http://www.mercury.co.nz/cable/index.html with updates on
the situation, this is a Mercury Energy site so be aware that it's subject to
the usual degree of spin control (there have been discrepancies to date
between their statements to the media and what's actually happening).
Updates to this info will periodically appear on
http://www.kcbbs.gen.nz/users/peterg/power.txt, moving back to my more usual
http://www.cs.auckland.ac.nz/~pgut001/misc/power.txt once the University has
power again.
Various data points:
- - The mayor has told businesses in the central city to either close or relocate
for at least a week. Staff are being relocated as far away as Australia.
- - At first the estimated time to repair was a week, now the official estimate
being fed to the media is 1-3 weeks but the estimate from power company
workers is a month at least (these figures change constantly, they seem to be
getting worse). Usually it takes about 3 weeks to repair a fault in a cable,
however with crews working around the clock and the bare minimum of testing
of the repair, it may be possible to do it in a week, provided it holds
together (however, see the comments further down on overloaded cables).
- - In the last five years, Mercury Energy have followed the present economic
wisdom of aiming for efficiency and a good return to their shareholders (the
Mercury Trust), raised power prices, reduced their field workforce by half,
and raised management salaries by 30%. In addition for the last three years
their energy has been poured mostly into an (ultimately fruitless) struggle
to take over their neighbouring power supplier, Power NZ, which cost Mercury
$300m. In the middle of the first week without power, the Auckland City
Council called an emergency meeting in the town hall to discuss the problems
people were facing. Some of the business owners who attended were on the
verge of bankruptcy because of the lack of power, but Mercury didn't even
bother turning up. This sort of thing isn't endearing them to their
clients/victims.
- - Workers from other power companies are being brought in and working in
civvies with company logos on their equipment painted over to hide the extent
of the problem. Workers were flown in from Sydney, Australia to fix the
cables because it's too costly to keep specialists like this on hand at all
times, so there's noone in NZ who can do this sort of work. The estimate is
that it'll take about a week without power to redo these, and if the load
placed on them is too high they'll fail again (the faster they bring the
cables back online, the greater the chance of them failing, making it
necessary to start again from scratch). Normally it takes a week of testing
after a cable has been repaired, this has been brought down to a single day
because of the urgency with which they're required, which isn't a good thing.
- - Apparently the idea of moving ships from the naval base on the other side of
the harbour across to the Auckland waterfront to act as floating generators
was considered, but there are problems with feeding the power from the ships
to the city. There's also the problem that there's nothing around which can
generate even a fraction of the power required. Another idea which was
considered is using one of the Cook Straight ferries (which could in theory
provide around 10MW) as a floating generator. Currently a couple of
waterfront businesses are being run with power from ships acting as floating
generators.
- - Because the central city was without power, there was a civil defence callout
to avoid a potential crime wave. Police were called in from other parts of
the city to patrol the city center, leading to a lack of policing in other
areas. The lack of power is affecting building access control systems and
alarms, buildings have to have doors propped open so people can get in and
out, so there's no real protection for the building contents. The services
of private security firms are in great demand.
Updates to this info will periodically appear on
http://www.kcbbs.gen.nz/users/peterg/power.txt, moving back to my more usual
http://www.cs.auckland.ac.nz/~pgut001/misc/power.txt once the University
has power again.
- - Since water and sewage rely on electrically-driven pumps to get them into
office blocks and towers, these services often aren't available either.
What little power is available is being used by emergency and civil services
as far as possible, with other services like traffic lights being run if
there's anything to spare. Many office blocks have no power, water, or
sewage services available. Combine the lack of sanitation with abscence of
airconditioning and you can imagine what conditions are like in parts of
these buildings.
- - There's a serious problem with food in restaurants spoiling due to the lack
of refridgeration, with health inspectors out in force checking for
compliance with safety regulations. The Ports of Auckland shifted 500
containers to Wellington for unloading before the goods in them spoiled, and
the city council parked a large refridgerated container unit powered by a
generator in a city square for use by restaurants, although it's unclear how
much this will really help.
- - Provisional tax payments are due at the end of the week, inland revenue have
so far indicated that they won't be making allowances for the fact that
businesses can't make the payments until power is restored. The matter has
now been taken to the minister.
- - In one 10-15 story office block, sprinklers were activated by the power
outages and continued spraying water into the building for quite some time.
A comment from someone who saw the aftermath was "They may as well demolish
the building and start again".
- - One company flew in a generator from Poland to try and keep things running.
Another company is flying in generators from Australia. The lack of power is
a UPS vendors dream, they're almost impossible to obtain. One company asked
that their order of UPS's be shipped with a full charge. Mercury are busy
hiring out what generators they have at $1000/day to the businesses they've
cut off, and on Wednesday chartered an Antunov 124 and flew in six large
generators scraped together from all over Australia at a cost of $1.5M. The
producers of Xena and Hercules have lent the generators they normally use to
the city (thanks guys).
- - The organisations which do have generators are finding that, although you can
run them for the usual smoke test every few months, they can't supply power
for any extended period of time (the generator from Poland couldn't actually
be run once it was landed due to noise or exhaust problems). Other companies
have found that their generators have problems with fuel tanks placed too
close to the generator or inadequate cooling and exhaust ducting. On the
first full day without power, there were four fire service callouts to
generators which caught fire because they weren't used to running for
extended durations.
- - There is by now enough raw data about Disaster Recovery Plans which don't
work for an entire conference.
- - From talking to people in various affected central city buildings, as soon as
the power comes back on the affected law firms will be handling enough
lawsuits to keep them in clover for years. In theory the current commercial
monopolies inherited the privileged positions of the old Power Boards from
which they're descended, making it impossible to sue them for failing to
provide a service. The only thing they can be sued for is negligence, there
is mounting evidence that this will be possible. Mercury say that the cables
were all dug up quite recently to sleeve the joints as required by the
Resource Management Act to prevent loss of oil or gas into the environment,
and that this indicates that they were being maintained (this, however, seems
more like "wave a dead chicken over them to stop us being fined under the
Resource Management Act" than any real maintenance). Contradicting this, the
chairman of the Mercury board has said that the best thing to do isn leave the
cables buried and not touch them, so digging them all up may have made things
worse.
In 1993, five years before the current crisis, Mercury Energy had become
sufficiently concerned at the condition of the cables that they asked Leyland
Consultants to prepare a report on them, which recommended that the main
cables were at the end of their life and should be replaced (the engineers at
Mercury had been expressing concern about the state of the cables for some
time before that). When the first cable failed, the fallout from the outages
it caused were such that Mercury avoided making any real cuts to try and
reduce the load on the remaining cables (this weeks Computerworld, prepared
before the outages, contains an article reassuring everyone that the problem
is solved and everything is OK). Mercury ran an emergency feed for several
miles over a string of poles, which had hardly been completed when the second
cable failed. They then tried to force a full load over the remaining cables
by management will-power alone, which unfortunately wasn't enough to overcome
the basic laws of physics, and everything which was left failed as well.
It's not unlikely that the combined legal resources of everyone they've
annoyed will find enough material in there to get to them (there are probably
armies of lawyers sitting around candles right now scrutinising the relevant
legislation), and the case will eventually get heard in the Auckland high
court by a judge who's just spent a week working by candlelight
because they don't have power either. There are plans for multiple
mass class action
suits against Mercury, one group has even said they'll get criminal charges
pressed against them for crimes against public welfare. I think I'll join
the class action suits as well, the fact that the university machines are
down means that I've had to use tin to read news for nearly a week, that's
got to be worth several hundred thousand dollars compensation for mental
anguish.
The Prime Minister has already made a plea for people not to engage in a
witch-hunt against Mercury, but this appears extremely unlikely as things get
more and more desperate each day. So far Mercury seem to have blamed the
whole mess on an act of God, knowing that it'll be more difficult for people
to get compensation cheques out of Him than from Mercury.
There's to be a government enquiry into the whole matter in the near future,
one of the things which will be investigated is the extremely peculiar
relationship Mercury have with the law firm Russell McVeagh. Although
Mercury is 100% owned by a trust, of the 9 Mercury directors only 4 are
appointed by the trust, with the majority of 5 being appointed by Russell
McVeagh. Mercury then in turn appointed Russell McVeagh to act as their
legal advisors. Although this is just the usual paranoid structure set up
when you ask lawyers to manage things for you, to the public it looks very
suspect, and is likely to be a primary target for any government
investigation.
- - Mercury have a tunnel under construction which (had it been finished) would
have brought in the required power, however it passes under the buildings
which are now affected and during the planning stages the owners of the
buildings raised all sorts of objections to it which held things up for some
time. It'll be another two years before it's complete. I see a long period
of finger-pointing to follow.
- - The cables in question generally consist of a central copper conductor inside
a fairly sizeable pipe filled with a pressurized oil dielectric, with tens to
hundreds of thousands of litres of oil contained in a typical complete pipe.
There are actually three central conductors, each about 8cm in diameter with
a 2.5cm solid copper core. These are wrapped in 118 layers of special paper
tape, followed by another copper sheath, and then a spiral wrap of 5mm bronze
wire. These three wrapped conductors are then twisted together and pulled
through a steel pipe in ~750m segments, that being the longest length they
can pull through the pipe. At each segment join is a splice which has
temperature sensors, oil flow rate monitors, and pressure monitors to check
oil conditions.
Once the initial cable is installed, it's evacuated to both leak test it and
remove any contaminants. Then it's flushed with dry nitrogen, re-evacuated,
and pumped full of high tension oil under vaccuum to force any remaining gas
and contaminants out. The oil is first sprayed into a holding chamber at
high temperature from hundreds of nozzles to get the maximum surface area,
the high temperature causes all the crap to boil out so the good stuff which
is left can be pumped into the pipeline. Finally, the oil-filled pipe is
pressurized to about 200 PSI for awhile, after which it's powered up. Each
of the three cores typically carries 600-800 amps of three-phase power.
At both ends of the pipeline are large holding tanks of oil, lightly
pressurized under a blanket of dry nitrogen. There are pumps at both ends
which are reversed every 6 hours so the oil oscillates back and forth in the
pipe, which eliminates hot spots and spreads the heat over a large area. The
last hour of the cycle, things are slowly run down to get zero oil velocity,
then things are reversed and slowly run up to move the oil in the opposite
direction.
Performing the repairs is difficult because at each point they have to build
a miniature clean room with special air conditiioning and filter units to
keep dirt, moisture, sweat, and other contaminants out of the joint. Closing
up the pipe after repairs is a special task in itself, since the pipe is
filled with oil and paper, it has to be done with special equipment and
operators, and takes 8 hours to weld one section of pipe. If the pipe is too
hot to touch 10cm from the welding, they have to stop and let it cool before
they can continue. After that, they have to go through the lengthy refilling
process described above.
- - Here's a panoramic photo of the central city by night:
+-------------------+
| |
| |
| |
| |
+-------------------+
In the foreground you can't see the town hall, with the Aotea Center and
council building in the background. To the right you can't make out the side
of the Sky Tower, with the casino beneath it.
Here's another shot, taken from down by the waterfront:
+-------------------+
| |
| |
| |
| |
+-------------------+
In this one you can't see Queen Street and the businesses up either side,
with the Coopers and Lybrand tower not visible to the right.
I'll put more shots online later if I can find somewhere to power a scanner.
- - The power outages did bring out some good things though. After the power had
been out for about half an hour on the first day, someone mentioned that the
fridges downstairs wouldn't be powered. In the spirit of true cooperation
and self-sacrifice, everyone immediately rushed downstairs and saved all the
beer from getting warm.
- - Auckland joke (you probably have to be a NZ'er to get this):
Q: If there are power shortages, which will you keep running, the cappucino
machine or the air conditioner?
A: Both.
- - More jokes:
Q: How many Aucklanders does it take to change a lightbulb?
A: Does it matter?
Because of the Y2K problem, various people have planned vacations away from
computer-powered devices in December 1999. Central Auckland is looking like
a good place to take this vacation.
Disclaimer: This information provided here is the best I can do, Mercury have
been rather active at keeping people in the dark recently.
------- End of Forwarded Message