DOE Releases Interim Report on Blackouts/Power Outages, Focus on Cyber Security

[Geoff Shively]

The Department of Energy (http://www.energy.gov) has released a interim
report on the Aug 14th power outages.

http://energy.gov/engine/content.doPUBLIC_ID=11860&BT_CODE=DOEHOME&TT_CODE=SPOTLIGHTDOCUMENT

*remote terminal units (RTU)
*distributed control systems (DCS)
*supervisory control and data acquisition (SCADA)

Straight to the good stuff, security.... which starts on page 29, (phase 2
of failure matrix) the SWG (Security Working Group, or Star Wars Galaxies ;)
outlines that computer failure was a primary cause of the blackouts. Logging
systems failed first, 3 Remote Terminals (RTUs) failed along with
unspecified 'dial-ups'. Can anyone shed some light on what 'dial-ups' they
are talking about?

~~~~~~~~~~~ Key Events ~~~~~~~~~~~
The first computer failure was to logging and alarm systems.

14:14 EDT: The control room operators lost the logging and alarm function.
14:20 EDT: Several remote location console failed
14:41 EDT: The primary control system server hosting the alarm function
failed/ Its applications and functions were passed over to a backup server
that was standing by.
14:54 EDT: The back-up computer failed, and all functions that were running
on it stopped.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Refresh rates of screens displaying critical data were reduced on an average
of 1 to 3 seconds, and slowed as long as 59 seconds per screen. For data
that changes hundreds of times a second, I would say that renders the
terminals useless. Quite similar to what Sandia National Laboratories
(http://www.sandia.gov/) illustrates here, misrepresentation or spoofed data
being displayed:
http://www.pbs.org/wgbh/pages/frontline/shows/cyberwar/view/

Sadly this report is over 130 pages, but draws no definitive links or cause
to the failures. In one paragraph malice activity is being denounced, the
next it's said that investigations are inconclusive and need more effort.

At least they admit all over the report that the SCADA, DCS, RTU, and EMS
systems are all insecure and highly vulnerable- something that many pledged
was untrue when this topic first surfaced.


Cheers,
Geoff Shively
CTO
PivX Solutions
24 Corporate Plaza #180
Newport Beach, CA 92660
http://www.pivx.com
gshively@pivx.com
949-720-4628

PivX defines "Proactive Threat Mitigation. Get a FREE Beta Version of
Qwik-Fix (tm)
<http://www.qwik-fix.net>

[Russ]

Well, they did specifically discount both current (at the time) Internet =
worms/activity, and terrorist activity, as having any part in the =
blackout. As for the RTU failures, FE told investigators they believed =
that was because they believed the RTU's "started queuing and =
overloading the terminals buffers". Given that the EMS Alarm program was =
already crashed at this stage, its feasible to see a real-time reporting =
terminal not know what to do (other than to page an FE IT person) when =
its host can't accept its input. Since they refer to the RTU's =
connectivity both as "dial-ups" and as "data links", its hard to say =
what they were. Nothing else in the EMS system failed until 14:54 when =
both primary and backup EMS servers were down, so its unlikely that any =
"network" connectivity between RTUs and EMS were interrupted due to the =
problems with the EMS systems until that time. Ergo we're left with =
"comms" problems between RTU and EMS that led some FE personnel to =
describe them as "network" problems. It may all have simply been the =
fact that the RTUs had stalled, sending no "comms".

Interesting that a page went to FE IT folks when the RTUs stopped, but =
nothing went to them with the EMS Alarm program "stalled".

I think the refresh rate of the EMS consoles isn't actually a factor. =
The alarm function "stalled", or "froze", and did not produce any =
alarms. That EMS consoles were being refreshed after 59 seconds didn't =
alter the fact operators weren't seeing new alarms. The lack of alarms =
coupled with the arrogance of the staff who insisted reports by others =
were mistaken led to critical failures in line load which ultimately =
left them unable to recover.

During the same period of time MISO's State Estimating system, which was =
receiving telemetry from much of FEs network, experienced a normal =
mis-match in load calculations. A manual process is used to correct =
this, and was done within ~30 minutes of its first occurrence near the =
FE problem time-frame. An operator at MISO, however, left the estimating =
system in manual mode and went to lunch. It was put back into automatic =
mode 93 minutes later, at which time it again had a mis-match =
solution...so it had to be manually corrected again. It wasn't back into =
automatic mode until 16:04. Hard to say it would have made a big =
difference if it had been running in automatic mode during this whole =
time. Probably yes, but given FE's adamancy they had good data, they may =
have spent an equal amount of time arguing over who knew what.

If either of these events occurred independently, its likely the =
blackout could have been avoided.

If FE's operators not been so sure of themselves, its likely the =
blackout could have been avoided.

Finally, FE's IT staff took 54 minutes to complete their first attempt =
at recovering the alarm process, this after both primary and backup =
servers had failed (14 minutes after both had failed.) They were =
obviously relying on the failure not transferring from the primary to =
the backup. 34 minutes after the first warm reboot, and 4 minutes before =
the EMS crashed again, they discussed with FE operators the possibility =
of doing a complete cold boot because only then were they informed that =
the alarm function wasn't running (still). FE operators dissuaded the IT =
staff from doing so, fearing they'd have less data then they already had =
(arrogance again, they had already demonstrated their inability to =
perform adequately with the "less" data.)

Unfortunately, nobody tells us how long it would have actually taken to =
do a cold boot, and FE's IT staff say they didn't find out that was the =
only way to recover the alarm system until after the blackout (meaning =
the warm boot was a useless effort in the first place.)

And during all this time there were those damn trees!!!

MISO failed to adequately warn, and FE failed to adequately control its =
security space (physically and electronically). And it all happened on a =
hot August afternoon.

Cheers,
Russ - NTBugtraq Editor