Re: Sun M-class hardware denial of service

[Theo de Raadt]

> I apologise if I'm misunderstanding you, but it seems to me that this
> issue can only be initiated by a privileged user on a domain.

If one domain can be broken into, and a Solaris kernel module is
loaded which then crashes that one domain, the entire machine
eventually has to be powered off to recover that one domain.

> The only system immediately affected is that particular domain.

That is WRONG. The long-term uptime of all other domains on the
machine are eventually impacted because the entire physical machine
must, after a service call to Sun, eventually be powered down.

Management eventually has to decide to impact the SLA's of all domains.
That means that Sun's promise of isolation is bunk.

> The
> removal of service from the other domains is a system/service
> management decision, rather than an exploit of some kind.

That is wrong, too. If any of the other domains were supposed to meet
five 9's SLA's, then the failure of one domain on that physical
hardware would impact the SLA's of all the other domains.

> That's why I don't view it as a DoS vulnerability.

How absolutely bizzare. Basically you spend half a million dollars on
Sun hardware, and it isn't required to do this better than VMWare? In
fact, it does it worse than VMWare. I am just stunned at your
acceptance of a serious problem.

> If you exploit this on your
> own domain, which then becomes unavailable then, frankly, tough.

If you exploit this on one domain, the other domains must all
eventually be powered off.

Compare this to VMWare. If an OS running inside VMWare was able to
cause a situation making it neccessary to reboot the host environment
and restart all VMWare instances, it would be considered a very
serious and significant security problem for VMWare. It would be all
over the news. (And at least with VMotion you can move the instances
to another machine; with Sun you cannot).

This issue is very significant.

> You
> wait until the frame administrators choose to power cycle the other
> domains to bring you back.

And what if your SLA's say that you are supposed to only go down for a
maximum of 3 hours a year, yet you need that dead domain back
immediately?

> You stated in your original message that this is a high-end frame, of
> the kind generally used by financial institutions etc. I would
> imagine any system which warrants this kind of hardware would have
> some level of redundancy or DR.

Oh great! Sun is off the hook for selling something which doesn't
work, and their customers must mitigate against it themselves.
Utterly ridiculous.

[Michael Wojcik]

> From: Theo de Raadt [mailto:deraadt@cvs.openbsd.org]
> Sent: Tuesday, 09 September, 2008 17:28
> To: B 650
> Cc: bugtraq@securityfocus.com
>
> > I apologise if I'm misunderstanding you, but it seems to me that
this
> > issue can only be initiated by a privileged user on a domain.
>
> If one domain can be broken into, and a Solaris kernel module
> is loaded which then crashes that one domain, the entire
> machine eventually has to be powered off to recover that one domain.

I agree with Theo. This is a privilege-escalation DOS attack, pure and
simple. A user with sufficient privilege in one domain, but not
necessarily in others, can 1) force that domain down for an extended
time, and/or 2) force all domains down.

"Privilege" isn't an absolute; there are degrees of privilege, and this
bug lets a user do more damage than their degree of privilege should
allow.

--
Michael Wojcik
Principal Software Systems Developer, Micro Focus

[Micheal Patterson]

----- Original Message -----
From: "Theo de Raadt" <deraadt@cvs.openbsd.org>
To: "B 650" <dunc.on.usenet@googlemail.com>
Cc: <bugtraq@securityfocus.com>
Sent: Tuesday, September 09, 2008 4:27 PM
Subject: Re: Sun M-class hardware denial of service



<snip>

>> You stated in your original message that this is a high-end frame, of
>> the kind generally used by financial institutions etc. I would
>> imagine any system which warrants this kind of hardware would have
>> some level of redundancy or DR.
>
> Oh great! Sun is off the hook for selling something which doesn't
> work, and their customers must mitigate against it themselves.
> Utterly ridiculous.

B 650, the major problem with that statement, is that most facilities
that have built up redundancy for such an issue have 100% or more backup
of the exact same gear. That means that their DR plan is still crippled
and subject to the exact same failure as the primary system. That isn't
an effective DR plan.

If the system were in place at say a nuclear power plant, and it was
sold as a method to have separation to eliminate any problems with one
system causing another to cascade crash, and this happens, that effects
many other systems. Regardless if the initiator of the failure is a
power user or not, the result is a total cascade failure and will result
in a full system shutdown shutdown to recover from. It's still, by
definition, a DOS. Simply because the actions of one individual, either
by accident or malice, results in the denial of access to a system or
group of systems.
If you're one of the domains that will be effected, and you're taken
down even though your network / system is stable and working properly,
that would be seen as an unnecessary outage. What happens if the system
doesn't boot back up properly after the power down? Now, the outage is
extended and perhaps critical systems are no longer available. I used a
nuclear power plant as an example, what if it were an airport, or a
city's 911 / Emergency service? Fire Department dispatch system? EMS
system? Do you still think that it's a non issue to take down an entire
system for one faulty domain?

--

Micheal Patterson
Senior Communications Systems Engineer
Rural Hospital Acquisition, LLC
405-917-0600

Confidentiality Notice: This e-mail message, including any attachments,
is for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all
copies of the original message.

[Florian Weimer]

* Theo de Raadt:

> That is WRONG. The long-term uptime of all other domains on the
> machine are eventually impacted because the entire physical machine
> must, after a service call to Sun, eventually be powered down.
>
> Management eventually has to decide to impact the SLA's of all domains.
> That means that Sun's promise of isolation is bunk.

The recovery strategy leaves something to be desired, true. It's
certainly a bug. I doubt it makes a difference whether it's labeled as
a security bug or not.

I don't want to downplay your frustration, but the pattern is fairly
common: When someone tries to port a new operating system to some
partitioning system, it's not totally unheard of that the new code takes
down (parts of) the sytem beyond the assigned partition.

> How absolutely bizzare. Basically you spend half a million dollars on
> Sun hardware, and it isn't required to do this better than VMWare?

I think you've got it exactly backwards: you don't let non-trusted
people run code on these machines because they are so expensive.

> If an OS running inside VMWare was able to cause a situation making it
> neccessary to reboot the host environment and restart all VMWare
> instances, it would be considered a very serious and significant
> security problem for VMWare.

Are you sure about this? Separation of virtual machines as if they were
real machines is not listed in the data sheet, and is not covered in the
security-related part of their website, either.

I'm sure they will fix bugs within their responsiblity, but as a
software vendor, they can only do so much about certain types of crasher
bugs. I may have missed some promises, but I doubt they make any hard
guarantees (like money-back if lack of separation is demonstrated).

[Brett Lymn]

On Wed, Sep 10, 2008 at 09:01:05PM +0200, Florian Weimer wrote:
>
> > How absolutely bizzare. Basically you spend half a million dollars on
> > Sun hardware, and it isn't required to do this better than VMWare?
>
> I think you've got it exactly backwards: you don't let non-trusted
> people run code on these machines because they are so expensive.
>

Right, and even if you are forced to allow root access to someone who
is not well trusted then run them in a zone on the hardware domain -
that way they cannot load random kernel modules even if they have root
in the zone.

The bug is bad but there are workarounds available that make it very
difficult to exploit.

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited. If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility. It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."

[Curtis Maloney]

Florian Weimer wrote:
> * Theo de Raadt:
>> Management eventually has to decide to impact the SLA's of all domains.
>> That means that Sun's promise of isolation is bunk.
>
> I don't want to downplay your frustration, but the pattern is fairly
> common: When someone tries to port a new operating system to some
> partitioning system, it's not totally unheard of that the new code takes
> down (parts of) the sytem beyond the assigned partition.

Something that seems to be being missed is that it doesn't matter HOW the
problem was found, NO domain usage (be it application or OS ) should be able
to impact the uptime of the other partitions.

>> How absolutely bizzare. Basically you spend half a million dollars on
>> Sun hardware, and it isn't required to do this better than VMWare?
>
> I think you've got it exactly backwards: you don't let non-trusted
> people run code on these machines because they are so expensive.

The point of the partitioning is that you can isolate semi-trusted usage so
it won't cause downtime for other semi-trusted users. The fact it
(a) marks the hardware as unusable
(b) doesn't let the system admin override/correct that
(c) requires _all_ hardware to be power cycled, not just the effected parts
is the problem. If (b) or (c) were fixed, I think the issue wouldn't be.

--
Curtis Maloney
cmaloney@cardgate.net

[Brett Lymn]

On Sun, Sep 28, 2008 at 08:14:16AM -0600, Theo de Raadt wrote:
>
> the only workaround is to buy a seperate machine for the other uses.
>

No.

> So you buy a machine that can be split up into different machines, and
> guess what, you still have to buy extra ones because it doesn't
> work.
>

Unless you misstated the exploit vector in your original message then
locking untrusted people into a zone will work. I can't help it if
you don't understand the implications of doing this.

> You really do hold vendors to a really really low standard.

Not really - what I am not doing is trying to beat up a firmware
problem that whilst being quite bad can be mitigated by using native
features of Solaris. Too bad if OpenBSD cannot do the same - I am not
really sure about the benefits of OpenBSD on that scale of hardware
anyway considering the lack of kernel threading and the parlous state
of userland threading.

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited. If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility. It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."

[Brett Lymn]

On Sun, Sep 28, 2008 at 07:53:11PM -0600, Theo de Raadt wrote:
>
> Apparently you just plain can't understand simple english.
>

and apparently you cannot read the whole message - I said "too bad if
OpenBSD cannot do this"...

> If you put someone running OpenBSD into a zone, and that zone locks up
> completely and cannot be reset because of a flaw Sun has now admitted,
> then if you NEED that zone back, you have to power the machine down.
>

are you talking hardware zone or a Solaris zone? You are being sloppy
with your terminology.

> If you don't understand that, you must be really really stupid.
>

Here we go again - any time anyone disagrees with you they are too
stupid to see. Sorry, I don't buy it. You are just trying to beat
something up.

>
> You want to talk about trust? The entire idea is that you could
> TRUST the zones to do their job.
>

Do you have any evidence to the contrary that a solaris zone cannot
prevent random kernel modules being loaded? If you don't then you are
just spreading FUD.

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited. If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility. It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."

[Brett Lymn]

On Sun, Sep 28, 2008 at 08:14:35PM -0600, Theo de Raadt wrote:
>
> OpenBSD of course cannot run in a Solaris zone.
>

Right. Glad that is clear.

> OpenBSD can run in a hardware zone, and when something it does (which
> we don't know yet) locks up that hardware zone, the only way to get
> the hardware zone back is to POWER THE MACHINE OFF. That is a lack
> of hardware zoning, or isolation. That is not what people paid a lot
> of money for.
>

Yes, we all agree that is bad but this is an OpenBSD specific problem
and, whilst interesting, the reality is that there are no going to be
many people that are lunatic enough to run an untrusted third party
operating system on a machine of this class.

> Those customers really expected that the machine would not need a
> powerdown to get around a bug in hardware zones.
>

Yes, no arguing with that.

>
> Noone is talking about Solaris zones except you.
>

I suggest you go read the archives again then - I was not the only one.

>
> Why don't we wait for Sun to release the fix, and then you can eat
> your words.

I can handle being wrong. Can you? As I said before, if you know of
a problem with solaris zones then that makes things a lot more of a
problem but all you have at the moment is a firmware bug which
requires loading a random kernel module - something that can be
controlled in Solaris.

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited. If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility. It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."

[Brett Lymn]

On Sun, Sep 28, 2008 at 08:36:17PM -0600, Theo de Raadt wrote:
>
> Oh I get it.
>

No you don't.

> You can use a "trust relationship with your
> administrators" to get around the fact that Sun sold a piece of
> hardware which does not provide the isolation they promised in their
> white papers and documentation.
>

It is a bug. What you seem to be unable to grasp is that you can
configure the officially supported operating system to mitigate the
effects of that bug. I guess you don't want to accept that because it
diminishes the size of the problem.

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited. If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility. It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."

[Brett Lymn]

On Sun, Sep 28, 2008 at 08:53:10PM -0600, Theo de Raadt wrote:
>
> Oh you can avoid the problem by using only the vendor recommended
> configurations!
>

Yes.

> Or so you think. A Solaris kernel module could trigger exactly the
> same bug.
>

Uh duh. You need to read a bit closer - you realise that in a solaris
zone you cannot load a kernel module? Probably not or you would not
be babbling like this.

--
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited. If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility. It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."

[Theo de Raadt]

> > > How absolutely bizzare. Basically you spend half a million dollars on
> > > Sun hardware, and it isn't required to do this better than VMWare?
> >
> > I think you've got it exactly backwards: you don't let non-trusted
> > people run code on these machines because they are so expensive.
> >
>
> Right, and even if you are forced to allow root access to someone who
> is not well trusted then run them in a zone on the hardware domain -
> that way they cannot load random kernel modules even if they have root
> in the zone.
>
> The bug is bad but there are workarounds available that make it very
> difficult to exploit.

the only workaround is to buy a seperate machine for the other uses.

So you buy a machine that can be split up into different machines, and
guess what, you still have to buy extra ones because it doesn't
work.

You really do hold vendors to a really really low standard.

[Bob Beck]

> Not really - what I am not doing is trying to beat up a firmware
> problem that whilst being quite bad can be mitigated by using native
> features of Solaris. Too bad if OpenBSD cannot do the same - I am not
> really sure about the benefits of OpenBSD on that scale of hardware
> anyway considering the lack of kernel threading and the parlous state
> of userland threading.

I don't think you get it. OpenBSD doesn't care a whit about
this. They stumbled upon it as the result of bringing up OpenBSD on
such a machine. No - currently I wouldn't run OpenBSD on an M-class
box either, other than for development purposes. but that's not really
the point is it. Nobody except you is saying this problem has anything
to do with running OpenBSD on a machine.

The point is anyone with a black hat with sufficient clue enough to
ignore this sort of ass-covering nonsense and write a kernel module,
and go look at what the OpenBSD kernel *does* to wedge the zone, and
make a solaris kernel module that does the same. At which point, at a
minimum, the same wedging becomes possible from solaris, so yes, this
is breaking separation.

You're saying "well golly gee, but it's still separation if you don't
let the attacker load kernel modules." good on you. have fun with your
attacker, may you meet one of competence level greater than a script
kiddie someday. I have, they're nice guys. and smart. smarter than me
in a lot of things :) Personally if I'm buying gear to drink the whole
virtualization kool-aid - the kool-aid has to work - meaning stuff done
in the guest OS should never be able to do stuff like this.

-Bob