Update: Web browsers - a mini-farce (MSIE gives in)

I've been following this discussion with a complete sense of shock and amazment.
Why are you people making up excuses for crap ass programming? It IS NOT
ACCEPTABLE, and if you keep putting forth the notion that it is, nothing is
going to change. The fact that a browser cannot accomplish its fundamental
purpose without crashing and burning is, well, pathetic. Now, I'm the first to
admit that I'm new and rather inexperienced to this game, BUT, regardless of
that, you don't need an "expert" in automotive technology to tell you that a new
car shouldn't lose all control, and fly off the road when the ground is a bit
slick. Why is that? Well, I imagine they TEST FOR THOSE TYPES OF THINGS. If
these browser manufacturers applied the same BASIC principles, and wrote a
SIMPLE (but excellent) tool like Michael's to test their products before
releasing them, well maybe we wouldn't be here right now.

Making arguments like "oh we can't possibly find every bug, so why try at all"
is just stupid. Yes, that was an exaggeration, but so was the original comment
made. Obviously we are all humans, and the best of us make the stupidest
mistakes, I sure know that I do. However, trying to downplay the significance
of bugs like these by blaming "time contstraints" is a feeble attempt to avoid
the real issue at hand. I don't think anyone is asking for silver bullets here
-more like simple and basic verifications of a product. Other industries are
held up to certain standards, and frankly I think the software industry should
be as well.

* dons flame-retardant suit and boots, and sets hose to full blast *

On Fri, 29 Oct 2004 15:38:51 -0400
Valdis.Kletnieks@vt.edu wrote:

> On Fri, 29 Oct 2004 15:25:26 EDT, David Brodbeck said:
>
> > This suggests that it's reasonable for a program to segfault because the
> > user made a mistake, instead of having some non-fatal form of error
> > handling. I don't think that should be acceptable at all, though I agree
> > it's very common. If I had a dollar for every time I've lost work because a
> > segfault or GPF happened before I saved my document...
>
> The problem is that if you say "it isn't acceptable at all", then you *VERY*
> quickly end up with almost *NO* software that's "acceptable".
>
> All software is buggy. *ALL* of it. Learn to accept it. Making a "this is not
> acceptable" declaration about something that in general is not totally
> preventable is just doomed to failure.
>
> Also, remember that programmer time is *FINITE*. Would you have been willing
> to not have the last 3 "this is indispensable" features in your favorite
> software in order that the programmers track down every single possible
> failure mode? Would you give up 2? 1?
>
> Most software projects could probably fairly easily eliminate (rough guess)
> some 75-90% of the really bonehead errors via better programming methodology
> and automated software testers/verifiers - but after that, it's going to be
> *really* hard to get much further improvement.
>
> There are no silver bullets....
>
>


--
-sean

> -----Original Message-----
> From: Tim Newsham [mailto:newsham@lava.net]

> But lets assume that a good programmer is writing software and
> it comes to his attention that there is a buffer overflow, or
> that user input is not being filtered, or that user input is being
> passed to a printf type function. What happens next? Well, it
> depends on how many bugs there are, how much other work needs
> to be done, and very importantly, what the perceived impact of
> that bug is. You cannot imagine how many times a bug is pointed
> out and the author of the software says "ok, that bug can only
> happen if the user does something stupid, and it is not exploitable.
> Lets defer that one."

This suggests that it's reasonable for a program to segfault because the
user made a mistake, instead of having some non-fatal form of error
handling. I don't think that should be acceptable at all, though I agree
it's very common. If I had a dollar for every time I've lost work because a
segfault or GPF happened before I saved my document...

On Wed, 27 Oct 2004 06:32:07 -0700, you wrote:

<snip>

>I write comms code - client- and server-side middleware. I wouldn't =
dream
>of implementing a protocol with code that didn't sanity-check the data =
it
>gets off the wire. =20

What's your point? That you don't believe IE has any sanity checks?
More likely, this bug lies IN one of the sanity checks.

No system is 100% secure. Fact. Every single software system on the
planet, at least anything complex enough to justify the name "system",
DOES HAVE security flaws. They may not be easy to find, but they're
there. Anyone who believes otherwise either has access to technology
=46AR in advance of the rest of us here on earth, or is a fool.

As proof of this, how about releasing some of *your* code? You
believe that it is possible to write code that is completely secure,
ie completely free of all security holes, both known and unknown, by
strictly adhering to and enforcing every single aspect of the protocol
it is designed to handle. I say that's not true - and I challenge you
to prove me otherwise.

Doesn't have to be a full application; a network protocol handler will
be sufficient; doesn't even need to be from a commercial app. Just
enough to demonstrate that you *are* telling the truth - either your
code will stand up, in which case most developers can learn a few
things by using your code as an example, or a flaw will be found to
prove the case that 100% security can never be acheived, regardless of
how well you believe you have validated your input.

How about it? Prove your case?

Chris

--=20
Chris Paget
ivegotta@tombom.co.uk

It's been interesting to follow the discussion after Michal's original
submission. Those readers who are familiar with the PROTOS style of
testing and the results of that research will know how effective
protocol-based invalid input testing has proven to be in catching at
least a majority of common implementation flaws. The PROTOS approach
calls this "robustness testing" as per the following IEEE definition:

"robustness. The degree to which a system or component can function
correctly in the presence of invalid inputs or stressful environmental
conditions." (IEEE Standard Glossary of Software Engineering
Terminology)

Of course, the idea of automated testing with invalid inputs harkens
back all the way to fuzz testing. Whereas fuzz testing typically
employs completely or very nearly random inputs, protocol-based
robustness testing focuses on modifying otherwise valid protocol data
in ways that have been proven during the last 20-30 years to most
likely trigger flaws in implementations. The common bugs that are
found in this type of testing would inevitably be found later on by
various auditors and guys with interesting-coloured hats, so this just
helps in catching some of these vulns early on.

Needless to say the idea can be applied besides network protocols to
almost any kind of structured data, including file formats. In
addition to HTML, image and audio/video formats lend themselves well
to this kind of testing. File format implementations are still not
usually implemented with malicious attacks in mind, which means that
they are trivial to break in most cases. (As compared with network
protocols, although the boundary between the two has steadily become
hazier in the last 10 years due to obvious reasons.)

Since invalid input space is quite large even for simple protocols and
data formats, automated test generation is a must, as is careful human
design for the initial parameters of the used invalid data.
Experience of past software vulnerabilities and a knowledge of common
programming errors certainly helps in this.

Disclaimer: I work for Codenomicon, which develops and sells test
tools for this kind of testing. Codenomicon is a spin-off from the
PROTOS research group OUSPG, and we sponsor them by supplying them
with our test generation framework.

However, I am not trying to sell anything, simply lauding Michal's
excellent foray into a topic I myself am passionate about. Automated
robustness testing is hardly a panacea, but it does help in catching a
lot of the trivial input-based errors. This is especially valuable if
the testing can be done even before the software is released. Mozilla
developers seemed to "get it", since I understood they very rapidly
integrated these tests to their regular (regression) testing cycle.

--
Heikki Kortti