Paper announcement: Is finding security holes a good idea?

[Eric Rescorla]

Bugtraq readers might be interested in this paper:

Is finding security holes a good idea?

Eric Rescorla
RTFM, Inc. <http://www.rtfm.com/>

A large amount of effort is expended every year on finding and patching
security holes. The underlying rationale for this activity is that it
increases welfare by decreasing the number of bugs available for
discovery and exploitation by bad guys, thus reducing the total cost of
intrusions. Given the amount of effort expended, we would expect to see
noticeable results in terms of improved software quality. However, our
investigation does not support a substantial quality improvement--the
data does not allow us to exclude the possibility that the rate of bug
finding in any given piece of software is constant over long periods of
time. If there is little or no quality improvement, then we have no
reason to believe that that the disclosure of bugs reduces the overall
cost of intrusions.

The paper can be downloaded from:
http://www.rtfm.com/bugrate.pdf
http://www.rtfm.com/bugrate.ps

[Oliver Friedrichs]

Eric, some thoughts (this is not an argument for or against
full-disclosure, please take it as constructive criticism):

After reading your paper I agree that the data you used may support your
arguments, however, you missed some important points. You don't take into
account the "type" of vulnerabilities being found in each of the
applications that you've analyzed (you address Severity, but thats a
seperate variable). I would argue that if you did that, you would come to
some different conclusions. I would also argue that the caliber of bugs
being found has increased quite substantially. Easy to find, easy to
exploit vulnerabilities have for the most part been exhausted in
applications that have been "sufficiently" scrutinized. The bugs being
found today (in applications that have had a sufficient amount of
scrutiny) are significantly more complex than a number of years ago (in
the same application). In the early 1990's, you could find a common
buffer overflow (using a blatant strcpy/strcat) in many common core
Internet applications. Today, the vulnerabilities being discovered in
these same applications are more complex off-by-one, signed/unsigned
integer, compilar casting, or byte/character processing problems. Some
good examples of this behaviour are sendmail and BIND. "Type" also infers
the skill level required for a researcher to find a given vulnerability,
due to it's difficulty, with some types being easy to find, while others
extremely difficult.

Entirely new classes of vulnerabilities are rarely discovered (but they
still are). There will only be a finite quantity of these, and once those
have been identified, the bar can't go any higher. Of course there can
only be a finite number of vulnerabilities in any given application as
well.

Another two metrics that you don't measure is the amount of scrutiny that
a particular application has had, or the size of an application. A large
application that has had 30 vulnerabilities found in it by one researcher
over 10 years cannot be compared to a small application that has had 30
vulnerabilities found in it by 200 researchers in 1 year. There were 30
vulnerabilities found in both, but the latter application will have
improved in quality quite significantly, while the former not (assuming
that both applications have the same average number of vulnerabilities per
line of code). Unfortunately scrutiny is likely something that you cannot
measure in many applications.

So, on average I would argue that software quality (in terms of
vulnerabilities being discovered) has improved for a given application
that has, and continues to be, sufficiently scrutinized (not including
substantial updates that introduce new bugs). You simply don't have all
of the data points to prove it, and therefore may be missing important
conclusions. I may be able to take specific applications, where we have
sufficient visibility into scrutiny, size, and type/difficulty of
vulnerabilities, and prove your theory wrong (sendmail is a possibility).

My conclusion isn't based on the numbers, but simply on my experience with
researching vulnerabilities since the early 1990's.

Oliver Friedrichs
Sr. Manager - DeepSight
Symantec, Inc. (650) 381-8045

> Bugtraq readers might be interested in this paper:
>
> Is finding security holes a good idea?
>
> Eric Rescorla
> RTFM, Inc. <http://www.rtfm.com/>
>
> A large amount of effort is expended every year on finding and patching
> security holes. The underlying rationale for this activity is that it
> increases welfare by decreasing the number of bugs available for
> discovery and exploitation by bad guys, thus reducing the total cost of
> intrusions. Given the amount of effort expended, we would expect to see
> noticeable results in terms of improved software quality. However, our
> investigation does not support a substantial quality improvement--the
> data does not allow us to exclude the possibility that the rate of bug
> finding in any given piece of software is constant over long periods of
> time. If there is little or no quality improvement, then we have no
> reason to believe that that the disclosure of bugs reduces the overall
> cost of intrusions.
>
> The paper can be downloaded from:
> http://www.rtfm.com/bugrate.pdf
> http://www.rtfm.com/bugrate.ps
>

[Kurt Seifried]

> Bugtraq readers might be interested in this paper:
>
> Is finding security holes a good idea?
>
> Eric Rescorla
> RTFM, Inc. <http://www.rtfm.com/>
>
> The paper can be downloaded from:
> http://www.rtfm.com/bugrate.pdf
> http://www.rtfm.com/bugrate.ps

This is a very interesting read. However there is one main problem:

It doesn't matter if finding security holes or not is a "good idea" or cost
effective since there are a number of groups for which finding bugs is a
vested interest:

1) "Blackhats" and their sponsors - you want to break into a system, you
need to either find a system with known issues that are unaddressed or find
new issues to exploit. Seeing as how Blackhats are now sometimes in the
employ of spammers and other groups for which the discovery and exploitation
of security flaws directly allows them to make money we have a powerful
group with money and a vested interest in finding flaws and exploiting them.

2) "Penetration testers" and their sponsors - you want to break into a
system, you need to either find a system with known issues that are
unaddressed or find new issues to exploit. Seeing as how Penetration testers
are often hired by companies in order to run assessments for which the
discovery and exploitation of security flaws directly allows them to make
money we have a powerful group with money and a vested interest in finding
flaws and exploiting them.

3) "Security vendors" and their sponsors - you want to sell a third party
product that prevents exploitation of buffer overflows for example there
needs to be a serious and identifiable problem with buffer overflows being
exploited in products and systems people want to secure. Same goes for
firewalls, viruses, etc. Imagine if people stopped writing viruses and
stopped spreading them. Significant amounts of money would be saved in
corporate IT budgets (typically anywhere from $10 to $100 per user for the
software alone).

So with these three large groups (and numerous other classes of people and
organizations with a vested interest in finding flaws) it doesn't matter or
not whether it's a good idea. The simple fact of the matter is that it will
continue.

Kurt Seifried, kurt@seifried.org
A15B BEE5 B391 B9AD B0EF
AEB0 AD63 0B4E AD56 E574
http://seifried.org/security/

[Benjamin Franz]

On Wed, 21 Jan 2004, Eric Rescorla wrote:

> Bugtraq readers might be interested in this paper:
>
> Is finding security holes a good idea?
>
> Eric Rescorla
> RTFM, Inc. <http://www.rtfm.com/>
>
> A large amount of effort is expended every year on finding and patching
> security holes. The underlying rationale for this activity is that it
> increases welfare by decreasing the number of bugs available for
> discovery and exploitation by bad guys, thus reducing the total cost of
> intrusions. Given the amount of effort expended, we would expect to see
> noticeable results in terms of improved software quality. However, our
> investigation does not support a substantial quality improvement--the
> data does not allow us to exclude the possibility that the rate of bug
> finding in any given piece of software is constant over long periods of
> time. If there is little or no quality improvement, then we have no
> reason to believe that that the disclosure of bugs reduces the overall
> cost of intrusions.

It is a very weakly justified assumption that the number of Black Hat
exploitations following Black Hat Disclosure is 'almost certainly less'
than the 'peak rate after (Public) disclosure'. The rest of your paper
rests heavily on that assumption - and it in fact nearly pre-determines
your conclusion.

It only takes _ONE_ more Warhol Worm from non-public security bugs than
publically disclosed ones to invalidate your argument (this is obviously
true in the _converse_ as well). This is because the _COST_ of such a worm
is magnified beyond reason by the network. Each individual worm represents
a large fraction of _all_ such intrusions and their cost. This is NOT a
large statistical universe. Individual events skew the whole dataset. One
major exploit may cost as much as thousands or even millions of smaller
exploits.

A bug allowing complete remote compromise of a box that is merely
_connected_ to the net is of a completely different magnitude than one
that causes Mozilla to mis-display a URL. Treating the two bugs as
comparable is a significant error. It is not justified to believe that
they are _generated_, _discovered_, _exploited_, or _fixed_ at the same
rates or that they have even order of magnitude similiar costs.

There remain too many large unknowns to justify _any_ good conclusions re
bug hunting and disclosure vs the cost of intrusions.

--
Benjamin Franz

On that of which one cannot speak, one must remain silent.
---Wittgenstein

[Oliver Friedrichs]

Eric, some thoughts (this is not an argument for or against
full-disclosure, please take it as constructive criticism):

After reading your paper I agree that the data you used may support your
arguments, however, you missed some important points. You don't take into
account the "type" of vulnerabilities being found in each of the
applications that you've analyzed (you address Severity, but thats a
seperate variable). I would argue that if you did that, you would come to
some different conclusions. I would also argue that the caliber of bugs
being found has increased quite substantially. Easy to find, easy to
exploit vulnerabilities have for the most part been exhausted in
applications that have been "sufficiently" scrutinized. The bugs being
found today (in applications that have had a sufficient amount of
scrutiny) are significantly more complex than a number of years ago (in
the same application). In the early 1990's, you could find a common
buffer overflow (using a blatant strcpy/strcat) in many common core
Internet applications. Today, the vulnerabilities being discovered in
these same applications are more complex off-by-one, signed/unsigned
integer, compilar casting, or byte/character processing problems. Some
good examples of this behaviour are sendmail and BIND. "Type" also infers
the skill level required for a researcher to find a given vulnerability,
due to it's difficulty, with some types being easy to find, while others
extremely difficult.

Entirely new classes of vulnerabilities are rarely discovered (but they
still are). There will only be a finite quantity of these, and once those
have been identified, the bar can't go any higher. Of course there can
only be a finite number of vulnerabilities in any given application as
well.

Another two metrics that you don't measure is the amount of scrutiny that
a particular application has had, or the size of an application. A large
application that has had 30 vulnerabilities found in it by one researcher
over 10 years cannot be compared to a small application that has had 30
vulnerabilities found in it by 200 researchers in 1 year. There were 30
vulnerabilities found in both, but the latter application will have
improved in quality quite significantly, while the former not (assuming
that both applications have the same average number of vulnerabilities per
line of code). Unfortunately scrutiny is likely something that you cannot
measure in many applications.

So, on average I would argue that software quality (in terms of
vulnerabilities being discovered) has improved for a given application
that has, and continues to be, sufficiently scrutinized (not including
substantial updates that introduce new bugs). You simply don't have all
of the data points to prove it, and therefore may be missing important
conclusions. I may be able to take specific applications, where we have
sufficient visibility into scrutiny, size, and type/difficulty of
vulnerabilities, and prove your theory wrong (sendmail is a possibility).

My conclusion isn't based on the numbers, but simply on my experience with
researching vulnerabilities since the early 1990's.

Oliver Friedrichs
Sr. Manager - DeepSight
Symantec, Inc. (650) 381-8045

> Bugtraq readers might be interested in this paper:
>
> Is finding security holes a good idea?
>
> Eric Rescorla
> RTFM, Inc. <http://www.rtfm.com/>
>
> A large amount of effort is expended every year on finding and patching
> security holes. The underlying rationale for this activity is that it
> increases welfare by decreasing the number of bugs available for
> discovery and exploitation by bad guys, thus reducing the total cost of
> intrusions. Given the amount of effort expended, we would expect to see
> noticeable results in terms of improved software quality. However, our
> investigation does not support a substantial quality improvement--the
> data does not allow us to exclude the possibility that the rate of bug
> finding in any given piece of software is constant over long periods of
> time. If there is little or no quality improvement, then we have no
> reason to believe that that the disclosure of bugs reduces the overall
> cost of intrusions.
>
> The paper can be downloaded from:
> http://www.rtfm.com/bugrate.pdf
> http://www.rtfm.com/bugrate.ps
>

[Kurt Seifried]

> Bugtraq readers might be interested in this paper:
>
> Is finding security holes a good idea?
>
> Eric Rescorla
> RTFM, Inc. <http://www.rtfm.com/>
>
> The paper can be downloaded from:
> http://www.rtfm.com/bugrate.pdf
> http://www.rtfm.com/bugrate.ps

This is a very interesting read. However there is one main problem:

It doesn't matter if finding security holes or not is a "good idea" or cost
effective since there are a number of groups for which finding bugs is a
vested interest:

1) "Blackhats" and their sponsors - you want to break into a system, you
need to either find a system with known issues that are unaddressed or find
new issues to exploit. Seeing as how Blackhats are now sometimes in the
employ of spammers and other groups for which the discovery and exploitation
of security flaws directly allows them to make money we have a powerful
group with money and a vested interest in finding flaws and exploiting them.

2) "Penetration testers" and their sponsors - you want to break into a
system, you need to either find a system with known issues that are
unaddressed or find new issues to exploit. Seeing as how Penetration testers
are often hired by companies in order to run assessments for which the
discovery and exploitation of security flaws directly allows them to make
money we have a powerful group with money and a vested interest in finding
flaws and exploiting them.

3) "Security vendors" and their sponsors - you want to sell a third party
product that prevents exploitation of buffer overflows for example there
needs to be a serious and identifiable problem with buffer overflows being
exploited in products and systems people want to secure. Same goes for
firewalls, viruses, etc. Imagine if people stopped writing viruses and
stopped spreading them. Significant amounts of money would be saved in
corporate IT budgets (typically anywhere from $10 to $100 per user for the
software alone).

So with these three large groups (and numerous other classes of people and
organizations with a vested interest in finding flaws) it doesn't matter or
not whether it's a good idea. The simple fact of the matter is that it will
continue.

Kurt Seifried, kurt@seifried.org
A15B BEE5 B391 B9AD B0EF
AEB0 AD63 0B4E AD56 E574
http://seifried.org/security/

[Benjamin Franz]

On Wed, 21 Jan 2004, Eric Rescorla wrote:

> Bugtraq readers might be interested in this paper:
>
> Is finding security holes a good idea?
>
> Eric Rescorla
> RTFM, Inc. <http://www.rtfm.com/>
>
> A large amount of effort is expended every year on finding and patching
> security holes. The underlying rationale for this activity is that it
> increases welfare by decreasing the number of bugs available for
> discovery and exploitation by bad guys, thus reducing the total cost of
> intrusions. Given the amount of effort expended, we would expect to see
> noticeable results in terms of improved software quality. However, our
> investigation does not support a substantial quality improvement--the
> data does not allow us to exclude the possibility that the rate of bug
> finding in any given piece of software is constant over long periods of
> time. If there is little or no quality improvement, then we have no
> reason to believe that that the disclosure of bugs reduces the overall
> cost of intrusions.

It is a very weakly justified assumption that the number of Black Hat
exploitations following Black Hat Disclosure is 'almost certainly less'
than the 'peak rate after (Public) disclosure'. The rest of your paper
rests heavily on that assumption - and it in fact nearly pre-determines
your conclusion.

It only takes _ONE_ more Warhol Worm from non-public security bugs than
publically disclosed ones to invalidate your argument (this is obviously
true in the _converse_ as well). This is because the _COST_ of such a worm
is magnified beyond reason by the network. Each individual worm represents
a large fraction of _all_ such intrusions and their cost. This is NOT a
large statistical universe. Individual events skew the whole dataset. One
major exploit may cost as much as thousands or even millions of smaller
exploits.

A bug allowing complete remote compromise of a box that is merely
_connected_ to the net is of a completely different magnitude than one
that causes Mozilla to mis-display a URL. Treating the two bugs as
comparable is a significant error. It is not justified to believe that
they are _generated_, _discovered_, _exploited_, or _fixed_ at the same
rates or that they have even order of magnitude similiar costs.

There remain too many large unknowns to justify _any_ good conclusions re
bug hunting and disclosure vs the cost of intrusions.

--
Benjamin Franz

On that of which one cannot speak, one must remain silent.
---Wittgenstein

[Daniel Whelan]

I am in a sinking ship. The water flows in at a constant rate and does
not diminish. I begin bailing.

After a little while, I notice that my efforts have had no 'measurable
effect'; the level of water in my ship has not gone down, so I decide to
focus my attention on trimming the sails or 'other' work . . .

Granted, the analogy is not perfect, but it holds some truth.

-----Original Message-----
From: Eric Rescorla [mailto:ekr@rtfm.com]
Sent: Wednesday, January 21, 2004 5:42 PM
To: bugtraq@securityfocus.com
Subject: Paper announcement: Is finding security holes a good idea?


Bugtraq readers might be interested in this paper:

Is finding security holes a good idea?

Eric Rescorla
RTFM, Inc. <http://www.rtfm.com/>

A large amount of effort is expended every year on finding and patching
security holes. The underlying rationale for this activity is that it
increases welfare by decreasing the number of bugs available for
discovery and exploitation by bad guys, thus reducing the total cost of
intrusions. Given the amount of effort expended, we would expect to see
noticeable results in terms of improved software quality. However, our
investigation does not support a substantial quality improvement--the
data does not allow us to exclude the possibility that the rate of bug
finding in any given piece of software is constant over long periods of
time. If there is little or no quality improvement, then we have no
reason to believe that that the disclosure of bugs reduces the overall
cost of intrusions.

The paper can be downloaded from: http://www.rtfm.com/bugrate.pdf
http://www.rtfm.com/bugrate.ps

[Daniel Whelan]

I am in a sinking ship. The water flows in at a constant rate and does
not diminish. I begin bailing.

After a little while, I notice that my efforts have had no 'measurable
effect'; the level of water in my ship has not gone down, so I decide to
focus my attention on trimming the sails or 'other' work . . .

Granted, the analogy is not perfect, but it holds some truth.

-----Original Message-----
From: Eric Rescorla [mailto:ekr@rtfm.com]
Sent: Wednesday, January 21, 2004 5:42 PM
To: bugtraq@securityfocus.com
Subject: Paper announcement: Is finding security holes a good idea?


Bugtraq readers might be interested in this paper:

Is finding security holes a good idea?

Eric Rescorla
RTFM, Inc. <http://www.rtfm.com/>

A large amount of effort is expended every year on finding and patching
security holes. The underlying rationale for this activity is that it
increases welfare by decreasing the number of bugs available for
discovery and exploitation by bad guys, thus reducing the total cost of
intrusions. Given the amount of effort expended, we would expect to see
noticeable results in terms of improved software quality. However, our
investigation does not support a substantial quality improvement--the
data does not allow us to exclude the possibility that the rate of bug
finding in any given piece of software is constant over long periods of
time. If there is little or no quality improvement, then we have no
reason to believe that that the disclosure of bugs reduces the overall
cost of intrusions.

The paper can be downloaded from: http://www.rtfm.com/bugrate.pdf
http://www.rtfm.com/bugrate.ps

[Christopher E. Cramer]

Eric,

I would tend to agree with the other critiques of the paper and would
include one more point. In your analysis of p_r (probability of
rediscovery), you assume its value to be very low, if not zero, for most
bugs (on a side note, vulnerabilities would be a much better term
here). This is derived by noting that the number of bugs is very large
(IEEE studies indicate # of bugs (though not necessarily
vulnerabilities) to be anywhere from 1/10th to 1/100th of the number of
lines of code) and that the number of discovered bugs is low. One thing
that this ignores is that the discovery of vulnerabilities is often
clustered as new classes of security issues are discovered.

For example, over the past two years, we've seen an explosion in the
number of cross site scripting vulnerabilities being reported. This
occurred (and continues) as the community recognized new types of XSS
vulnerabilities and tested these techniques against older software
packages. Another example is the increasing number of "off by one"
reports as researchers (both black and white hatted) search through
applications for code that might be impacted.

Of course, that theory breaks down somewhat when considering the
classics (like buffer overflows). However, since the discovery of some
classes of vulnerabilities are clustered, it might be interesting to
search through ICAT to see if there is a temporal correlation in the
types of bugs discovered and to see if you can factor that into your
paper.

-c

--
Christopher E. Cramer, Ph.D.
University Information Technology Security Officer
Duke University, Office of Information Technology
PGP Public Key: http://www.duke.edu/~cramer/cramer.pgp


On Wed, 2004-01-21 at 18:41, Eric Rescorla wrote:
> Bugtraq readers might be interested in this paper:
>
> Is finding security holes a good idea?
>
> Eric Rescorla
> RTFM, Inc. <http://www.rtfm.com/>
>
> A large amount of effort is expended every year on finding and patching
> security holes. The underlying rationale for this activity is that it
> increases welfare by decreasing the number of bugs available for
> discovery and exploitation by bad guys, thus reducing the total cost of
> intrusions. Given the amount of effort expended, we would expect to see
> noticeable results in terms of improved software quality. However, our
> investigation does not support a substantial quality improvement--the
> data does not allow us to exclude the possibility that the rate of bug
> finding in any given piece of software is constant over long periods of
> time. If there is little or no quality improvement, then we have no
> reason to believe that that the disclosure of bugs reduces the overall
> cost of intrusions.
>
> The paper can be downloaded from:
> http://www.rtfm.com/bugrate.pdf
> http://www.rtfm.com/bugrate.ps

[Christopher E. Cramer]

Eric,

I would tend to agree with the other critiques of the paper and would
include one more point. In your analysis of p_r (probability of
rediscovery), you assume its value to be very low, if not zero, for most
bugs (on a side note, vulnerabilities would be a much better term
here). This is derived by noting that the number of bugs is very large
(IEEE studies indicate # of bugs (though not necessarily
vulnerabilities) to be anywhere from 1/10th to 1/100th of the number of
lines of code) and that the number of discovered bugs is low. One thing
that this ignores is that the discovery of vulnerabilities is often
clustered as new classes of security issues are discovered.

For example, over the past two years, we've seen an explosion in the
number of cross site scripting vulnerabilities being reported. This
occurred (and continues) as the community recognized new types of XSS
vulnerabilities and tested these techniques against older software
packages. Another example is the increasing number of "off by one"
reports as researchers (both black and white hatted) search through
applications for code that might be impacted.

Of course, that theory breaks down somewhat when considering the
classics (like buffer overflows). However, since the discovery of some
classes of vulnerabilities are clustered, it might be interesting to
search through ICAT to see if there is a temporal correlation in the
types of bugs discovered and to see if you can factor that into your
paper.

-c

--
Christopher E. Cramer, Ph.D.
University Information Technology Security Officer
Duke University, Office of Information Technology
PGP Public Key: http://www.duke.edu/~cramer/cramer.pgp


On Wed, 2004-01-21 at 18:41, Eric Rescorla wrote:
> Bugtraq readers might be interested in this paper:
>
> Is finding security holes a good idea?
>
> Eric Rescorla
> RTFM, Inc. <http://www.rtfm.com/>
>
> A large amount of effort is expended every year on finding and patching
> security holes. The underlying rationale for this activity is that it
> increases welfare by decreasing the number of bugs available for
> discovery and exploitation by bad guys, thus reducing the total cost of
> intrusions. Given the amount of effort expended, we would expect to see
> noticeable results in terms of improved software quality. However, our
> investigation does not support a substantial quality improvement--the
> data does not allow us to exclude the possibility that the rate of bug
> finding in any given piece of software is constant over long periods of
> time. If there is little or no quality improvement, then we have no
> reason to believe that that the disclosure of bugs reduces the overall
> cost of intrusions.
>
> The paper can be downloaded from:
> http://www.rtfm.com/bugrate.pdf
> http://www.rtfm.com/bugrate.ps

[Robert Lemos]

On Wed, 2004-01-21 at 18:41, Eric Rescorla wrote:
> However, our
> investigation does not support a substantial quality improvement--the
> data does not allow us to exclude the possibility that the rate of bug
> finding in any given piece of software is constant over long periods of
> time. If there is little or no quality improvement, then we have no
> reason to believe that that the disclosure of bugs reduces the overall
> cost of intrusions.

While I have argued against using CERT numbers as an indicator of
security trends, I would point out that the number of vulnerabilities
logged by the organization has (it looks like) fallen this year from the
2002 (based on three quarters of data).

It is way too early to tell what this means, if anything. But assuming
that a variety of variables are held constant (such as CERT's threshold
for what constitutes a vuln, the number of apps audited by
researchers/crackers, the amount of new code generated by developers
this past year, etc.) then this could indicate that software quality has
improved.

Granted, it also likely means that less vulnerabilities have been found
by current low- to medium-level of security auditing (i.e., the "bar"
has been raised and the low-hanging fruit is all being picked).
Researchers who look at more obscure flaws (Oulu University's
investigation of ASN.1 issues is a good example), still seem to be
finding enough of them.

-R

--
| robert lemos | senior staff writer | cnet news.com |
| v: (415) 344-2000 | e: rob.lemos@cnet.com |

[Robert Lemos]

On Wed, 2004-01-21 at 18:41, Eric Rescorla wrote:
> However, our
> investigation does not support a substantial quality improvement--the
> data does not allow us to exclude the possibility that the rate of bug
> finding in any given piece of software is constant over long periods of
> time. If there is little or no quality improvement, then we have no
> reason to believe that that the disclosure of bugs reduces the overall
> cost of intrusions.

While I have argued against using CERT numbers as an indicator of
security trends, I would point out that the number of vulnerabilities
logged by the organization has (it looks like) fallen this year from the
2002 (based on three quarters of data).

It is way too early to tell what this means, if anything. But assuming
that a variety of variables are held constant (such as CERT's threshold
for what constitutes a vuln, the number of apps audited by
researchers/crackers, the amount of new code generated by developers
this past year, etc.) then this could indicate that software quality has
improved.

Granted, it also likely means that less vulnerabilities have been found
by current low- to medium-level of security auditing (i.e., the "bar"
has been raised and the low-hanging fruit is all being picked).
Researchers who look at more obscure flaws (Oulu University's
investigation of ASN.1 issues is a good example), still seem to be
finding enough of them.

-R

--
| robert lemos | senior staff writer | cnet news.com |
| v: (415) 344-2000 | e: rob.lemos@cnet.com |