Onderwerp: Dangers of discarding duplicated messages

Some people use programs as part of their email delivery that
automatically discard duplicate messages (e.g. sent to two mailing
lists the receiver is both subscribed to) based on their Message-ID.

Currently, someone on linux-kernel automatically sends an email to
everyone who sent an email to linux-kernel with the same Message-ID as
the original email. If this email is faster than the original email
(which happens quite often in this example), a program that
automatically discards duplicate emails based on the message ID discards
the original email.

But even more severe attacks are thinkable:

If you can guess the message ID (since many MUAs have predictable
message IDs), an attacker C could use this to suppress a message from
person A to person B by sending an email with the message ID to person B
before person B gets the email from person A.

An example:

If person A uses a MUA that encodes only the current time in seconds
plus a constant string (e.g. the hostname) in the Message-ID and
person B uses a spam filter after the discarding of the duplicate
messages, attacker C could suppress any message person A would send to
person B between 10 and 11 o'clock today by sending 3600 obvious [1]
spam emails with all possible message IDs before 10 o'clock. Since the
spam filter has catched the malicious emails it's quite possible that
person B will not notice the 3600 emails.

It seems to be required that programs that automatically discard
duplicate messages have to use a checksum over the body and part of the
header of the emails instead of relying on the message ID.

cu
Adrian

[1] obvious for a spam filter

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

Hello Adrian,

Thursday, February 17, 2005, 7:57:01 PM, you wrote:

> It seems to be required that programs that automatically discard
> duplicate messages have to use a checksum over the body and part of the
> header of the emails instead of relying on the message ID.
Very interesting indeed Adrian.
So to sum it all up just for now, the advisories are two.
One for users, one for developers of MUAs.

Users: Beware of the fact that automatic discarding of duplicated messages
may result in you not getting the original mail in case someone exploits
the effect Adrian depicted.

Developers: Consider using checksum of the email messages, not only the
Message-ID to distinguish between duplicated messages.

Am I correct?

Fortunately I was never using this MUA feature - I just take care of the
duppies manually, which is not a big chore for me I must say.


--
Best regards,
Maciej Soltysiak

On Fri, 18 Feb 2005 10:31:50 +0100, Maciej Soltysiak
<maciej@soltysiak.com> wrote:
> Users: Beware of the fact that automatic discarding of duplicated messages
> may result in you not getting the original mail in case someone exploits
> the effect Adrian depicted.
>
> Developers: Consider using checksum of the email messages, not only the
> Message-ID to distinguish between duplicated messages.


This makes me wonder about GMail. GMail does this "feature"
automatically and I don't see how to turn it off. I have never used
this, i have always deleted the dups manually as you said.

Jon

There are lots of other things that may need to be done to handle
dealing with duplicated message-id's as well. Note that the RFC's
mandate that the message-id is unique per message.

A number of IMAP packages use the message-id as a way of keeping
track of the message in the various folders.
Having messages with the message-id breaks the standards, may
cause problems in message handling, etc.

The problem is bigger than just the anti-spam packages out there.

From the original message it was indicated that these many dups
might be picked up by the AV/AS software on the server. If it is,
the blocking of the message should happen before the message-id
gets entered into the users cache of delivered messages. Therefore
the 1st non-spam message should still be able to get through to the
end user.

-_Gene


Maciej Soltysiak made the following keystrokes:
>Hello Adrian,
>
>Thursday, February 17, 2005, 7:57:01 PM, you wrote:
>
>> It seems to be required that programs that automatically discard
>> duplicate messages have to use a checksum over the body and part of the
>> header of the emails instead of relying on the message ID.
>Very interesting indeed Adrian.
>So to sum it all up just for now, the advisories are two.
>One for users, one for developers of MUAs.
>
>Users: Beware of the fact that automatic discarding of duplicated messages
> may result in you not getting the original mail in case someone exploi
> ts
> the effect Adrian depicted.
>
>Developers: Consider using checksum of the email messages, not only the
> Message-ID to distinguish between duplicated messages.
>
>Am I correct?
>
>Fortunately I was never using this MUA feature - I just take care of the
>duppies manually, which is not a big chore for me I must say.
>
>
>--
>Best regards,
>Maciej Soltysiak
>
>
>

On Fri, 18 Feb 2005, Maciej Soltysiak wrote:

> Developers: Consider using checksum of the email messages, not only the
> Message-ID to distinguish between duplicated messages.

We have a product that needs to detect duplicate messages (retransmissions
in the face of temporary SMTP failures.) Obviously, we'd never use
the Message-ID for this. Less obviously, you can't always use a hash
because some mailers actually change the message body on each retransmission!

For example, Lotus Domino appears to store the e-mail in some internal
format, and generates a new MIME message on each retransmission. Each
generated message has different MIME boundary delimiters.

So we have to calculate a "modified" hash and ignore bits of the message
that mutate.

--
David.