ms.smtp.DoS.txt

1999-08-17T00:00:00
ID PACKETSTORM:12135
Type packetstorm
Reporter Packet Storm
Modified 1999-08-17T00:00:00

Description

                                        
                                            `Date: Sun, 14 Mar 1999 20:49:30 -0600  
From: Chris Adams <cadams@RO.COM>  
To: BUGTRAQ@netspace.org  
Subject: Microsoft's SMTP service broken/stupid  
  
Our mail servers came to a screeching halt today thanks to Microsoft.  
Our servers are still running sendmail 8.8 (we've got custom stuff and  
are working on upgrading to 8.9, but it has been slow), so any kind of  
DNS error (like invalid reverse DNS) returns a 4xx error - temporary  
problem. When we get a message like this, the sending site will requeue  
the message and try again in 30 minutes to an hour. After a bit, they  
stop trying. It is not a perfect solution, but it is all that is  
available under sendmail 8.8 (sendmail 8.9 differentiates between  
temporary and permanent DNS errors).  
  
Well, that has been fine, but now Microsoft's SMTP service comes along.  
When it gets that temporary error (for invalid reverse DNS), it tries  
again. Fast. Like, right away, with no delay. This bogs down our  
servers a bit, especially the extra logging load, but eventually they go  
away.  
  
Yesterday, we got hit by four different servers running Microsoft's  
software. One attempted delivery nearly 200,000 times, and the other  
three attempted to 30,000-40,000 times each. This on a server that  
usually sees ~40,000 messages a day. This filled up our logs, bogged  
everything down, and basically killed us.  
  
This is not a configuration issue AFAIK. In the past, I've worked  
through it with one person, and he said he bumped up his retry time to 3  
hours and his server was still attempting multiple deliveries per  
second.  
  
This basically amounts to a denial of service attack by Microsoft's SMTP  
service. Here is the connect string from several of the servers that  
hit us (I've changed the hostname):  
  
220-example.com Microsoft SMTP MAIL ready at Sun, 14 Mar 1999 21:44:02 -0500 Version: 5.5.1877.977.9  
  
I was able to connect to several of the SMTP servers that hit us and  
they are all running this version. Several of them don't accept  
incoming connections (gee thanks - send me junk and don't accept any  
back).  
  
I haven't been able to find anything at Microsoft about this. I would  
think that attempting several outgoing connections per second would tend  
to bog down the NT server as well, so I figured they might have  
mentioned it. Has anybody else seen this?  
--  
Chris Adams - cadams@ro.com  
System Administrator - Renaissance Internet Services  
I don't speak for anybody but myself - that's enough trouble.  
  
-------------------------------------------------------------------------------  
  
Date: Wed, 17 Mar 1999 14:58:43 -0800  
From: "David Lemson (Exchange)" <dlemson@EXCHANGE.MICROSOFT.COM>  
To: BUGTRAQ@netspace.org  
Subject: Re: Microsoft's SMTP service broken/stupid  
  
We have confirmed this as a problem with the Microsoft SMTP Service, as  
shipped in Windows NT 4 Service Pack 4. We are working on a fix and will  
have it tested shortly. We will have a Knowledge Base article created as  
soon as the fix is ready so that people can find the fix when they run into  
the problem. Any Microsoft customer who is hitting this problem (such as  
the people whose servers are connecting to you over and over) may e-mail me  
directly to get set up with the right people to get the fix. The service is  
erroneously not treating the 4xx error as a reason to defer the delivery  
until the next queue run. Instead, it treats it as a very transient error  
and retries immediately.  
  
In your case, there is another (easier) solution for the servers that are  
connecting to you over and over: it sounds like if they were to fix their  
inverse DNS entries, so you didn't give them a temporary error code, the  
mail would succeed. This is not to say that what the SMTP Service is doing  
is right, but there may be another way to solve this particular problem.  
  
Another solution, which you allude to, is for your server to issue a  
permanent (5xx) code to a problem that will not get corrected on its own  
(such as an invalid inverse DNS record).  
  
David Lemson  
Microsoft SMTP Service Program Manager  
dlemson@microsoft.com  
  
-------------------------------------------------------------------------------  
  
Date: Wed, 17 Mar 1999 19:47:52 +1300  
From: Alan Brown <alan@MANAWATU.GEN.NZ>  
To: BUGTRAQ@netspace.org  
Subject: Re: Microsoft's SMTP service broken/stupid  
  
On Sun, 14 Mar 1999, Chris Adams wrote:  
  
> Well, that has been fine, but now Microsoft's SMTP service comes along.  
> When it gets that temporary error (for invalid reverse DNS), it tries  
> again. Fast. Like, right away, with no delay. This bogs down our  
> servers a bit, especially the extra logging load, but eventually they go  
> away.  
>  
> Yesterday, we got hit by four different servers running Microsoft's  
> software. One attempted delivery nearly 200,000 times, and the other  
> three attempted to 30,000-40,000 times each. This on a server that  
> usually sees ~40,000 messages a day. This filled up our logs, bogged  
> everything down, and basically killed us.  
  
What's needed is judicious tuning of these configuration items:  
  
# load average at which we just queue messages  
O QueueLA=8  
  
# load average at which we refuse connections  
O RefuseLA=12  
  
The above two default to 8 and 12  
  
# maximum number of children we allow at one time  
O MaxDaemonChildren=12  
  
# maximum number of new connections per second  
O ConnectionRateThrottle=3  
  
If the above two aren't defined, there are no limits.  
  
Also useful for load control are these two items:  
  
# deliver each queued job in a separate process?  
#O ForkEachJob  
  
# single thread deliveries (requires HostStatusDirectory)?  
#O SingleThreadDelivery  
  
AB  
  
-------------------------------------------------------------------------------  
  
Date: Wed, 17 Mar 1999 17:49:57 -0700  
From: Bob Beck <beck@BOFH.UCS.UALBERTA.CA>  
To: BUGTRAQ@netspace.org  
Subject: Re: Microsoft's SMTP service broken/stupid  
  
"David Lemson (Exchange)" <dlemson@EXCHANGE.MICROSOFT.COM> writes:  
  
  
> connecting to you over and over: it sounds like if they were to fix their  
> inverse DNS entries, so you didn't give them a temporary error code, the  
> mail would succeed. This is not to say that what the SMTP Service is doing  
> is right, but there may be another way to solve this particular problem.  
>  
> Another solution, which you allude to, is for your server to issue a  
> permanent (5xx) code to a problem that will not get corrected on its own  
> (such as an invalid inverse DNS record).  
  
You miss the point - this could simply be that their DNS is  
down or unreachable in a timely manner so the lookup fails. When this  
produces an nonexistent entry it actually *does* often correct itself  
"on it's own" once the DNS server is reachable again. Similarly the  
invalid entry may in fact be corrected before the usual timeout and  
allow the mail to proceed. That's the whole point of returning 4XX.  
  
The only way your "solution" is a solution is to return 5XX  
errors for *ALL* situations that currently return a 4XX, otherwise we  
risk being DOS'ed by a poorly written server that doesn't treat SMTP  
errors right. And this isn't a solution - You lose mail that you  
shouldn't. So now I should lose mail on a transient DNS failure  
because microsoft distrbutes code that doesn't play SMTP nicely? I  
don't think so.  
  
The only solution is to fix the buggy code. and/or block access  
>from sites running buggy code.  
  
-Bob  
  
-------------------------------------------------------------------------------  
  
Date: Wed, 17 Mar 1999 17:39:16 -0800  
From: "David Lemson (Exchange)" <dlemson@EXCHANGE.MICROSOFT.COM>  
To: BUGTRAQ@netspace.org  
Subject: Re: Microsoft's SMTP service broken/stupid  
  
I have gotten a few queries and I realized that I was not completely clear.  
This bug does not, to our knowledge, affect any version of Microsoft  
Exchange's Internet Mail Service. It is solely with the SMTP Service  
component of Internet Information Service (IIS) v4.0.  
  
`