hi there,

what follows is some detail into some of the problems we've seen here
at yahoo over the past several days.

sorry for not getting this information out there sooner -- we needed to be
sure we are well protected first...

we've had at least 4 separate attacks to our globalcenter hosted
network over the past week.  we have fewer details on the earlier
ones, but they all seem to have similarities.  they all seem to at
least have a large distributed smurf component to them.  we haven't
been able to verify anything more than icmp smurf attacks, but as
stated, early attacks had little or no data to base this on.

from what we've heard from other sites, many of the other attacks have
been similar to what we've seen, while some have been completely
different.  e.g. we've heard of a syn-attack that was either single
sourced or at most had a few sources.  one would assume there has been
a fair amount of copycat activity.

the only attack that had a significant impact on us was the highly
publicized one that began monday morning at about 10:30am PST.  at
that time we didn't realize that we had been attacked earlier (on a
smaller scale) and therefore an outside attack wasn't at the top of
our list of troubleshooting.  definitely a surprise attack.

the initial flood of packets, which we later realized was in excess of
1G bits/sec, took down one of our routers.  after the router recovered
we lost all routing to our upstream isp.  having had recent problems
with some of our network gear we were sidetracked for the first hour+
trying to eliminate other possible issues.

we finally had all traffic to us turned off upstream.  this allowed us
to get basic routing back up and then realize that we were under a DoS
attack.  it was somewhat difficult to tell what was going on, but at
the very least we noticed lots of icmp traffic.  it took us and our
isp sometime to stabilize things: basically temporary filtering all
icmp - we were then able to glue things back together.

getting things back up then caused major disruption as pent up real
traffic began to overwhelm the system.  things finally stabilized and
at the time it wasn't clear whether the reason for recovery was due to
the fact that the attack had ceased.  our isp managed to log only a
few packets during this time.  our isp did record that a significant
percentage of their peering circuits were taking part in this attack.
i.e. it was widely distributed.

subsequent attacks have had little to no effect.  of course we were
better prepared and knew what to expect.  but most importantly because
of measures our isp took to limit such attacks.

currently our isp is using CAR to rate limit icmp at various points
within their network.  important things to note is that doing CAR or
filtering on echo request / echo reply is probably not enough.

these icmp types can probably also replace echo packets during an
attack:

router advertisement    information request
router solicitation     information reply
  
timestamp request       address mask request
timestamp reply         address mask reply

while an attacker(s) won't be able to generate these packets with a
large payload (my guess, i don't have time to verify specific icmp
protocol details), these packets can come in at the same high rate as
echo packets would.

these rate limiters did their job nicely on subsequent attacks.  the
most recent one resulted in a manageable 150M bits/sec coming through
to the yahoo site.  our isp was able to log quite a few of these
packets during this bout.  we noticed at this time that most of the
icmp packets were destined to www.yahoo.com servers as well as our
nameservers.  i.e. "well-known" servers were the focus of the attack.

now about 'no ip directed-broadcast' business. it won't help with
DDoS.  the whole point of DDoS is to get by that countermeasure. if
you have for example 2,000+ hosts and you want to make sure you can't
be an amp site, you put 'no ip directed-broadcast' .. well .. if i can
break into one of those 2,000+ hosts, i just ping from within that
network.

in fact, on one of the traces we got, this is exactly what it looks
like.  the destination ip was 255.255.255.255 and the source was us.

the attack was against our routers (spoofed src ip was that of our
router interface). it seem that attacker(s) knew about our topology
and planned this large scale attack in advance. in talking to
different other companies it seem they also were hit "where it hurts"
the most"

it seems that this is definitely a DDoS attack, with attacker(s) been
smart and above your average script kiddie. attacker(s) probably know
both unix and networking (cisco, etc) pretty well and learn about site
topology to find weak spots.

we have a few emails sent to us from people who's sites been used as a
amplifier. our isp probably has better/more logs then we do.
 
if you would like more information or have question about anything
above, please feel free to drop me email: jkb@yahoo-inc.com


-- yan


btw, here is a last minute note from our isp on measures taken:

Currently, we've throttling all forms of ICMP at our borders, so other 
ICMP messages types wouldn't be all that effective either.  (We plan to
modify to permit ttl-exceeded so traceroute still works under an attack).

During the large attack, upstream sources were restricted to a single  
gigabit ethernet denying ICMP and throtting syn-- we recovered routing at
this time, and applied the same filters to all 4 Gigabit interfaces.  

Making the attack a bit harder to diagnose was the loss of OSPF           
adjacencies-- when routing would get hosed, the attack would be sunk at
the hold-down routes on the route reflectors.  Once the router(s) had time
to recover, the attack would resume.