Attack summary:

Starting on September 2, 2013 around 8AM (EST GMT-5) unknown attackers started a Distributed Denial of Service attack against the DPD cart and checkout domain.    When the attack started we immediately assessed the situation, posted a notice to the DPD admin, released Twitter updates from our official @getdpd account, and made a blog post with updates.

A DDoS attack works by flooding the DPD servers with garbage data from hundreds or thousands of compromised computers around the world.  Because of its distributed nature, it is impossible to simply block the attackers since the attack is quite literally "coming from everywhere."  

Our response:

DPD has systems in place at our datacenter to mitigate such attacks but the scale of this attack caused upstream providers to null route our traffic, effectively disconnecting DPD from the internet.

Once the attack traffic returned to manageable levels, our DDoS response plan was put in to place and the datacenter attack mitigation hardware was activated.  At this point DPD began accepting legitimate traffic again.

This resulted in a temporary loss of service for everyone lasting approximately 4 hours and an extended period of degraded service lasting approximately 18 hours.  Due to the way that attack mitigation hardware works, some vendors and buyers who were on the same networks (ISPs, geographical regions) as the attacking computers were also blocked.  This is unavoidable and by blocking these attacking networks we were able to bring service back online for the majority of the internet.  

As of the time of this email we have employed a cloud based DDoS solution to bring the quality of service back up to normal for everyone.  This cloud based service will provide continuous, ongoing protection to all DPD services going forward.  It is an improvement over the hardware based threat mitigation appliance at the datacenter and will prevent automatic disconnection in the future by arresting DDoS attempts before they reach our datacenter and threaten a disconnect like that which occurred on Sep 2nd.

What this means for you, the vendor:

Due to an large scale attack that exceeded both our and our datacenter's mitigation capabilities your stores experienced a total loss of service on September 2 for a 4 hour period, followed by degraded service where some buyers could not access your store for a period of approximately 18 hours.

During this period, customers may not have been able to purchase products or access their downloads.

Over the next 48 hours we're going to be making changes to the DPD code to accommodate the new attack mitigation systems and strategy going forward.  This may result in several extremely short, temporary outages while we update DNS entries, adjust our codebase, etc.

What we could have done better:

Communication

When the attack started we immediately notified everyone via Twitter, blog, and notice in the DPD admin.   This is our standard procedure for a temporary problem that we expect to resolve quickly.

This turned out to be not so quick.   We did not email vendors right away, instead focusing on restoring service to everyone.   While we did post blog updates and tweets, and prioritized tickets related to the outage over all other support requests, we didn't email.  We screwed up, and you've let us know loud and clear you expected an email.  Honestly, we were pretty frantic there for a while and our entire team pulled a nearly 24 hour shift getting services restored.   We should have sent an email broadcast and we didn't.  

In the future, we're going to email everyone as soon as we assess the problem.

More Robust DDoS Protection

Our DDoS mitigation plan relied on the available mitigation appliances at the datacenter.  These appliances included a Cisco Guard solution and TMS.  Unfortunately, the amount of data exceeded these systems and the carrier  null routed our traffic to protect their network, resulting in our server being knocked off the network for several hours.  

We've employed a cloud based, scalable solution that intercepts DDoS traffic before it reaches our datacenter, hopefully eliminating any service interruptions in the future.  There is no 100% guarantees with attacks of this nature, but we feel this new cloud based solution will be more robust and able to handle issues of this scale moving forward.

The current State of DPD Support:

To put it plainly, we're slammed.  We've had a 300% increase in support issues over the past 24 hours and our team is hammered from working non-stop to mitigate this attack.   You can expect slower support responses over the next 48 hours while we catch up.  Every request is important to us and we're going to get them all done as fast as we can, but please be patient with us!

If you have any questions or comments about this attack and our response, please contact us by submitting a support ticket.  We'll respond to every ticket that is sent in as soon as possible.