Tag Archive - cloudmaster

Surviving DDOS – discussion on building resilient networks/data infrastructure.

Note: This is a selection of very early draft of a document I’m writing – As such, those are extract of “working notes” and should be considered as beta (not Google definition of beta ; true beta)… lots will change.

[...]
Internet being a jungle (or a city, whatever you find most dangerous), your infrastructure will be preyed upon. it can be by customers requiring services (too much of them can create difficult situations) or by malevolent individuals wanting to see your service off Internet.
[...]
Of the techniques available, dos/ddos might be the worst. Here’s a quick non technical theory review:

DOS: Denial of services
For a single attacker, cutting access to your services can be accomplished by solving this equation:
Attacker resource * resource(attack function) > Defender resource * resource(defense function)
The defense against the attack is simply the reverse of the equation. Using decent servers (for processing power) in a decent datacenter (for bandwidth) can help solve this equation to the defender advantage without having to modify services. If it doesn’t work, modifying the defense function (such as implementing a firewall correlating a source IP and the attacker function) will allow required resources for defense to be minimal and thus win the fight.
[...]

DDOS: Distributed Denial Of Services
The DDOS add the dimension of multiple (in the order of hundreds or thousands) attackers systems. This will bypass of most of the standard defenses “resource reduction function” since the resulting traffic will be tangent to a normal usage pattern. Randomly blocking visitor (or user) cannot be accomplished without risking blocking valid one and user pattern analysis is generally resource intensive.
[...]

How to survive DDOS
A lot of services and devices are available to mitigate the attack of a DDOS. Some can be implemented by the end user (server administrator) or by the upstream provider. However, most of them must be deployed as a planned feature, not while the network is under attack.
* drop spoofed/invalid packets at upstream provider (packets with invalid source IP (see RFC 1918), implement ingress filtering (see RFC 2267)) – it is also call dark address filtering.
* prepare rate-limiting function ‘per-vhost’ (if service = webpage), or ‘per-services’, and ‘per-source’.
* implement black hole filtering procedure (an in-line router / packet analyzer able to black hole packet will leave your server doing service computing, not routing).
* request analysis. SNORT is a well know and very good ingress filtering agent that can be used to filter traffic that does not match normal usage pattern.
* enable syn cookie (valid only against syn flood).
* always allow establish connections priority over new ones.
* off load as much as you can (mainly: DNS services in separate network, dropping both is harder).

And I’ll allow a bit of additional informations on this last one, because it is often overlooked and can represent your salvation when you are attacked. Either the attacker will use a specific IP, which is easy to mitigate by changing to any other you reserved for that and changing the DNS (5 minutes downtime is nothing in a major DDOS) OR the attacker is resolving your domain name through your DNS. This latest fact is quite important, because it mean the attack can be mitigated by using geo-localisation on your DNS system : different servers will answers requests from different part of the world. MaxMIND does offer a very up-to-date database of IP/Country and IP/Town ; and using Amazon AWS (cloud computing service by Amazon), new servers can be launched at minutes notice and your DNS (when properly configured) can be modified to provide specific IP “to-peoples-outside-your-normal-business-area”. You don’t even have to involve your upstream provider and you will be able to offset a very big part of the attack (as long as your normal business area is not russia + china).

Or, if implementing those recommendation are not a possibility, there is always services/devices available for sales. Be ready to pay a very big price for them.
[...]

ec2 arrive en europe

Tel que prévu dans les dernières semaines et stipulé dans mon dernier article pour Linux+DVD (qui devrait bientot parraitre), l’infrastructure Amazon EC2 est maintenant disponible en europe.

Cette annonce va assèner un coup dur à tous les autres providers de services de cloud computing qui ne font pas la distinction géographiquement… On parle tout de meme de 50ms de latence de moins si la connection n’a pas a passer par l’atlantique…

Quoi qu’il en soit, je complete présentement un autre article pour une revue qui accompagne un sysadmin dans son premier déploiement dans le réseau d’Amazon. En espérant qu’il arrive assez vite ;-)

Source: Amazon EC2 detail page, Amazon EC2 Description page.

[labsphoenix] GeoIP patch to BIND

Long time since my last technical post on this blog. Today, we will go through the installation of the GeoIP libraries (from MaxMind) feature linked with a very common and well known DNS server (bind9 (from ISC)). Before going any further in the labs, please note that I do not consider the GeoIP patch “feature complete”, since it does cover all all GeoIP type (only countries), does not apply to bind-9.5.x and the install process isn’t “fine-tuned” as it should be.

NOTE: I will be releasing, this week-end, a new, enhanced, patch covering those issues. So, lets go on.

Why this lab ?

Simply because distributed infrastructure are common sight and load-balancing traffic across multiple web load-balancer (yeah, I know, balancing on balancer) can sometime be very tricky. Solution such as round robin DNS is, at most, a “best effort” mechanism. Network architects with valid demographic statistics will be able to offer “nearest server” and enhanced experience through geo-localization. This also allows to create a poor man’s CDN (Content Delivery Network) without having to learn / deploy very complex infrastructures.

Installation

We start by installing MaxMind’s GeoIP libraries. It comes with a free database of ip/countries. We follow by retrieving Bind for ISC’s server and applying the patch to link the 2 together. This is the patch I’ll be enhancing.

#changing to src directory
cd /usr/local/src

#getting geoip libraries
wget http://www.maxmind.com/download/geoip/api/c/GeoIP-1.4.5.tar.gz
tar zxf GeoIP-1.4.5.tar.gz
cd GeoIP-1.4.5

#configure & install of libraries
./configure ; make ; make install

#getting bind-9.4.3
wget http://ftp.isc.org/isc/bind9/9.4.3/bind-9.4.3.tar.gz
tar zxf bind-9.4.3

#getting geodns (geoip binding to dns software)
wget http://www.caraytech.com/geodns/patch.diff

#we patch bind
cd bind-9.4.3
patch -p1 < ../patch.diff


#we configure bind with the new libs. (On one line & this procedure will change with the new patch)
CFLAGS=”-I/usr/local/include” LDFLAGS=”-L/usr/local/lib -lGeoIP” ./configure –prefix=/usr/local/bind

#we compile + install bind.
make ; make install

Configuration
We now have a default installation of a patched BIND9 server & GeoIP libraries. The next step is to create configuration files. I will not be going into the big details here, plenties of how-to are available. The principe of GeoIP is matching-clients through country code, not only IP – this is the value of the patch we applied.

/usr/local/bind/etc/named.conf

options {
directory “/usr/local/bind/var/bind”;
listen-on-v6 { none; };
pid-file “/usr/local/bind/var/run/named/named.pid”;
};

view “us” {
// Match clients from US
match-clients { country_US; };
recursion no;
zone “example.com” {
type master;
file “pri/example-us.db”;
};
zone “.” IN {
type hint;
file “named.ca”;
};
};

view “ca” {
// match from Canada
match-clients { country_CA; };
recursion no;
zone “example.com” {
type master;
file “pri/example-ca.db”;
};
zone “.” IN {
type hint;
file “named.ca”;
};
};

view “other” {
// Match all others
match-clients { any; };
recursion no;
zone “example.com” {
type master;
file “pri/example-other.db”;
};
zone “.” IN {
type hint;
file “named.ca”;
};
};

/usr/local/bind/var/named.ca ; this file can be retrieved from almost anywere. Google it.

And we also need zone definitons:
/usr/local/bind/var/bind/pri/example-us.db
/usr/local/bind/var/bind/pri/example-ca.db
/usr/local/bind/var/bind/pri/example-other.db

Conclusion

At this point, you have a BIND server running on your server with views defined following the dns-client country. Using the following command (since I do not really own example.com) will give different result if you are in the USA or Canada. “dig @air0.labsphoenix.com test.example.com“. Btw: the “dig” command is part of dnsutils package. Have fun!

Cloudbursting

Cloudbursting

Technical term referring to the transition to a dynamic state of a network infrastructure as an events mitigation process. It entail to both asynchronous and synchronous expansion toward or inside a computer cloud. Each such burst is followed by a period of quiescence before the next burst occurs.

Event such as high workload and extreme traffic spike can cause a correctly configured environment to cloudburst.

The term seem to have been used first, in July 2008, by William Fellows (Principal Analyst @ 451group) in a report and pushed forward by Jeffrey Barr (Technology evangelist of Amazon AWS) on a blog in august 2008.

This is my personal definition, yet I think it summarize the process. Anything that needs to be added ?

email servers “in the cloud”

I’ve been asked about the possibility of harnessing the power “of the cloud” in the context of an email server. For the simplicity of this blog post, I’ll assume the definition of “cloud computing” to be equivalent to “Amazon AWS” offer.

When emails goes in
This is the easy part. Receiving email in an EC2 (Elastic Cloud Computing) instance is as easy as receiving it anywhere. You launch 2 instances in different availability zone, grab 2 IP and change your MX records. With the recent availability of EBS (Elastic blocks store), you even have access to permanent storage for email. In hours (big maximum) you have a complete setup supporting fail-over and backup capability (leave your queue/data store on EBS for persistence and snapshot for backup).

Being in a full virtual environment also negate most scaling problems. You dynamically start and stop anti-{spam,virus} scanning instances following the need of your clients and customers. This setup is also very cost-effective: you don’t have to pay for hardware (servers, switches, hard drive..), maintenance, power and all the network management involved in having public infrastructure (bgp, firewall, etc…).You don’t even have to vouch for a long term contract.

For your customer, this represent a very decent offer: speed and latency in the Amazon cloud are very nice – way better than most small technical shop can afford.

Then emails have recipient
Emails are not only coming IN your infrastructure, they – sometime – must be transmitted to other people’s networks. This is where archaic email management style really fail. Emails as a services is a dynasty based on the conception that internet proprieties are big, controllable, static and permanent. This is the exact opposite of what you would get placing an email server inside Amazon Cloud.

You do not control IP space/range – even if, you are leased “1″ IP. This is the big “bug”. You have no idea what peoples do in their instances. Get used to it, your range will be tagged, {grey,black} listed often in dns based blocking list. Very often. White list will refuse your queries, since you cannot vouch for Amazon customer use of the cloud.

Solution, you can still use a smtp server install somewhere else, but… kind of defeat the whole purpose. The financial exercise of fighting dnsbl vs maintaining hardware infrastructure is left to the reader.

Page 1 of 212»