Building Dreamhack, part two

The next generation network or IPv6

Why change to IPv6
Every device connected to the internet need to have a IP address to be able to communicate.
Today on internet the main OSI layer 3 (network layer) protocol used is IPv4. The IPv4 addresses is 32-bit, in total that makes 4294967296 addresses. The main problem with IPv4 is that it is running out of addresses. By using NAT (Network Address Translation) the problem has been managed.

But the more internet grows, the more you need a long term solution. The long term solution is IPv6. The length of an IPv6 address is 128 bits, in total that makes 4,8 × 10^28 addresses, which is a gigantic amount. One other advantage over IPv4 is that IPv6 has the multicast specification. Multicast is the transmission of a packet to multiple destinations in a single send operation.

Configuring IPv6 address
There are three ways to set a IPv6 address:
-Manual
-DHCPv6
-SLAAC(Stateless address autoconfiguration)
I will describe those three options in detail below:

Manual
You configure the IPv6 address, netmask, gateway and DNS servers manual. This method is preferable on servers and routers where the address needs to be consistent over time.

DHCPv6
The client ask for a IPv6 address, the DHCPv6 server gives the client its IPv6 address, netmask and options like gateway, DNS servers, NTP servers etc. The DHCPv6 server know about which MAC address has what IP and for how long, which means DHCPv6 is statefull.

SLAAC(Stateless address autoconfiguration)
Every IPv6 enabled device has a link-local IPv6 address, this address is used to communicate on the local network. This link-local address is set if the client has IPv6 enabled. The link-local address always begin with FE80:: and is calculated on the client. When the client connects to a network physically, the client will use its link-local address as a source and send a router solicitation message to a specific multicast group on the local network. Every router on the local network will listen to this specific multicast group and answer the clients with a RA (Router Advertisement) message. In the RA there is a prefix specified, the client will use this prefix to calculate its own IPv6 address. The client will use the routers link-local address as the default gateway.

In the initial IPv6 specification there where no way of setting the DNS server using the SLAAC method. Instead there are a M-FLAG in the RA package specifying a DHCPv6 server that the client can connect to get options like DNS, NTP etc. Lately support for setting DNS server using the SLAAC method has been developed, but it is not supported by all operating systems.

IPv6 implementation at Dreamhack
At Dreamhack the client set its IPv6 address, prefix and gateway using the SLAAC method. In the RA there are a DHCPv6 server specified that the client use to set the DNS server.

Example:
1: Client connect to network.
2: Client calculate its own link-local address.
3: Client send out a router solicitation package on the specific multicast group using the link-local address as source.
4: Router listen on the specific multicast group and answer the client with a RA(Router Advertisement) containing the network prefix using its own link-local address as source.
5: The client calculates its own IPv6 address using the prefix in the RA.
6: The client sets the gateway to the routers link-local address.
7: Client ask the DHCPv6 server for the DNS server.
8: DHCPv6 server response with a list of DNS IPv6 servers.
8: Client is IPv6 ready 🙂

Problems with IPv6
Even if your computer, your ISP and you destination is fully IPv6 functional you can have problems with IPv6 because all ISP/routers etc where the IPv6 package travel from, source to destination needs to be IPv6 functional.

Example:
1: Client gets a IPv6 address.
2: Client now has dualstack IPv4 and IPv6 addresses.
3: Client asks the DNS server for the ip of www.youtube.com.
4: DNS server answer with the IPv6 address for www.youtube.com.
5: Client tries to communicate with the IPv6 address of www.youtube.com.
6: If IPv6 is not configured correctly form source to destination the request will timeout when it reaches the none IPv6 ready router/network.
7: Client fallbacks to IPv4.
8: Client is unhappy because request took a long time.

Building Dreamhack, part one

Dreamhack is the world’s largest digital festival and holds the official world record as the world’s largest LAN party in the Guinness Book of World Records. Last event (november 2011) the network had 13 292 uniqe devices connected.

The Dreamhack network team is responsible for planning, building, development, operations and teardown of the network. The team consists of 30 people with a great passion for technology from different companies and universities. The team is divided into four subgroups: core, services, access and logistics. I’m a member of the services group which is responsible for the services required in the network.

Part one: building anycast DNS system supporting IPV4 and IPV6
Anycast is a technology where an (anycast) IP is announced on more than one location using a routing protocol. By doing this the routing protocol thinks that it has multiple routes to the (anycast) IP when in fact there are two different endpoints with the same (anycast) IP. The routing protocol will send the client packages to the endpoint with the shortest path from the client. To achieve high availability you need to be able to remove an endpoint service when errors occur. You can do this by removing the specific route to the broken endpoint from the routing table.

 

In the example image above, the client computer’s request to the anycast IP 9.9.9.9 will be routed to the adns server with the IP 2.2.2.2 because that is the shortest path to the anycast IP 9.9.9.9. If the route saying 9.9.9.9 is reachable via 2.2.2.2 is removed the client’s request will be routed to the server adns with IP 1.1.1.1 instead.

To build our anycast DNS infrastructure at Dreamhack we use Debian GNU/Linux, Bind, iptables, ip6tables and quagga with the routing protocol BGP. We have two anycast DNS servers connected to two different Cisco ASR 9000 routers. On the servers we have loopback interfaces that have the anycast IPV4 and IPV6 address configured. We are then using iptables to forward DNS requests from the interface connected to the routers to the loopback interface. On the servers, bind is handling the DNS requests. To achieve high availability we have built a service which checks if a DNS server is unable to answer 5 different DNS request in a row. If it is, the route to that specific DNS server will be removed from the routing table making all the clients’ DNS request go to the other working DNS server.

Dreamhack anycast DNS design.

During Dreamhack winter 2011, me and my colleague Karl Andersson held a lecture where i discuss the Dreamhack anycast DNS implementation. You can find this presentation on Youtube: Dreamarena Orange – Dreamhack Behind the Scenes.