Tag Archives: ipv6

IPv6 for SysAdmins

IPv6 is a new and complicated piece of technology; like any new technology it takes a while to get used to and discover what does and doesn’t really work.

Who am I? I’m Robert Mibus, and I helped roll out many of the IPv6-enabled services at Internode (now part of iiNet). I’m writing this based on my personal experience and opinion, so even though I reference Internode frequently, please don’t consider any of the below official communication from them, OK? :)

What this page is:

  • Systems-focused
  • Some network stuff you need to know
  • What I wish I knew three years ago

What this page is not:

  • Waxing on about infinite address space for the sake of itself
  • Pretty graphs on consumer uptake
  • Spiffy future IPv6-could-do-this explanations

Network Stuff

Connectivity

The first thing about IPv6 is how you’re going to get your connectivity. You have two basic options – obtain it natively (eg. via ethernet or PPP from your ISP or colo provider), or via an IPv6-specific tunnel (eg. from a tunnel provider like HurricaneElectric).

So… Don’t use tunnels; just harass your ISP. IPv6 is critical to future growth on the Internet, and is no longer a “maybe sometime in the future thing”.

Addressing

An IPv4 address is like this: 192.231.203.132.

An IPv6 address is like this: 2001:44b8:0001:0000:0000:0000:0000:0001

In order to make the addresses shorter, you can drop leading zeroes in any group, ie: 2001:44b8:1:0:0:0:0:1

You can also replace one (and *only* one) string of :0:0:0:... (for any number of :0:), with “::“…

2001:44b8:1::1

That’s Internode’s IPv6 DNS resolver address. The IPv4 address is actually longer:

192.231.203.132

It’s worth pointing out that many apps don’t accept IPv6 addresses “as-is”; primarily because adding port identifiers (like :8080) causes ambiguity. In those cases, typically the address has square brackets placed around it, eg. [2001:db8:1::1]:8080

Allocations

An ISP will get a large allocation of IPv6 addresses. Typically this will be a /32 or larger (leaving at least 96 bits for the ISP to play with).

eg.

2001:db8::/32 → ISP

The ISP is likely to hand a customer a block somewhere between a /48 and /64. Internode currently gives out /56 as standard, so a customer’s allocation from an ISP might look like this:

2001:db8:1c7:2d00::/56 → ISP Customer

Individual VLANs are typically cut up as /64s, for reasons we’ll get in to later. A /56 thus gives you 256 fully addressable VLANs. You might carve them up like this:

2001:db8:1c7:2d00::/64 → Servers
2001:db8:1c7:2d01::/64 → Ethernet PCs
2001:db8:1c7:2d02::/64 → Wireless PCs

The first 64 bits is then the “network part” and the second 64 bits is the “host part” of the address.

Allocations – Pitfalls!

Is 2001:db8:1c7:2d::/56

  • 2001:db8:1c7:2d00::/56, or
  • 2001:db8:1c7:002d::/56?

Leading zeroes are removed, so it’s 002d not 2d00. If you’re at all uncertain when subnetting, put the zeroes back in when you’re working it out! A tool like “sipcalc” on Linux can also help with calculations like that.

While I’m on subnetting – only subnet on 4-bit boundaries, please… (each hex character is 4 bits). Why? You know how in IPv4, it’s easy to see if a given IPv4 address falls inside or outside a /24 subnet, but harder on a /28 or /29? Subnetting inside the 4 bits is going to drive you insane with craziness in a similar way.

(ie., don’t go making a /63, /62, or /61 – round up and make it a /60!)

Topological Division

Without trying to contradict the previous section too much, all those extra bits can be useful for route aggregation. If you run a large network with multiple campuses, you can use all the extra bits to create routes that can be easily aggregated together.

Imagine the topology for connecting a large set of schools. You might take a chunk of your address like this – :0000:. That’s really 0000 0000 0000 0000 in binary. You might take a bit for region (East-coast or not?), a couple of bits for sub-region (City, or other major routing point), and a couple for the specific site number.

One thing you should always try to do is keep the top-most bit reserved.

Why?

Because it means that if you’ve divided yourself too far and run out of usable networks, you can make a whole new subdivision scheme because you still have half of our addresses left!

Network Stuff – Going Smaller Than /64

It’s really not going to be worth it. “Saving address space” isn’t the argument it used to be!

Among other things, you won’t be able to use SLAAC (stateless address autoconfiguration).

Chances are, plenty of other things will be awkward – probably the one I’d be most concerned with is that you’re going to be breaking lots of people’s expectations about what a network should look like, which may confuse others looking at your network.

Technically, going smaller than /64 works in some cases – like point-to-point links – and may even have some benefits in those edge cases. Feel free to do it if there’s a really really really good reason.

But allocate the /64, at least notionally – That way you can still change your mind later!

You may end up being given something smaller than a /64 in cases where you might only normally get a single IP – eg. in a cheap VPS. Take what you can get :)

Hardware Support

For non-enterprise-grade routers, you can check places like Whirlpool. You can also take a look at Internode’s router offerings – all should now support native IPv6.

Be warned though; some hardware might “support” IPv6, but do so in a limited fashion – it might be software-routing rather than hardware, or it might not expose all of the same statistics via SNMP for IPv6, etc. As always, it’s best to test hardware before making a big purchase.

ICMP Firewalling

Don’t. It wasn’t a great idea for IPv4, and IPv6 relies even heavier on ICMP control messages.

There’s RFC4890 to cover precisely how you can filter ICMP safely – it’s 38 pages long.

But, seriously, just don’t. (Unless you need to)

Firewalling!

The suckiest part of dealing with IPv6 firewalls right now, is that they’re typically wholly separate from IPv4 firewalls.

Using iptables? You need to use ip6tables too. Configuring it on your CPE? It’s probably not smart enough to have the two really intermingled either.

What this means is that documentation is even more important than ever. Each and every set of firewall entries in both firewalls should be documented in enough detail that you can see clearly that the two sets of firewalls are functionally identical.

One nice benefit of IPv6 when firewalling, is that you can afford to have an IPv6 address per service on a given host. That makes firewalling easier, and can make it easier to migrate the service to another host later. But, if you’re lazy (like me! :), you’ll probably still be happy using just one IP per host. (Virtualisation does mean we get to have a lot of hosts, though…).

ARP! No, Neighbour Discovery…

In IPv4, “ARP” is used to discover the MAC address of a host you want to talk to. In IPv6, the comparable protocol is Neighbor Discovery (ND) – which uses ICMPv6. The response comes in the form of a Neighbor Advertisment, also ICMPv6.

With ARP, your request is broadcast across the whole VLAN. ND uses link-local multicast to target just a subset of hosts – typically a small enough subset that it is only hitting the one host! (For the technically-inclined, the multicast packet is sent to all hosts that share the last 24 bits of their IPv6 address with the address being looked for).

Like with ARP, this isn’t routed traffic, it’s specific to the layer-2 network that the host is physically connected to.

Broadcast Ping

Much like ARP’s replacement, the replacement for a broadcast ping uses local network multicast too.

All nodes on a network will join a special multicast group with an address of ff02::1. Ping that instead of an address like 192.168.1.255.

There’s a similar one that is only joined by routers (rather than hosts), ff02::2.

Duplicate Address Detection

An in-built part of IPv6 is “Duplicate Address Detection”, or DAD.

In short, when an IP address is brought up on an interface, it uses Neighbour Discovery to make sure it’s the only node with that address. Due to this, freshly-added addresses on an interface will show as “tentative”, and not actually be usable.

This includes all configured IPv6 addresses during boot, so applications need to cope with this if they try to bind to a specific address on startup. Sometimes they don’t cope, and just bomb out with an error that they couldn’t bind to the IP address as it’s not configured. (ISC BIND is an example of this).

Router Advertisements

Similar to a Neighbor Advertisement, a Router Advertisement is a network router saying “Hi, I’m here!”. This can be used by hosts to auto-configure themselves with an IPv6 address, and set a default route for IPv6. (This is called SLAAC – StateLess Address AutoConfiguration).

An advertisement like that includes a “preferred” and “valid” lifetimes – how long they should be used for new connections, and how long they should be allowed for existing connections, respectively. Routers can re-send the advertisements and refresh those times, kinda like DHCP.

It’s not, however, a full DHCP replacement – it doesn’t have acknowledgements, logging, DDNS, or any number of other fancy features.

For raw usefulness getting a set of desktops on an existing network to use IPv6… priceless.

Neighbor Discovery Security

ARP isn’t really secure by design, so it should be no surprise that Neighbor Discovery has similar sorts of issues possible. If you’re the particularly security-conscious sort, you should check out “Secure Neighbor Discovery” (SEND) or “RA-Guard”.

Unique Local Addresses (ULA)

Roughly equivalent to RFC1918 space, but (probably) still unique to a single environment.

It’s really not needed in most cases – you’re normally expected to just use addresses from your service provider.

Prefix Delegation (PD)

This is the means by which large blocks of address space (prefixes!) can be requested by clients. It’s actually a function of DHCPv6!

Prefix Delegation is how your ADSL modem can request a chunk of address-space from your ISP, and get back a /56 (or whatever is configured for your account) to further subdivide.

Hosts

Desktops

If you want to get a set of desktop PCs on a LAN using IPv6, you’re in luck.

Once your router has an IPv6 range allocated to it (by Prefix Delegation or some other means), then it can send on Router Advertisements to local LAN segments; PCs and other devices will use SLAAC to autoconfigure themselves and use the connectivity.

DHCP (via IPv4) still reigns supreme – handing out DNS and the like – but you nearly instantly have IPv6 connectivity on any modern system.

It’s worth noting that most current-generation OSes will actually end up randomly creating and cycling through IPv6 addresses on a regular basis; this is due to “Privacy Extensions”, an extension of SLAAC’s normally determininstic address selection that uses random numbers instead.

Desktops, the future!

DHCPv6 will, one day, be the norm. But not today, so I’ll leave that for another day :)

Servers

SLAAC is a really bad idea for servers, as it means that changing your host machine (or just a NIC) would change it’s IP! Statically configure the addresses instead.

You can still use Router Advertisements for setting a default gateway, if you want – as mentioned above, there are security concerns, but it can still be pretty convenient.

Services

Common Architecture: Reverse Proxy

One very easy way of making a service IPv6-capable, is by sticking some sort of reverse proxy in front of it. IPv4 traffic can flow as per usual – but when you configure an IPv6 address for the service, the address is actually on a dual-stacked box. This box can then redirect traffic back to the IPv4-only host.

For HTTP, you can set the X-Forwarded-For header with the original IPv6 address (since it’ll be lost otherwise). Make sure your backend application knows to trust the header when (and only when) it comes from the proxy, and make sure it can deal with IPv6 addresses properly!

You’ll also want to make sure that any ACLs or other IP-based restrictions (eg. Geo-IP blocking) take into consideration the proxy’s existence. It’s probably good to set a fair amount of logging on the proxy, too, so you can tie things together when troubleshooting later.

Proxies would work fairly well for:

  • HTTP/S
  • IMAP/POP3
  • DNS, NTP

They won’t work for cases where you actually care a lot about the source – IP address reputation checks for SMTP, rate-limits on web pages, IP-blacklists on forum software, etc.

Anything using bidirectional communications is pretty much out too (eg. FTP, backups), and you also have to note that (like NAT) it risks unbalancing load-balancers if you use IP-based sticky sessions.

Common Architecture: IPv4-only Load Balancer

There’s a couple of different ways to work around an IPv4-only load balancer:

  • Make an IPv6 load-balancer in software
  • Go directly to one backend
  • Round-robin to all backends

I actually think the “direct to backend” solutions aren’t too bad… hacky, but survivable. The vast majority of applications fall back to IPv4 sufficiently well, don’t forget. Make sure to set a low TTL on the DNS, so you can remove it in event of a failure.

Your app sucks

Let’s face it, most sysadmins have to look after any number of custom in-house applications. They’re probably going to need some love. I can’t help you with that.

There’s also amusing side-effects; several software products still truncate IPv6 addresses like “last” outputs in some *nix-based systems, or in logs in database tables. Speaking of databases, what sort of field type are your applications using to store IP addresses? Will it fit an IPv6 address in it? Is it just a 32-bit int? Many databases now support a native “IP address” type – use it if you can.

Rate-limits

While we’re on a tangent, let’s talk about those rate limits too.

  • Is it useful to rate-limit a single IPv6 IP? (A /128). Probably not, it’s too easy to change.
  • Is it useful to rate-limit a /64? Probably; there’s a good chance the user has multiple /64s at their disposal, but only so many…
  • Is it useful to rate-limit a /56? Useful? It’s not even safe – Internode hand out /56s, other ISPs can (and do!) regularly hand out smaller.

Limit the /64 – it’s not perfect, and it’s still not even entirely “fair” in some edge cases, but it’s the best you’ve got.

DNS

DNS is a super awesome and easy service to enable for IPv6, so it’s a great first choice. The protocol has native fallback to secondary servers in case of failure, for both a client accessing a DNS resolver and for a DNS resolver talking to nameservers.

DNS: Aside

You do also end up having a rather large reverse address space to manage DNS for. A /56 has 4722366482869645213696 addresses!!

The only sane solution (other than having no reverse DNS) is procedural DNS generation. Internode do this for their customers (using the “pymds” open source project), optionally delegating it to nameservers of the customer’s choosing. After that, it’s up to you :)

SMTP MTA

Don’t forget to configure your Reverse DNS for SMTP – Yes, people really still do look at this, even when using IPv6.

Connectivity is key; if your MTA software thinks it has IPv6 connectivity then it really needs to work. Check your fallback to IPv4 when IPv6 connectivity is missing – some software doesn’t!

It’s worth noting that people don’t interactively use their mail servers very often, so there’s no implicit verification of connectivity, or firewall validity. Mistakes can go unnoticed for a very long time!

The other common issue running an IPv6-capable MTA is broken [far-end] DNS; entirely too many systems return SERVFAIL rather than an empty NOERROR response, causing extra retries and often making the sending MTA never fall back to IPv4 at all. There’s no real fix here, this is just a heads-up…

SMTP MX

Running an IPv6 MX means a moderate amount of pain right now (though it’s getting quickly better now that GMail runs an IPv6-capable MX).

Running an IPv6-only MX, with an IPv4-capable secondary MX, is not currently feasible – a small number of real-world MTAs break horribly trying to deliver email to a domain using such a setup.

It might be worthwhile using a dual-stack primary MX, and an IPv4-only secondary MX (even if it’s the same IPv4 address!), to encourage semi-broken MTAs to fall back to the secondary MX (and the IPv4 address it carries) if they don’t fall back properly from IPv6 to IPv4 on the primary MX.

Postfix will only (by default) check five addresses from DNS before stopping the delivery attempt; so if Postfix thinks trying over IPv6 is OK, and you have 5+ IPv6 addresses on your MX, *it will never fall back to IPv4 if something goes wrong*. So, don’t have more than 3-4 IPv6 addresses for your MX.

Finally, you still have to deal with broken far-end DNS, because many modern systems will check if a bounce message could be delivered in principle before accepting a message (by checking the sending domain exists and has a valid MX).

It’s worth doing if you have the patience, and things will only get better, but it’s not necessarily going to be the smoothest ride – use your own judgement.

(Internode have done it and are striving forwards with it, because that’s how we roll. I’d like to think that you’re going to do the same! :)

Monitoring

You need to dual-stack your monitoring system, naturally – all those IPv6 services aren’t really in production if they’re not monitored… but, which IP stack will actually get used to do the checks?

Do all of your hosts support all features on both protocols?

I can’t tell you the solution here, because it’s going to be very situation-specific – but there’s a good chance you’ll have to define a lot of your checks twice (once per protocol).

Don’t forget, if you can’t IPv6-enable your existing host (eg. because it’d suddenly make a lot of legacy checks try to use IPv6 and that won’t work) then you can always proxy checks via dual-stacked host.

Opinions

Lots of IPs…

Since your VLANs now have a very large address space, you can put it to some good use.

For example, if you’re mixing different services in the same /64 network, why not use the first few bits to define what sort of service they are, and create yor ACLs around that?

eg., using /80 ACLs:

An IP could be formatted like so – 2001:db8:n:n:apptype:x:x:id – where “apptype” is one of the following:

  • Infrastructure = 0000
  • Web Server = 1000
  • Mail = 2000
  • SMTP = 2010
  • IMAP = 2020
  • Database = 3000

So you could have a firewall rule allowing access to [2001:db8:n:n:1000::/80]:80 from the Internet, making your website public but nothing else. Conveniently, you can spin up further web servers inside that /80 without needing to alter your firewalls.

Technically, you should always pay attention to bit 7 inside the /64, it should always be zero for statically-assigned addresses as it’s the “Unique vs Local interface” bit, used to determine if a given host-part of the address (the last 64 bits of the IP address) is globally unique, or just locally unique. I don’t think anything really uses this right now, but you’ve been warned :)

Server Numbering

If you decide that the suffix of the address is the host number, that’s not a bad idea. BUT!

If:

  • Fooserver1 → ::1, and
  • Fooserver2 → ::2

Is:

  • Fooserver10 → ::10, or
  • Fooserver10 → ::A?

It’s important to remember that IPv6 addresses are hexadecimal!

You can also embed IPv4 IPs in to IPv6 addresses – a host on 192.231.203.132 could have an IPv6 address of 2001:db8:1234:1234:192:231:203:132 – but, should it?

It’s also tempting to be clever and use port numbers – ::25 for mail servers, ::53 for DNS servers, and ::80 for web sites – but again, should you? That’s hex… for 37, 83, & 128!

Conclusion

IPv6 isn’t really all that hard, just new. The best way to get familiar with it, is just to get on with using it :)

IPv6 is BORING!

Please note that this is all personal opinion, and is not a reflection on the opinion, or policies, of any employer, past, present, future, or comma-less.

What is IPv6?

IPv6 is the next step after our current IPv4 addressing scheme – and not a lot more. Instead of the we-thought-it’d-be-ample 2^32ish addresses, we get the it’ll-be-enough-this-time-for-sure 2^128ish addresses. If nothing else, it’ll take us a long time to run out again!

So how do we, as sysadmins, use it?

Software developers have taken a lot of the fun out of this already. Most software – Apache, BIND, SSH, etc – have already had IPv6 support for a very long time. You can just add an IPv6 address to the “bind” or “listen” address of the vast majority of non-custom-built software and it will Just Work. Plenty of software doesn’t even need that, and will bind to the IPv6 address just as soon as it exists on your system.

If you don’t want to use IPv6′s native autoconfiguration, statically configuring your address and gateway is a simple enough process on any vaguely modern OS. Probably Windows, too ;).

What about gotchas?

There are plenty of places this can fall down… but none that typically hurt too much.

When you add IPv6 support to a server, note that you’re also adding IPv6 support for it to talk to other services. If it makes calls out to another server, it will try to do so over IPv6. So, make sure your IPv6 connectivity is good, and all of the services that your server might connect to work over IPv6 (if they have AAAA DNS records – if not, it will quickly fall back to IPv4 anyway). This is especially true if your servers chatter amongst themselves to share data (like caching resolvers and authoritative nameservers, or webservers and database servers). You’ll also be at the mercy of the IPv6 support of any providers that you consume services from.

Anywhere that an IP address gets treated as “special” and not just an opaque block of text, could cause issues. For most, this won’t be an issue. But, if you’re storing it in a database, make sure you aren’t using a short fixed-length string – IPv4 addresses max out at 15 characters but IPv6 goes to 11. I mean, 39.

Why haven’t you done it yet?

There are plenty of reasons not to have IPv6 enabled your services.

The easiest answer is that everyone will need an IPv4 address to see everyone else’s services, so they’ll already have one to see mine. Right? Realistically, IPv4 will probably continue to be supplied behind a carrier-grade-NAT system (or “something”), and most stuff on the internet works just fine behind NAT anyway. It will be a long time before IPv4 addresses start getting dropped entirely from consumer broadband connections.

Not everyone has an IPv6-capable upstream provider. If you are with a particular hosting provider and they don’t provide IPv6 connectivity, you’re pretty much stuck. Find a new one (and migrate all of your hosts) or just wait it out.

Moving on, and a lot of “miscellaneous” equipment doesn’t support IPv6. An up-front example of this is hardware load-balancers. Months out from the beginning of ‘IPv4 exhaustion’, you can buy new hardware that doesn’t support hardware-accelerated IPv6. Sure, you can work around it, or buy from an alternate vendor – but really, that’s pretty excessive just to support 128-bit addresses, when we all know that everyone will have IPv4 connectivity for years to come. Do your “appliance” boxes and VMs support IPv6?

The same lack of hardware applies in the consumer-space – seen very many DSL modems with native IPv6 lately? The biggest driver there will be gamers and P2P users wanting to restore end-to-end connectivity.

There are also potential security issues – how do you make sure your IPv4 and IPv6 firewalls are equivalent? Typically, they’re treated as independent… changes to one but not the other can lead to breakage or security holes. (Forget to lock off SSH over IPv6 so only your corporate network can see it? Oops! Locked down IPv6 too tight, so upstream service connections over IPv6 suddenly fail? Oops!)

You’re being very negative.

Yes, I am. IPv6 is crucial to the long-term health of the Internet. But its biggest driver (in most industries) is merely that it’s the “new shiny”, and not any real business-case. Even when there is a business case, it’s typically for the network, not for the systems. IPv6 connectivity, *check*. IPv6ified webserver… come back next year. Maybe.

So, what if I do find it fun?

Seek professional help. Or, just admit you’re a geek.