mibus.org http://www.mibus.org secret society of super sydney sysadmins Wed, 05 Mar 2014 07:16:05 +0000 en-US hourly 1 http://wordpress.org/?v=3.8.2 IPv4 and NAT, the future http://www.mibus.org/2014/03/05/ipv4-and-nat-the-future/ http://www.mibus.org/2014/03/05/ipv4-and-nat-the-future/#comments Wed, 05 Mar 2014 07:11:34 +0000 http://www.mibus.org/?p=1627 A discussion came up during lunch at the IPv6 Roadshow… is NAT robust and well understood?

Since this is kind-of an opinion piece, I’m not going to cite evidence; it’s just random experience and anecdote that (to me) makes a lot of sense.

NAT is robust: NAT, for the sake of NAT – sure, maybe it’s robust (or “robust enough”). But even if you argue that NAT itself is robust, it in turn breaks a host of other things. Off the top of my head, some examples are:

IP Reputation: Any sizable NAT endpoint (like 3G carriers) has a rubbish IP Reputation. Want to send mail directly to the internet, instead your 3G provider’s server? Simply not going to work reliably.

Rate limiting: Any thing that rate-limits by IP (say, to block obvious DoS attacks) is going to get triggered by heavy use from a NAT. Similar for things that limit the number of simultaneous connections by IP.

ALGs: ALGs in CPE are a crufty workaround that just as often as not, break the protocol they’re trying to fix. (Like rewriting SIP packets and messing things up). Likewise

Any P2P technology (Torrent, Skype, etc): If you don’t have a public IP just for yourself, these are either going to not work, or they’re going to increase in latency compared to the ideal. The common ways of making this stuff work (port forwarding, UPnP IGD) just isn’t likely to be supported at the carrier level (IMHO). More dependencies of routing through third-party servers, and more off-net traffic for carriers. There is a thing called PCP that may allow port forwarding in a CGNAT world, but I’m not holding my breath.

Timeouts: If your carrier going forwards needs to maintain a NAT entry for every connection, they’re going to have to time it out at some point. You’re going to be absolutely forced to use keepalives in every protocol that you don’t want summarily disconnected. And I hope those NAT tables are replicated to other gateways for if they fail…

Summary: There are challenges with this stuff in IPv6 – eg. you don’t want each /128 to have its own reputation – but the sort of wide-scale NAT that the future may hold is going to really mess up a lot of things and make them worse… but only over IPv4.

But it’s well understood - Actually, I’d also disagree that NAT is all that “well understood” – amongst sysadmins, do we all understand the different types of NAT (symmetric, full-cone; 1:1/1:N, NAT+PAT vs just NAT, etc) and the implications of each of those on every end-user applications we support?

]]>
http://www.mibus.org/2014/03/05/ipv4-and-nat-the-future/feed/ 0
Moving! http://www.mibus.org/2013/03/27/moving/ http://www.mibus.org/2013/03/27/moving/#comments Wed, 27 Mar 2013 01:15:30 +0000 http://www.mibus.org/?p=1604 I’ve been [comparatively] quiet for a while now, and I can finally announce the results of my endeavours: I’ve resigned from Internode (ie. iiNet) in order to accept a position at Google in Sydney.

This means that Missy and I and our kids are all packing our bags and getting ready to head eastwards, looking forward to new and exciting adventures.

FAQ:

  1. When? My last week in Adelaide is the week of the 22nd of April.
  2. What will you do at Google? I’ll be a Site Reliability Engineer (which is kinda like a sysadmin).
  3. Why leave? Because it’s the right time, and I’m looking for new challenges.
  4. Why Google? It seems like a good fit for interests and for me personally. The folk I know there are generally pretty awesome, and I intend to learn as much as I can from them.

As a final note, I’d like to thank Simon Hackett and Adam Fox for their support and encouragement over the past few years; without them, I don’t think I’d be where I am now.

(FWIW, I started coming up with a list of people I’d like to thank, but it was unmanageably long — “Thank you” to the rest of you too; you should know who you are).

]]>
http://www.mibus.org/2013/03/27/moving/feed/ 0
Making GNOME-Shell plugins save their config http://www.mibus.org/2013/02/15/making-gnome-shell-plugins-save-their-config/ http://www.mibus.org/2013/02/15/making-gnome-shell-plugins-save-their-config/#comments Thu, 14 Feb 2013 21:51:47 +0000 http://www.mibus.org/?p=1593 I’m working on a GNOME-Shell plugin that can show alternate timezones. As part of the plugin, I want it to remember what the user’s selected timezone is.

A good extension to pull apart seems to be the “System Monitor“.

The “short version” of the minimum you need to do is as follows…

Add a schema name to your metadata.json, eg:

"settings-schema": "org.gnome.shell.extensions.system-monitor",

(Replace “system-monitor” with a unique name for your own extension)

Create a schemas directory inside your extension’s directory, and create a file inside that called YOURSCHEMANAME.gschema.xml, eg. org.gnome.shell.extensions.system-monitor.gschema.xml.

Populate this with appropriate gschema magic; the system-monitor schema is pretty handy again here.

Inside your extension’s schemas directory, run:

glib-compile-schemas .

(Don’t miss the “.” :)

Download convenience.js into your extension’s directory.

Inside your extension.js, set up some new imports:

const ExtensionUtils = imports.misc.extensionUtils;
const Me = ExtensionUtils.getCurrentExtension();
const Convenience = Me.imports.convenience;

Somewhere during init() or enable(), you’ll want to grab a reference to your loaded schema object:

this._schema = Convenience.getSettings();

You can now get and set keys, eg. I’m using:

this._schema.get_string('tz'));

and

this._schema.get_string('tz'));

You probably should set up an actual preferences window… you can learn more about that at http://blog.mecheye.net/2012/02/more-extension-api-breaks/.

]]>
http://www.mibus.org/2013/02/15/making-gnome-shell-plugins-save-their-config/feed/ 0
ping -f[aux flood] http://www.mibus.org/2012/12/02/ping-faux-flood/ http://www.mibus.org/2012/12/02/ping-faux-flood/#comments Sun, 02 Dec 2012 06:42:56 +0000 http://www.mibus.org/?p=1588 I’m having trouble admitting this publicly, but I learnt something new recently about ping, of all things.

To quote from the man page of ping:

-f Flood ping. For every ECHO_REQUEST sent a period ``.'' is
 printed, while for ever ECHO_REPLY received a backspace is
 printed. This provides a rapid display of how many packets are
 being dropped.

Seems reasonable enough. I mean, “flood” is pretty clear, right? Except, during troubleshooting this week I found that pinging a responding host and a non-responding host resulted in sending a different rate of packets.

Reading on in the man page, it becomes apparent why:

If interval is not given, it sets interval to
zero and outputs packets as fast as they come back or one hun‐
dred times per second, whichever is more. Only the super-user
may use this option with zero interval.

So what you really want, if you want a consistent number of packets per second, is to use the command like this:

ping -f -i 0.01 somehost.example.net

An interval of zero is actually an interval of “anything from zero to 10 milliseconds, depending on the RTT to the remote host”.

]]>
http://www.mibus.org/2012/12/02/ping-faux-flood/feed/ 0
Multiple SSH port-forwards, all in a row… http://www.mibus.org/2012/11/23/multiple-ssh-port-forwards-all-in-a-row/ http://www.mibus.org/2012/11/23/multiple-ssh-port-forwards-all-in-a-row/#comments Thu, 22 Nov 2012 21:36:31 +0000 http://www.mibus.org/?p=1580 Sometimes, you really need to get from point “A” to point “B”, but you can’t. Restrictive firewalls, poor change control, you don’t own the infrastructure, or maybe it’s just “temporary” and you don’t want to have to go to all that effort just for a few days.

Struggle no more!

SSH Port Forwarding Primer

Let’s just say that you have an application on server “A”, and you want it to be able to reach a web service on your local desktop. You want to open a socket listening on the Remote end, which forwards the connection back over SSH to your local machine. (Maybe a NAT gateway is in the way).

Assuming the server is called “server-a” and your web service is on port 8080, the following may well suffice:

ssh -R8080:localhost:8080 server-a.example.com

“-R” for Remote listening socket, the port number for the designated [remote, in this case] side, then a host and port on the other [local] side.

If you want a local listening socket to forward the packets securely to the remote server – maybe a firewall is in the way this time – just change the Remote for Local:

ssh -L8080:localhost:8080 server-a.example.com

Now, connecting to your desktop’s port 8080 will land on the far-end’s localhost:8080. Again, the first 8080 is for the designated side [local, this time] and the localhost:8080 for the other [remote] side.

Linking port-forwards together

Here’s a hypothetical scenario, where Server D needs to be reached by Server A. For whatever reason, no single host has connectivity the whole distance.

In short, if an SSH port forward opens a remote socket and sends it locally, then that local port can be opened by a different SSH session and forwarded elsewhere too.

You can set this example up thusly:

Diagram of a complex SSH forwarding situation

On the desktop:

ssh -R8080:localhost:8080 server-a.example.com
ssh -L8080:localhost:8080 server-b.example.com

These two together gets Server A connectivity to Server B (via port 8080). Right now, it ends there, because Server B isn’t listening on port 8080.

On Server B:

ssh -L8080:server-d.example.com:8080 server-c.example.com

This time, we don’t want to forward the local socket’s connections to localhost on the remote server, we want to pass it all the way to Server D. What happens is that Server B listens on port 8080, bundles all the data up and over SSH to Server C, then Server C unpacks it all and sends it to the nominated address (server-d.example.com:8080).

Now Server A can point to it’s localhost:8080, and end up on server-d.example.com:8080.

Spiffy, huh!

Please don’t do this in production, though – but if you do, please don’t tell anybody I told you how! :)

You might also be interested in my other SSH port-forwarding hacky trick, where I show how to make a remote server appear local without needing proxy support in your application.

]]>
http://www.mibus.org/2012/11/23/multiple-ssh-port-forwards-all-in-a-row/feed/ 0
GNOME-Shell multi-timezone clock http://www.mibus.org/2012/11/17/gnome-shell-multi-timezone-clock/ http://www.mibus.org/2012/11/17/gnome-shell-multi-timezone-clock/#comments Sat, 17 Nov 2012 04:36:38 +0000 http://www.mibus.org/?p=1562 As many of you know, the company I work for (Internode) was recentlyish purchased by iiNet.

iiNet’s headquarters is in Perth, which is 1.5 or 2.5 hours “behind” Adelaide time.

There’s no longer a multi-timezone clock available for the panel, so… oh well I just wrote one myself.

Screenshot of multi-timezone clock in GNOME-Shell top panel

Right now, it only supports GMT+8 as the “remote” timezone, but it’s easy enough to change the code.

The code is up on GitHub: https://github.com/mibus/MultiClock

Thanks to Marco Dallagiacoma for writing the “Fuzzy Clock” plugin, as I used it as a basis for my own code.

]]>
http://www.mibus.org/2012/11/17/gnome-shell-multi-timezone-clock/feed/ 0
NetWorker – Random stalling http://www.mibus.org/2012/11/11/networker-random-stalling/ http://www.mibus.org/2012/11/11/networker-random-stalling/#comments Sun, 11 Nov 2012 02:42:40 +0000 http://www.mibus.org/?p=1572 One of the things I’ve spent a lot of time with, has been EMC NetWorker (previously Legato NetWorker).

A vaguely common issue is for a process of some kind – backups, staging to tape, restores, etc – for no reason just stop making any new progress.

Once you’ve checked off the common reasons – like making sure you haven’t run out of disk space or usable tapes – it seems like the only option is to restart NetWorker as a whole, losing any in-progress actions (even ones that are to devices that haven’t stalled).

I suspect that random underlying I/O issues can occasionally upset it, and it doesn’t quite recover. But, whatever. How do you make it recover a single device, without restarting the whole thing?

First up, get the PID of the main nsrd process. On Solaris, ps -ef | grep nsrd; or on Linux ps uaxw | grep nsrd.

Assuming the PID is 1234, you next need to run: dbgcommand -p 1234 PrintDevInfo

It should pretty quickly spit out a whole stack of debugging info to /nsr/logs/daemon.raw. It’s moderately complicated, but you should see that it’s a dump of its internal state of each device, including d_device – the *nix device or directory, and mm_number – the unique ID for the nsrmmd process for that device.

So – find the device you’re interested in, and find the mm_number for that device.

Get a list of your nsrmmd processes, eg. ps -ef | nsrmmd or ps auxw | grep nsrmmd. If your mm_number is 5, then there will be a process nsrmmd -n 5

Kill the process, and it should re-spawn by itself on further access.

]]>
http://www.mibus.org/2012/11/11/networker-random-stalling/feed/ 0
IPv6 for SysAdmins http://www.mibus.org/2012/11/08/ipv6-for-sysadmins/ http://www.mibus.org/2012/11/08/ipv6-for-sysadmins/#comments Wed, 07 Nov 2012 13:30:19 +0000 http://www.mibus.org/?p=1554 IPv6 is a new and complicated piece of technology; like any new technology it takes a while to get used to and discover what does and doesn’t really work.

Who am I? I’m Robert Mibus, and I helped roll out many of the IPv6-enabled services at Internode (now part of iiNet). I’m writing this based on my personal experience and opinion, so even though I reference Internode frequently, please don’t consider any of the below official communication from them, OK? :)

What this page is:

  • Systems-focused
  • Some network stuff you need to know
  • What I wish I knew three years ago

What this page is not:

  • Waxing on about infinite address space for the sake of itself
  • Pretty graphs on consumer uptake
  • Spiffy future IPv6-could-do-this explanations

Network Stuff

Connectivity

The first thing about IPv6 is how you’re going to get your connectivity. You have two basic options – obtain it natively (eg. via ethernet or PPP from your ISP or colo provider), or via an IPv6-specific tunnel (eg. from a tunnel provider like HurricaneElectric).

So… Don’t use tunnels; just harass your ISP. IPv6 is critical to future growth on the Internet, and is no longer a “maybe sometime in the future thing”.

Addressing

An IPv4 address is like this: 192.231.203.132.

An IPv6 address is like this: 2001:44b8:0001:0000:0000:0000:0000:0001

In order to make the addresses shorter, you can drop leading zeroes in any group, ie: 2001:44b8:1:0:0:0:0:1

You can also replace one (and *only* one) string of :0:0:0:... (for any number of :0:), with “::“…

2001:44b8:1::1

That’s Internode’s IPv6 DNS resolver address. The IPv4 address is actually longer:

192.231.203.132

It’s worth pointing out that many apps don’t accept IPv6 addresses “as-is”; primarily because adding port identifiers (like :8080) causes ambiguity. In those cases, typically the address has square brackets placed around it, eg. [2001:db8:1::1]:8080

Allocations

An ISP will get a large allocation of IPv6 addresses. Typically this will be a /32 or larger (leaving at least 96 bits for the ISP to play with).

eg.

2001:db8::/32 → ISP

The ISP is likely to hand a customer a block somewhere between a /48 and /64. Internode currently gives out /56 as standard, so a customer’s allocation from an ISP might look like this:

2001:db8:1c7:2d00::/56 → ISP Customer

Individual VLANs are typically cut up as /64s, for reasons we’ll get in to later. A /56 thus gives you 256 fully addressable VLANs. You might carve them up like this:

2001:db8:1c7:2d00::/64 → Servers
2001:db8:1c7:2d01::/64 → Ethernet PCs
2001:db8:1c7:2d02::/64 → Wireless PCs

The first 64 bits is then the “network part” and the second 64 bits is the “host part” of the address.

Allocations – Pitfalls!

Is 2001:db8:1c7:2d::/56

  • 2001:db8:1c7:2d00::/56, or
  • 2001:db8:1c7:002d::/56?

Leading zeroes are removed, so it’s 002d not 2d00. If you’re at all uncertain when subnetting, put the zeroes back in when you’re working it out! A tool like “sipcalc” on Linux can also help with calculations like that.

While I’m on subnetting – only subnet on 4-bit boundaries, please… (each hex character is 4 bits). Why? You know how in IPv4, it’s easy to see if a given IPv4 address falls inside or outside a /24 subnet, but harder on a /28 or /29? Subnetting inside the 4 bits is going to drive you insane with craziness in a similar way.

(ie., don’t go making a /63, /62, or /61 – round up and make it a /60!)

Topological Division

Without trying to contradict the previous section too much, all those extra bits can be useful for route aggregation. If you run a large network with multiple campuses, you can use all the extra bits to create routes that can be easily aggregated together.

Imagine the topology for connecting a large set of schools. You might take a chunk of your address like this – :0000:. That’s really 0000 0000 0000 0000 in binary. You might take a bit for region (East-coast or not?), a couple of bits for sub-region (City, or other major routing point), and a couple for the specific site number.

One thing you should always try to do is keep the top-most bit reserved.

Why?

Because it means that if you’ve divided yourself too far and run out of usable networks, you can make a whole new subdivision scheme because you still have half of our addresses left!

Network Stuff – Going Smaller Than /64

It’s really not going to be worth it. “Saving address space” isn’t the argument it used to be!

Among other things, you won’t be able to use SLAAC (stateless address autoconfiguration).

Chances are, plenty of other things will be awkward – probably the one I’d be most concerned with is that you’re going to be breaking lots of people’s expectations about what a network should look like, which may confuse others looking at your network.

Technically, going smaller than /64 works in some cases – like point-to-point links – and may even have some benefits in those edge cases. Feel free to do it if there’s a really really really good reason.

But allocate the /64, at least notionally – That way you can still change your mind later!

You may end up being given something smaller than a /64 in cases where you might only normally get a single IP – eg. in a cheap VPS. Take what you can get :)

Hardware Support

For non-enterprise-grade routers, you can check places like Whirlpool. You can also take a look at Internode’s router offerings – all should now support native IPv6.

Be warned though; some hardware might “support” IPv6, but do so in a limited fashion – it might be software-routing rather than hardware, or it might not expose all of the same statistics via SNMP for IPv6, etc. As always, it’s best to test hardware before making a big purchase.

ICMP Firewalling

Don’t. It wasn’t a great idea for IPv4, and IPv6 relies even heavier on ICMP control messages.

There’s RFC4890 to cover precisely how you can filter ICMP safely – it’s 38 pages long.

But, seriously, just don’t. (Unless you need to)

Firewalling!

The suckiest part of dealing with IPv6 firewalls right now, is that they’re typically wholly separate from IPv4 firewalls.

Using iptables? You need to use ip6tables too. Configuring it on your CPE? It’s probably not smart enough to have the two really intermingled either.

What this means is that documentation is even more important than ever. Each and every set of firewall entries in both firewalls should be documented in enough detail that you can see clearly that the two sets of firewalls are functionally identical.

One nice benefit of IPv6 when firewalling, is that you can afford to have an IPv6 address per service on a given host. That makes firewalling easier, and can make it easier to migrate the service to another host later. But, if you’re lazy (like me! :), you’ll probably still be happy using just one IP per host. (Virtualisation does mean we get to have a lot of hosts, though…).

ARP! No, Neighbour Discovery…

In IPv4, “ARP” is used to discover the MAC address of a host you want to talk to. In IPv6, the comparable protocol is Neighbor Discovery (ND) – which uses ICMPv6. The response comes in the form of a Neighbor Advertisment, also ICMPv6.

With ARP, your request is broadcast across the whole VLAN. ND uses link-local multicast to target just a subset of hosts – typically a small enough subset that it is only hitting the one host! (For the technically-inclined, the multicast packet is sent to all hosts that share the last 24 bits of their IPv6 address with the address being looked for).

Like with ARP, this isn’t routed traffic, it’s specific to the layer-2 network that the host is physically connected to.

Broadcast Ping

Much like ARP’s replacement, the replacement for a broadcast ping uses local network multicast too.

All nodes on a network will join a special multicast group with an address of ff02::1. Ping that instead of an address like 192.168.1.255.

There’s a similar one that is only joined by routers (rather than hosts), ff02::2.

Duplicate Address Detection

An in-built part of IPv6 is “Duplicate Address Detection”, or DAD.

In short, when an IP address is brought up on an interface, it uses Neighbour Discovery to make sure it’s the only node with that address. Due to this, freshly-added addresses on an interface will show as “tentative”, and not actually be usable.

This includes all configured IPv6 addresses during boot, so applications need to cope with this if they try to bind to a specific address on startup. Sometimes they don’t cope, and just bomb out with an error that they couldn’t bind to the IP address as it’s not configured. (ISC BIND is an example of this).

Router Advertisements

Similar to a Neighbor Advertisement, a Router Advertisement is a network router saying “Hi, I’m here!”. This can be used by hosts to auto-configure themselves with an IPv6 address, and set a default route for IPv6. (This is called SLAAC – StateLess Address AutoConfiguration).

An advertisement like that includes a “preferred” and “valid” lifetimes – how long they should be used for new connections, and how long they should be allowed for existing connections, respectively. Routers can re-send the advertisements and refresh those times, kinda like DHCP.

It’s not, however, a full DHCP replacement – it doesn’t have acknowledgements, logging, DDNS, or any number of other fancy features.

For raw usefulness getting a set of desktops on an existing network to use IPv6… priceless.

Neighbor Discovery Security

ARP isn’t really secure by design, so it should be no surprise that Neighbor Discovery has similar sorts of issues possible. If you’re the particularly security-conscious sort, you should check out “Secure Neighbor Discovery” (SEND) or “RA-Guard”.

Unique Local Addresses (ULA)

Roughly equivalent to RFC1918 space, but (probably) still unique to a single environment.

It’s really not needed in most cases – you’re normally expected to just use addresses from your service provider.

Prefix Delegation (PD)

This is the means by which large blocks of address space (prefixes!) can be requested by clients. It’s actually a function of DHCPv6!

Prefix Delegation is how your ADSL modem can request a chunk of address-space from your ISP, and get back a /56 (or whatever is configured for your account) to further subdivide.

Hosts

Desktops

If you want to get a set of desktop PCs on a LAN using IPv6, you’re in luck.

Once your router has an IPv6 range allocated to it (by Prefix Delegation or some other means), then it can send on Router Advertisements to local LAN segments; PCs and other devices will use SLAAC to autoconfigure themselves and use the connectivity.

DHCP (via IPv4) still reigns supreme – handing out DNS and the like – but you nearly instantly have IPv6 connectivity on any modern system.

It’s worth noting that most current-generation OSes will actually end up randomly creating and cycling through IPv6 addresses on a regular basis; this is due to “Privacy Extensions”, an extension of SLAAC’s normally determininstic address selection that uses random numbers instead.

Desktops, the future!

DHCPv6 will, one day, be the norm. But not today, so I’ll leave that for another day :)

Servers

SLAAC is a really bad idea for servers, as it means that changing your host machine (or just a NIC) would change it’s IP! Statically configure the addresses instead.

You can still use Router Advertisements for setting a default gateway, if you want – as mentioned above, there are security concerns, but it can still be pretty convenient.

Services

Common Architecture: Reverse Proxy

One very easy way of making a service IPv6-capable, is by sticking some sort of reverse proxy in front of it. IPv4 traffic can flow as per usual – but when you configure an IPv6 address for the service, the address is actually on a dual-stacked box. This box can then redirect traffic back to the IPv4-only host.

For HTTP, you can set the X-Forwarded-For header with the original IPv6 address (since it’ll be lost otherwise). Make sure your backend application knows to trust the header when (and only when) it comes from the proxy, and make sure it can deal with IPv6 addresses properly!

You’ll also want to make sure that any ACLs or other IP-based restrictions (eg. Geo-IP blocking) take into consideration the proxy’s existence. It’s probably good to set a fair amount of logging on the proxy, too, so you can tie things together when troubleshooting later.

Proxies would work fairly well for:

  • HTTP/S
  • IMAP/POP3
  • DNS, NTP

They won’t work for cases where you actually care a lot about the source – IP address reputation checks for SMTP, rate-limits on web pages, IP-blacklists on forum software, etc.

Anything using bidirectional communications is pretty much out too (eg. FTP, backups), and you also have to note that (like NAT) it risks unbalancing load-balancers if you use IP-based sticky sessions.

Common Architecture: IPv4-only Load Balancer

There’s a couple of different ways to work around an IPv4-only load balancer:

  • Make an IPv6 load-balancer in software
  • Go directly to one backend
  • Round-robin to all backends

I actually think the “direct to backend” solutions aren’t too bad… hacky, but survivable. The vast majority of applications fall back to IPv4 sufficiently well, don’t forget. Make sure to set a low TTL on the DNS, so you can remove it in event of a failure.

Your app sucks

Let’s face it, most sysadmins have to look after any number of custom in-house applications. They’re probably going to need some love. I can’t help you with that.

There’s also amusing side-effects; several software products still truncate IPv6 addresses like “last” outputs in some *nix-based systems, or in logs in database tables. Speaking of databases, what sort of field type are your applications using to store IP addresses? Will it fit an IPv6 address in it? Is it just a 32-bit int? Many databases now support a native “IP address” type – use it if you can.

Rate-limits

While we’re on a tangent, let’s talk about those rate limits too.

  • Is it useful to rate-limit a single IPv6 IP? (A /128). Probably not, it’s too easy to change.
  • Is it useful to rate-limit a /64? Probably; there’s a good chance the user has multiple /64s at their disposal, but only so many…
  • Is it useful to rate-limit a /56? Useful? It’s not even safe – Internode hand out /56s, other ISPs can (and do!) regularly hand out smaller.

Limit the /64 – it’s not perfect, and it’s still not even entirely “fair” in some edge cases, but it’s the best you’ve got.

DNS

DNS is a super awesome and easy service to enable for IPv6, so it’s a great first choice. The protocol has native fallback to secondary servers in case of failure, for both a client accessing a DNS resolver and for a DNS resolver talking to nameservers.

DNS: Aside

You do also end up having a rather large reverse address space to manage DNS for. A /56 has 4722366482869645213696 addresses!!

The only sane solution (other than having no reverse DNS) is procedural DNS generation. Internode do this for their customers (using the “pymds” open source project), optionally delegating it to nameservers of the customer’s choosing. After that, it’s up to you :)

SMTP MTA

Don’t forget to configure your Reverse DNS for SMTP – Yes, people really still do look at this, even when using IPv6.

Connectivity is key; if your MTA software thinks it has IPv6 connectivity then it really needs to work. Check your fallback to IPv4 when IPv6 connectivity is missing – some software doesn’t!

It’s worth noting that people don’t interactively use their mail servers very often, so there’s no implicit verification of connectivity, or firewall validity. Mistakes can go unnoticed for a very long time!

The other common issue running an IPv6-capable MTA is broken [far-end] DNS; entirely too many systems return SERVFAIL rather than an empty NOERROR response, causing extra retries and often making the sending MTA never fall back to IPv4 at all. There’s no real fix here, this is just a heads-up…

SMTP MX

Running an IPv6 MX means a moderate amount of pain right now (though it’s getting quickly better now that GMail runs an IPv6-capable MX).

Running an IPv6-only MX, with an IPv4-capable secondary MX, is not currently feasible – a small number of real-world MTAs break horribly trying to deliver email to a domain using such a setup.

It might be worthwhile using a dual-stack primary MX, and an IPv4-only secondary MX (even if it’s the same IPv4 address!), to encourage semi-broken MTAs to fall back to the secondary MX (and the IPv4 address it carries) if they don’t fall back properly from IPv6 to IPv4 on the primary MX.

Postfix will only (by default) check five addresses from DNS before stopping the delivery attempt; so if Postfix thinks trying over IPv6 is OK, and you have 5+ IPv6 addresses on your MX, *it will never fall back to IPv4 if something goes wrong*. So, don’t have more than 3-4 IPv6 addresses for your MX.

Finally, you still have to deal with broken far-end DNS, because many modern systems will check if a bounce message could be delivered in principle before accepting a message (by checking the sending domain exists and has a valid MX).

It’s worth doing if you have the patience, and things will only get better, but it’s not necessarily going to be the smoothest ride – use your own judgement.

(Internode have done it and are striving forwards with it, because that’s how we roll. I’d like to think that you’re going to do the same! :)

Monitoring

You need to dual-stack your monitoring system, naturally – all those IPv6 services aren’t really in production if they’re not monitored… but, which IP stack will actually get used to do the checks?

Do all of your hosts support all features on both protocols?

I can’t tell you the solution here, because it’s going to be very situation-specific – but there’s a good chance you’ll have to define a lot of your checks twice (once per protocol).

Don’t forget, if you can’t IPv6-enable your existing host (eg. because it’d suddenly make a lot of legacy checks try to use IPv6 and that won’t work) then you can always proxy checks via dual-stacked host.

Opinions

Lots of IPs…

Since your VLANs now have a very large address space, you can put it to some good use.

For example, if you’re mixing different services in the same /64 network, why not use the first few bits to define what sort of service they are, and create yor ACLs around that?

eg., using /80 ACLs:

An IP could be formatted like so – 2001:db8:n:n:apptype:x:x:id – where “apptype” is one of the following:

  • Infrastructure = 0000
  • Web Server = 1000
  • Mail = 2000
  • SMTP = 2010
  • IMAP = 2020
  • Database = 3000

So you could have a firewall rule allowing access to [2001:db8:n:n:1000::/80]:80 from the Internet, making your website public but nothing else. Conveniently, you can spin up further web servers inside that /80 without needing to alter your firewalls.

Technically, you should always pay attention to bit 7 inside the /64, it should always be zero for statically-assigned addresses as it’s the “Unique vs Local interface” bit, used to determine if a given host-part of the address (the last 64 bits of the IP address) is globally unique, or just locally unique. I don’t think anything really uses this right now, but you’ve been warned :)

Server Numbering

If you decide that the suffix of the address is the host number, that’s not a bad idea. BUT!

If:

  • Fooserver1 → ::1, and
  • Fooserver2 → ::2

Is:

  • Fooserver10 → ::10, or
  • Fooserver10 → ::A?

It’s important to remember that IPv6 addresses are hexadecimal!

You can also embed IPv4 IPs in to IPv6 addresses – a host on 192.231.203.132 could have an IPv6 address of 2001:db8:1234:1234:192:231:203:132 – but, should it?

It’s also tempting to be clever and use port numbers – ::25 for mail servers, ::53 for DNS servers, and ::80 for web sites – but again, should you? That’s hex… for 37, 83, & 128!

Conclusion

IPv6 isn’t really all that hard, just new. The best way to get familiar with it, is just to get on with using it :)

]]>
http://www.mibus.org/2012/11/08/ipv6-for-sysadmins/feed/ 0
Doctor Who, sans Python-iView http://www.mibus.org/2012/09/23/doctor-who-sans-python-iview/ http://www.mibus.org/2012/09/23/doctor-who-sans-python-iview/#comments Sun, 23 Sep 2012 02:23:10 +0000 http://www.mibus.org/?p=1535 Just to illustrate the insanity of ABC’s position in the takedown of Python-iView – here is how to “download” the latest Doctor Who episode from iView, without using an application that infringes the takedown notice as I understand it.

Update: To be clear, this may well still violate the iView Terms of Service… that I’ve never read, and that you don’t get prompted to if you follow the below; the below I pieced together without use of the site. Don’t actually download the file if you want to follow the iView Terms of Service. I’m not a lawyer and can’t comment on whether it’s enforceable on people who are never presented with it. I’d really hope that accessing internet-accessible URLs without an explicit contract between the two parties, isn’t – but what I consider “right” isn’t always.

(Dear ABC: If I’m wrong, please send me an email with where I went wrong and I’ll readily take this post down; my address is in the sidebar on the right).

  1. Visit http://tviview.abc.net.au/iview/auth/?v2 and do a “View Source”.
  2. The “host” will be the IP from the “server” field.
  3. The “app” will be the directory-part of the “server” field, plus “?auth=” plus the contents of the “token” field. (So, ondemand?auth=ABCD1234ABCD1234 or something similar).
  4. rtmpdump --host (host) --app (app) --playpath mp4:threepower_xx_xx -o threepower_xx_xx.mp4 -X --swfUrl http://www.abc.net.au/iview/images/iview.jpg

mp4:threepower_xx_xx is the remote filename.

This may not work from all ISPs, as the path structure is slightly different for Akamai; I’m writing this to prove a point, not so you can actually download an episode.

The only “protection” I see as arguable is the use of a hash of a SWF file – that’s what the last URL passed to rtmpdump, it calculates the hash and passes it in the RTMP request.

What Python-iView adds above and beyond this, is pulling the (unobfuscated) list of show names and episode names (which is where the filename is readily discoverable). So, uh, nothing infringing itself.

Update: What’s really wrong, though, is that the ABC has apparently agreed in legal contracts not to allow downloads from iView, but can only prevent it through mean wording in a Terms of Service.

Update 2: So, I went looking for the Terms of Service. You need to load iView, go to Help, click the FAQ link, scroll up three pages (I’m not kidding), then click the Terms Of Use link. I consider it hideously unfair to hide your Terms Of Use and then complain that they get violated. Really, guys?

By the reasoning in ABC’s Terms Of Use, this post may be seen to “provide the means for others to…circumvent these technical measures”. But only if they consider accessing readily available URLs with established non-infringing applications to be bypassing technical measures. (Dear ABC: I watch more ABC than the remainder of the commercial stations put together, I’m actually a big fan – so please don’t sue me! :).

]]>
http://www.mibus.org/2012/09/23/doctor-who-sans-python-iview/feed/ 0
Letter To the ABC, re the takedown of Python-iView http://www.mibus.org/2012/09/18/letter-to-the-abc-re-the-takedown-of-python-iview/ http://www.mibus.org/2012/09/18/letter-to-the-abc-re-the-takedown-of-python-iview/#comments Tue, 18 Sep 2012 05:46:14 +0000 http://www.mibus.org/?p=1530 Dear ABC,

As a regular ABC watcher, I was dismayed to hear that a takedown notice was sent to Jeremy Visser with regards to his distribution of the “Python-iView” software:

https://jeremy.visser.name/2009/08/python-iview/

Python-iView is, in my mind, little more than a VCR or PVR for the iView service – and from a moral perspective is thus no more or less “wrong”. The approach you have chosen to take would not be legal to take against Tivo or FetchTV recording the FTA DVB-T broadcasts, so it should not be taken against Python-iView.

The software that Jeremy HAD been distributing was absolutely necessary for users of a multitude of platforms to watch content hosted on iView. This includes Android devices that are not compatible with a Flash player, and other embedded devices. It makes it possible to watch iView shows on platforms based on Linux, a free and community-driven operating system (rather than the expensive and proprietary options that the Flash Player fully supports). It also made it possible to watch iView-hosted shows where Internet access is slow or unreliable (eg. during public transport).

I recognise that some content rights-holders might “get their knickers in a knot”, but there are two other approaches that the ABC can take:

  • Encouraging rightsholders to join the 21st Century and allow “fair use”-style distribution via the Internet (particularly with the low-quality video that iView uses, contrasted with high-definition material readily available via illegitimate means).
  •  Having a mechanism to allow for at least locally-produced shows (and other shows where the rightsholder agrees) to be available in full for non-Flash-supporting devices.

The ABC already half-heartedly supports the latter idea, with video podcasts available for some shows (such as Media Watch) – but Media Watch is by far the exception, rather than the rule.

Please consider working _with_ Jeremy, rather than against him, to ensure that ALL Australians have access to ABC programming. The insanity of the copyright restrictions being used against Python-iView are only in effect because the ABC chooses to allow it when licensing content.

Regards;
Robert Mibus.

To the regular ABC viewers reading this letter – I encourage you to support Jeremy and Python-iView and the freedoms and opportunities it represents. Make yourself heard to the ABC, and perhaps we can gain their cooperation.

]]>
http://www.mibus.org/2012/09/18/letter-to-the-abc-re-the-takedown-of-python-iview/feed/ 0