Tag Archives: sysadmin

Moving!

I’ve been [comparatively] quiet for a while now, and I can finally announce the results of my endeavours: I’ve resigned from Internode (ie. iiNet) in order to accept a position at Google in Sydney.

This means that Missy and I and our kids are all packing our bags and getting ready to head eastwards, looking forward to new and exciting adventures.

FAQ:

  1. When? My last week in Adelaide is the week of the 22nd of April.
  2. What will you do at Google? I’ll be a Site Reliability Engineer (which is kinda like a sysadmin).
  3. Why leave? Because it’s the right time, and I’m looking for new challenges.
  4. Why Google? It seems like a good fit for interests and for me personally. The folk I know there are generally pretty awesome, and I intend to learn as much as I can from them.

As a final note, I’d like to thank Simon Hackett and Adam Fox for their support and encouragement over the past few years; without them, I don’t think I’d be where I am now.

(FWIW, I started coming up with a list of people I’d like to thank, but it was unmanageably long — “Thank you” to the rest of you too; you should know who you are).

ping -f[aux flood]

I’m having trouble admitting this publicly, but I learnt something new recently about ping, of all things.

To quote from the man page of ping:

-f Flood ping. For every ECHO_REQUEST sent a period ``.'' is
 printed, while for ever ECHO_REPLY received a backspace is
 printed. This provides a rapid display of how many packets are
 being dropped.

Seems reasonable enough. I mean, “flood” is pretty clear, right? Except, during troubleshooting this week I found that pinging a responding host and a non-responding host resulted in sending a different rate of packets.

Reading on in the man page, it becomes apparent why:

If interval is not given, it sets interval to
zero and outputs packets as fast as they come back or one hun‐
dred times per second, whichever is more. Only the super-user
may use this option with zero interval.

So what you really want, if you want a consistent number of packets per second, is to use the command like this:

ping -f -i 0.01 somehost.example.net

An interval of zero is actually an interval of “anything from zero to 10 milliseconds, depending on the RTT to the remote host”.

Multiple SSH port-forwards, all in a row…

Sometimes, you really need to get from point “A” to point “B”, but you can’t. Restrictive firewalls, poor change control, you don’t own the infrastructure, or maybe it’s just “temporary” and you don’t want to have to go to all that effort just for a few days.

Struggle no more!

SSH Port Forwarding Primer

Let’s just say that you have an application on server “A”, and you want it to be able to reach a web service on your local desktop. You want to open a socket listening on the Remote end, which forwards the connection back over SSH to your local machine. (Maybe a NAT gateway is in the way).

Assuming the server is called “server-a” and your web service is on port 8080, the following may well suffice:

ssh -R8080:localhost:8080 server-a.example.com

“-R” for Remote listening socket, the port number for the designated [remote, in this case] side, then a host and port on the other [local] side.

If you want a local listening socket to forward the packets securely to the remote server – maybe a firewall is in the way this time – just change the Remote for Local:

ssh -L8080:localhost:8080 server-a.example.com

Now, connecting to your desktop’s port 8080 will land on the far-end’s localhost:8080. Again, the first 8080 is for the designated side [local, this time] and the localhost:8080 for the other [remote] side.

Linking port-forwards together

Here’s a hypothetical scenario, where Server D needs to be reached by Server A. For whatever reason, no single host has connectivity the whole distance.

In short, if an SSH port forward opens a remote socket and sends it locally, then that local port can be opened by a different SSH session and forwarded elsewhere too.

You can set this example up thusly:

Diagram of a complex SSH forwarding situation

On the desktop:

ssh -R8080:localhost:8080 server-a.example.com
ssh -L8080:localhost:8080 server-b.example.com

These two together gets Server A connectivity to Server B (via port 8080). Right now, it ends there, because Server B isn’t listening on port 8080.

On Server B:

ssh -L8080:server-d.example.com:8080 server-c.example.com

This time, we don’t want to forward the local socket’s connections to localhost on the remote server, we want to pass it all the way to Server D. What happens is that Server B listens on port 8080, bundles all the data up and over SSH to Server C, then Server C unpacks it all and sends it to the nominated address (server-d.example.com:8080).

Now Server A can point to it’s localhost:8080, and end up on server-d.example.com:8080.

Spiffy, huh!

Please don’t do this in production, though – but if you do, please don’t tell anybody I told you how! :)

You might also be interested in my other SSH port-forwarding hacky trick, where I show how to make a remote server appear local without needing proxy support in your application.

GNOME-Shell multi-timezone clock

As many of you know, the company I work for (Internode) was recentlyish purchased by iiNet.

iiNet’s headquarters is in Perth, which is 1.5 or 2.5 hours “behind” Adelaide time.

There’s no longer a multi-timezone clock available for the panel, so… oh well I just wrote one myself.

Screenshot of multi-timezone clock in GNOME-Shell top panel

Right now, it only supports GMT+8 as the “remote” timezone, but it’s easy enough to change the code.

The code is up on GitHub: https://github.com/mibus/MultiClock

Thanks to Marco Dallagiacoma for writing the “Fuzzy Clock” plugin, as I used it as a basis for my own code.

NetWorker – Random stalling

One of the things I’ve spent a lot of time with, has been EMC NetWorker (previously Legato NetWorker).

A vaguely common issue is for a process of some kind – backups, staging to tape, restores, etc – for no reason just stop making any new progress.

Once you’ve checked off the common reasons – like making sure you haven’t run out of disk space or usable tapes – it seems like the only option is to restart NetWorker as a whole, losing any in-progress actions (even ones that are to devices that haven’t stalled).

I suspect that random underlying I/O issues can occasionally upset it, and it doesn’t quite recover. But, whatever. How do you make it recover a single device, without restarting the whole thing?

First up, get the PID of the main nsrd process. On Solaris, ps -ef | grep nsrd; or on Linux ps uaxw | grep nsrd.

Assuming the PID is 1234, you next need to run: dbgcommand -p 1234 PrintDevInfo

It should pretty quickly spit out a whole stack of debugging info to /nsr/logs/daemon.raw. It’s moderately complicated, but you should see that it’s a dump of its internal state of each device, including d_device – the *nix device or directory, and mm_number – the unique ID for the nsrmmd process for that device.

So – find the device you’re interested in, and find the mm_number for that device.

Get a list of your nsrmmd processes, eg. ps -ef | nsrmmd or ps auxw | grep nsrmmd. If your mm_number is 5, then there will be a process nsrmmd -n 5

Kill the process, and it should re-spawn by itself on further access.

IPv6 for SysAdmins

IPv6 is a new and complicated piece of technology; like any new technology it takes a while to get used to and discover what does and doesn’t really work.

Who am I? I’m Robert Mibus, and I helped roll out many of the IPv6-enabled services at Internode (now part of iiNet). I’m writing this based on my personal experience and opinion, so even though I reference Internode frequently, please don’t consider any of the below official communication from them, OK? :)

What this page is:

  • Systems-focused
  • Some network stuff you need to know
  • What I wish I knew three years ago

What this page is not:

  • Waxing on about infinite address space for the sake of itself
  • Pretty graphs on consumer uptake
  • Spiffy future IPv6-could-do-this explanations

Network Stuff

Connectivity

The first thing about IPv6 is how you’re going to get your connectivity. You have two basic options – obtain it natively (eg. via ethernet or PPP from your ISP or colo provider), or via an IPv6-specific tunnel (eg. from a tunnel provider like HurricaneElectric).

So… Don’t use tunnels; just harass your ISP. IPv6 is critical to future growth on the Internet, and is no longer a “maybe sometime in the future thing”.

Addressing

An IPv4 address is like this: 192.231.203.132.

An IPv6 address is like this: 2001:44b8:0001:0000:0000:0000:0000:0001

In order to make the addresses shorter, you can drop leading zeroes in any group, ie: 2001:44b8:1:0:0:0:0:1

You can also replace one (and *only* one) string of :0:0:0:... (for any number of :0:), with “::“…

2001:44b8:1::1

That’s Internode’s IPv6 DNS resolver address. The IPv4 address is actually longer:

192.231.203.132

It’s worth pointing out that many apps don’t accept IPv6 addresses “as-is”; primarily because adding port identifiers (like :8080) causes ambiguity. In those cases, typically the address has square brackets placed around it, eg. [2001:db8:1::1]:8080

Allocations

An ISP will get a large allocation of IPv6 addresses. Typically this will be a /32 or larger (leaving at least 96 bits for the ISP to play with).

eg.

2001:db8::/32 → ISP

The ISP is likely to hand a customer a block somewhere between a /48 and /64. Internode currently gives out /56 as standard, so a customer’s allocation from an ISP might look like this:

2001:db8:1c7:2d00::/56 → ISP Customer

Individual VLANs are typically cut up as /64s, for reasons we’ll get in to later. A /56 thus gives you 256 fully addressable VLANs. You might carve them up like this:

2001:db8:1c7:2d00::/64 → Servers
2001:db8:1c7:2d01::/64 → Ethernet PCs
2001:db8:1c7:2d02::/64 → Wireless PCs

The first 64 bits is then the “network part” and the second 64 bits is the “host part” of the address.

Allocations – Pitfalls!

Is 2001:db8:1c7:2d::/56

  • 2001:db8:1c7:2d00::/56, or
  • 2001:db8:1c7:002d::/56?

Leading zeroes are removed, so it’s 002d not 2d00. If you’re at all uncertain when subnetting, put the zeroes back in when you’re working it out! A tool like “sipcalc” on Linux can also help with calculations like that.

While I’m on subnetting – only subnet on 4-bit boundaries, please… (each hex character is 4 bits). Why? You know how in IPv4, it’s easy to see if a given IPv4 address falls inside or outside a /24 subnet, but harder on a /28 or /29? Subnetting inside the 4 bits is going to drive you insane with craziness in a similar way.

(ie., don’t go making a /63, /62, or /61 – round up and make it a /60!)

Topological Division

Without trying to contradict the previous section too much, all those extra bits can be useful for route aggregation. If you run a large network with multiple campuses, you can use all the extra bits to create routes that can be easily aggregated together.

Imagine the topology for connecting a large set of schools. You might take a chunk of your address like this – :0000:. That’s really 0000 0000 0000 0000 in binary. You might take a bit for region (East-coast or not?), a couple of bits for sub-region (City, or other major routing point), and a couple for the specific site number.

One thing you should always try to do is keep the top-most bit reserved.

Why?

Because it means that if you’ve divided yourself too far and run out of usable networks, you can make a whole new subdivision scheme because you still have half of our addresses left!

Network Stuff – Going Smaller Than /64

It’s really not going to be worth it. “Saving address space” isn’t the argument it used to be!

Among other things, you won’t be able to use SLAAC (stateless address autoconfiguration).

Chances are, plenty of other things will be awkward – probably the one I’d be most concerned with is that you’re going to be breaking lots of people’s expectations about what a network should look like, which may confuse others looking at your network.

Technically, going smaller than /64 works in some cases – like point-to-point links – and may even have some benefits in those edge cases. Feel free to do it if there’s a really really really good reason.

But allocate the /64, at least notionally – That way you can still change your mind later!

You may end up being given something smaller than a /64 in cases where you might only normally get a single IP – eg. in a cheap VPS. Take what you can get :)

Hardware Support

For non-enterprise-grade routers, you can check places like Whirlpool. You can also take a look at Internode’s router offerings – all should now support native IPv6.

Be warned though; some hardware might “support” IPv6, but do so in a limited fashion – it might be software-routing rather than hardware, or it might not expose all of the same statistics via SNMP for IPv6, etc. As always, it’s best to test hardware before making a big purchase.

ICMP Firewalling

Don’t. It wasn’t a great idea for IPv4, and IPv6 relies even heavier on ICMP control messages.

There’s RFC4890 to cover precisely how you can filter ICMP safely – it’s 38 pages long.

But, seriously, just don’t. (Unless you need to)

Firewalling!

The suckiest part of dealing with IPv6 firewalls right now, is that they’re typically wholly separate from IPv4 firewalls.

Using iptables? You need to use ip6tables too. Configuring it on your CPE? It’s probably not smart enough to have the two really intermingled either.

What this means is that documentation is even more important than ever. Each and every set of firewall entries in both firewalls should be documented in enough detail that you can see clearly that the two sets of firewalls are functionally identical.

One nice benefit of IPv6 when firewalling, is that you can afford to have an IPv6 address per service on a given host. That makes firewalling easier, and can make it easier to migrate the service to another host later. But, if you’re lazy (like me! :), you’ll probably still be happy using just one IP per host. (Virtualisation does mean we get to have a lot of hosts, though…).

ARP! No, Neighbour Discovery…

In IPv4, “ARP” is used to discover the MAC address of a host you want to talk to. In IPv6, the comparable protocol is Neighbor Discovery (ND) – which uses ICMPv6. The response comes in the form of a Neighbor Advertisment, also ICMPv6.

With ARP, your request is broadcast across the whole VLAN. ND uses link-local multicast to target just a subset of hosts – typically a small enough subset that it is only hitting the one host! (For the technically-inclined, the multicast packet is sent to all hosts that share the last 24 bits of their IPv6 address with the address being looked for).

Like with ARP, this isn’t routed traffic, it’s specific to the layer-2 network that the host is physically connected to.

Broadcast Ping

Much like ARP’s replacement, the replacement for a broadcast ping uses local network multicast too.

All nodes on a network will join a special multicast group with an address of ff02::1. Ping that instead of an address like 192.168.1.255.

There’s a similar one that is only joined by routers (rather than hosts), ff02::2.

Duplicate Address Detection

An in-built part of IPv6 is “Duplicate Address Detection”, or DAD.

In short, when an IP address is brought up on an interface, it uses Neighbour Discovery to make sure it’s the only node with that address. Due to this, freshly-added addresses on an interface will show as “tentative”, and not actually be usable.

This includes all configured IPv6 addresses during boot, so applications need to cope with this if they try to bind to a specific address on startup. Sometimes they don’t cope, and just bomb out with an error that they couldn’t bind to the IP address as it’s not configured. (ISC BIND is an example of this).

Router Advertisements

Similar to a Neighbor Advertisement, a Router Advertisement is a network router saying “Hi, I’m here!”. This can be used by hosts to auto-configure themselves with an IPv6 address, and set a default route for IPv6. (This is called SLAAC – StateLess Address AutoConfiguration).

An advertisement like that includes a “preferred” and “valid” lifetimes – how long they should be used for new connections, and how long they should be allowed for existing connections, respectively. Routers can re-send the advertisements and refresh those times, kinda like DHCP.

It’s not, however, a full DHCP replacement – it doesn’t have acknowledgements, logging, DDNS, or any number of other fancy features.

For raw usefulness getting a set of desktops on an existing network to use IPv6… priceless.

Neighbor Discovery Security

ARP isn’t really secure by design, so it should be no surprise that Neighbor Discovery has similar sorts of issues possible. If you’re the particularly security-conscious sort, you should check out “Secure Neighbor Discovery” (SEND) or “RA-Guard”.

Unique Local Addresses (ULA)

Roughly equivalent to RFC1918 space, but (probably) still unique to a single environment.

It’s really not needed in most cases – you’re normally expected to just use addresses from your service provider.

Prefix Delegation (PD)

This is the means by which large blocks of address space (prefixes!) can be requested by clients. It’s actually a function of DHCPv6!

Prefix Delegation is how your ADSL modem can request a chunk of address-space from your ISP, and get back a /56 (or whatever is configured for your account) to further subdivide.

Hosts

Desktops

If you want to get a set of desktop PCs on a LAN using IPv6, you’re in luck.

Once your router has an IPv6 range allocated to it (by Prefix Delegation or some other means), then it can send on Router Advertisements to local LAN segments; PCs and other devices will use SLAAC to autoconfigure themselves and use the connectivity.

DHCP (via IPv4) still reigns supreme – handing out DNS and the like – but you nearly instantly have IPv6 connectivity on any modern system.

It’s worth noting that most current-generation OSes will actually end up randomly creating and cycling through IPv6 addresses on a regular basis; this is due to “Privacy Extensions”, an extension of SLAAC’s normally determininstic address selection that uses random numbers instead.

Desktops, the future!

DHCPv6 will, one day, be the norm. But not today, so I’ll leave that for another day :)

Servers

SLAAC is a really bad idea for servers, as it means that changing your host machine (or just a NIC) would change it’s IP! Statically configure the addresses instead.

You can still use Router Advertisements for setting a default gateway, if you want – as mentioned above, there are security concerns, but it can still be pretty convenient.

Services

Common Architecture: Reverse Proxy

One very easy way of making a service IPv6-capable, is by sticking some sort of reverse proxy in front of it. IPv4 traffic can flow as per usual – but when you configure an IPv6 address for the service, the address is actually on a dual-stacked box. This box can then redirect traffic back to the IPv4-only host.

For HTTP, you can set the X-Forwarded-For header with the original IPv6 address (since it’ll be lost otherwise). Make sure your backend application knows to trust the header when (and only when) it comes from the proxy, and make sure it can deal with IPv6 addresses properly!

You’ll also want to make sure that any ACLs or other IP-based restrictions (eg. Geo-IP blocking) take into consideration the proxy’s existence. It’s probably good to set a fair amount of logging on the proxy, too, so you can tie things together when troubleshooting later.

Proxies would work fairly well for:

  • HTTP/S
  • IMAP/POP3
  • DNS, NTP

They won’t work for cases where you actually care a lot about the source – IP address reputation checks for SMTP, rate-limits on web pages, IP-blacklists on forum software, etc.

Anything using bidirectional communications is pretty much out too (eg. FTP, backups), and you also have to note that (like NAT) it risks unbalancing load-balancers if you use IP-based sticky sessions.

Common Architecture: IPv4-only Load Balancer

There’s a couple of different ways to work around an IPv4-only load balancer:

  • Make an IPv6 load-balancer in software
  • Go directly to one backend
  • Round-robin to all backends

I actually think the “direct to backend” solutions aren’t too bad… hacky, but survivable. The vast majority of applications fall back to IPv4 sufficiently well, don’t forget. Make sure to set a low TTL on the DNS, so you can remove it in event of a failure.

Your app sucks

Let’s face it, most sysadmins have to look after any number of custom in-house applications. They’re probably going to need some love. I can’t help you with that.

There’s also amusing side-effects; several software products still truncate IPv6 addresses like “last” outputs in some *nix-based systems, or in logs in database tables. Speaking of databases, what sort of field type are your applications using to store IP addresses? Will it fit an IPv6 address in it? Is it just a 32-bit int? Many databases now support a native “IP address” type – use it if you can.

Rate-limits

While we’re on a tangent, let’s talk about those rate limits too.

  • Is it useful to rate-limit a single IPv6 IP? (A /128). Probably not, it’s too easy to change.
  • Is it useful to rate-limit a /64? Probably; there’s a good chance the user has multiple /64s at their disposal, but only so many…
  • Is it useful to rate-limit a /56? Useful? It’s not even safe – Internode hand out /56s, other ISPs can (and do!) regularly hand out smaller.

Limit the /64 – it’s not perfect, and it’s still not even entirely “fair” in some edge cases, but it’s the best you’ve got.

DNS

DNS is a super awesome and easy service to enable for IPv6, so it’s a great first choice. The protocol has native fallback to secondary servers in case of failure, for both a client accessing a DNS resolver and for a DNS resolver talking to nameservers.

DNS: Aside

You do also end up having a rather large reverse address space to manage DNS for. A /56 has 4722366482869645213696 addresses!!

The only sane solution (other than having no reverse DNS) is procedural DNS generation. Internode do this for their customers (using the “pymds” open source project), optionally delegating it to nameservers of the customer’s choosing. After that, it’s up to you :)

SMTP MTA

Don’t forget to configure your Reverse DNS for SMTP – Yes, people really still do look at this, even when using IPv6.

Connectivity is key; if your MTA software thinks it has IPv6 connectivity then it really needs to work. Check your fallback to IPv4 when IPv6 connectivity is missing – some software doesn’t!

It’s worth noting that people don’t interactively use their mail servers very often, so there’s no implicit verification of connectivity, or firewall validity. Mistakes can go unnoticed for a very long time!

The other common issue running an IPv6-capable MTA is broken [far-end] DNS; entirely too many systems return SERVFAIL rather than an empty NOERROR response, causing extra retries and often making the sending MTA never fall back to IPv4 at all. There’s no real fix here, this is just a heads-up…

SMTP MX

Running an IPv6 MX means a moderate amount of pain right now (though it’s getting quickly better now that GMail runs an IPv6-capable MX).

Running an IPv6-only MX, with an IPv4-capable secondary MX, is not currently feasible – a small number of real-world MTAs break horribly trying to deliver email to a domain using such a setup.

It might be worthwhile using a dual-stack primary MX, and an IPv4-only secondary MX (even if it’s the same IPv4 address!), to encourage semi-broken MTAs to fall back to the secondary MX (and the IPv4 address it carries) if they don’t fall back properly from IPv6 to IPv4 on the primary MX.

Postfix will only (by default) check five addresses from DNS before stopping the delivery attempt; so if Postfix thinks trying over IPv6 is OK, and you have 5+ IPv6 addresses on your MX, *it will never fall back to IPv4 if something goes wrong*. So, don’t have more than 3-4 IPv6 addresses for your MX.

Finally, you still have to deal with broken far-end DNS, because many modern systems will check if a bounce message could be delivered in principle before accepting a message (by checking the sending domain exists and has a valid MX).

It’s worth doing if you have the patience, and things will only get better, but it’s not necessarily going to be the smoothest ride – use your own judgement.

(Internode have done it and are striving forwards with it, because that’s how we roll. I’d like to think that you’re going to do the same! :)

Monitoring

You need to dual-stack your monitoring system, naturally – all those IPv6 services aren’t really in production if they’re not monitored… but, which IP stack will actually get used to do the checks?

Do all of your hosts support all features on both protocols?

I can’t tell you the solution here, because it’s going to be very situation-specific – but there’s a good chance you’ll have to define a lot of your checks twice (once per protocol).

Don’t forget, if you can’t IPv6-enable your existing host (eg. because it’d suddenly make a lot of legacy checks try to use IPv6 and that won’t work) then you can always proxy checks via dual-stacked host.

Opinions

Lots of IPs…

Since your VLANs now have a very large address space, you can put it to some good use.

For example, if you’re mixing different services in the same /64 network, why not use the first few bits to define what sort of service they are, and create yor ACLs around that?

eg., using /80 ACLs:

An IP could be formatted like so – 2001:db8:n:n:apptype:x:x:id – where “apptype” is one of the following:

  • Infrastructure = 0000
  • Web Server = 1000
  • Mail = 2000
  • SMTP = 2010
  • IMAP = 2020
  • Database = 3000

So you could have a firewall rule allowing access to [2001:db8:n:n:1000::/80]:80 from the Internet, making your website public but nothing else. Conveniently, you can spin up further web servers inside that /80 without needing to alter your firewalls.

Technically, you should always pay attention to bit 7 inside the /64, it should always be zero for statically-assigned addresses as it’s the “Unique vs Local interface” bit, used to determine if a given host-part of the address (the last 64 bits of the IP address) is globally unique, or just locally unique. I don’t think anything really uses this right now, but you’ve been warned :)

Server Numbering

If you decide that the suffix of the address is the host number, that’s not a bad idea. BUT!

If:

  • Fooserver1 → ::1, and
  • Fooserver2 → ::2

Is:

  • Fooserver10 → ::10, or
  • Fooserver10 → ::A?

It’s important to remember that IPv6 addresses are hexadecimal!

You can also embed IPv4 IPs in to IPv6 addresses – a host on 192.231.203.132 could have an IPv6 address of 2001:db8:1234:1234:192:231:203:132 – but, should it?

It’s also tempting to be clever and use port numbers – ::25 for mail servers, ::53 for DNS servers, and ::80 for web sites – but again, should you? That’s hex… for 37, 83, & 128!

Conclusion

IPv6 isn’t really all that hard, just new. The best way to get familiar with it, is just to get on with using it :)

gSolaris

Among other things at work, I help look after a small set of Solaris hosts. One of the real frustrations I have with Solaris is the lack of functionality in many of the “standard” applications, things that I’ve come to consider usual (and often essential).

We work around these shortcomings by having GNU utilities installed, but with a “g” prefix so that scripts depending on the native Solaris flags and options still work perfectly.

This naturally leads to many jokes about how much a particular command is painful under Solaris, and “just put a ‘g’ in front and it’ll be fine”.

Amusingly, this is a better plan than you’d expect; it’s not unheard of to construct a long command-line built *soley* from the GNU commands, just to work around Solaris’ limitations (and maybe a little bit just for the fun of it).

eg, our internal documentation for a certain procedure has a very long command that basically boils down to:

gfind -type f -print0 | gxargs -0 gmd5sum | gawk '{..process output..}'

gSolaris: Solaris without the suck. Or, maybe at least with less.

I tease – Solaris has some real nice features that we use heavily, like ZFS, zones, dtrace, and it’s real good at heavy-duty NFS serving. I just feel like it’s stuck in 1988 for much of the rest of the core functionality.

MythTV “du” – find out what shows take up the most space

I was wondering where all my space was going in MythTV recently, so I figured I’d whip up a quick “du” that will give me a per-title breakdown of usage.

It assumes you have just the one storage area for recordings.


#!/bin/bash
STORAGEPATH='/mnt/storage/Media/MythTV/recordings'
DBPASS='abc123'

QUERY="SELECT basename,title FROM recorded ORDER BY title;"
MYSQL="mysql -u mythtv --password="$DBPASS' mythconverg"
SIZE=0
COUNTING_FOR=''
echo $QUERY | $MYSQL | tail -n +2 | while read BASENAME TITLE; do
if [ "$COUNTING_FOR" != "$TITLE" -a "$COUNTING_FOR" != "" ]; then
echo "${SIZE} MB - $COUNTING_FOR"
SIZE=0
fi
COUNTING_FOR="$TITLE"
SIZE=$(($SIZE + `du -m "$STORAGEPATH/$BASENAME"|cut -f1`))
done

Use it like this:

./mythtv_du.sh|sort -nr|head -20

GoDaddy – mixed-case DNS WTFery.

A friend passed me a bounce of mail to my domain; DNS-related it said.

Dutifully, I checked the record:

$ dig @ns43.domaincontrol.com mibus.org mx

; <<>> DiG 9.7.3 <<>> @ns43.domaincontrol.com mibus.org mx
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 541
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 2, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;MibUs.OrG. IN MX

;; ANSWER SECTION:
MibUs.OrG. 3600 IN MX 10 aspmx.l.google.com.
MibUs.OrG. 3600 IN MX 20 alt2.aspmx.l.google.com.
MibUs.OrG. 3600 IN MX 20 alt1.aspmx.l.google.com.

;; AUTHORITY SECTION:
MibUs.OrG. 3600 IN NS ns43.domaincontrol.com.
MibUs.OrG. 3600 IN NS ns44.domaincontrol.com.

;; Query time: 89 msec
;; SERVER: 216.69.185.22#53(216.69.185.22)
;; WHEN: Fri Feb 24 08:42:50 2012
;; MSG SIZE rcvd: 155

Yep, there it is.

(...and yes, I'm with GoDaddy. I'm horribly likely to shift at my next renewal - if not before. But anyway).

Wait. "MibUs.OrG."?

It's repeatable, on that _one_ NS, from both the US and here in Australia. The other NS, is fine. Non-MX queries... are also fine. Mixed-case queries for the MX, all come back in that one same (different) case.

What. The. Hell.

I made a change to my zone data with their pretty web-based console. Re-ran the query... and it was fine. Except, all the mixed-case queries all came back lowercase.

Hmm. I wonder.

I made another change.

$ dig @ns43.domaincontrol.com Mibus.Org mx

; <<>> DiG 9.7.3 <<>> @ns43.domaincontrol.com Mibus.Org mx
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57417
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 2, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;Mibus.Org. IN MX

;; ANSWER SECTION:
Mibus.Org. 3600 IN MX 10 aspmx.l.google.com.
Mibus.Org. 3600 IN MX 20 alt2.aspmx.l.google.com.
Mibus.Org. 3600 IN MX 20 alt1.aspmx.l.google.com.

;; AUTHORITY SECTION:
Mibus.Org. 3600 IN NS ns43.domaincontrol.com.
Mibus.Org. 3600 IN NS ns44.domaincontrol.com.

;; Query time: 86 msec
;; SERVER: 216.69.185.22#53(216.69.185.22)
;; WHEN: Fri Feb 24 08:47:52 2012
;; MSG SIZE rcvd: 155

...and now, all DNS queries come back with this "proper case" version.

It's my humble opinion as a DNS administrator elsewhere, that they're running some sort of fancy reverse-caching DNS server in front of their "real" DNS servers; one that fakes the "AA" flag on responses, doesn't drop the TTL, and is cleared by their software on updates.

Oh, and one that preserves the case of the first query it sees in its cache, and keeps it around.

Why is this important? Well, for starters it's just stupid. For seconds, people are starting to use bit 0x20 (ie., the "shift" bit) for adding extra entropy in to DNS queries. GoDaddy's DNS servers go well beyond breaking it and in to the territory of royally messing it up.

Desktop SCRAM

So the problem goes like this:

  • A coworker asks if I want to join them for a coffee. (This happens multiple times per day, FWIW). I say “Yes!” and jump up…
  • …then sit back down to lock my computer screen, and hop back up…
  • …only to realise I’ve left my music playing – so I sit down, unlock my screen, pause my music, and re-lock my screen…
  • …at which point, my coworkers have already long since disappeared.

My solution? A Desktop SCRAM button, where I can just go whack and walk away with impunity.

It’s made up of a USB-serial cable, and a ~$10 button. The button’s two contacts are wired to the DSR and Ground lines respectively.

The code is available (GPLv3) at: http://github.com/mibus/DesktopScram/