Get the custom pymds fork here; it'll be merged upstream shortly-ish.

Caught out needing new "Sun" Java packages for Ubuntu, by the DLJ revocation?

Grab a usable set of build source packages from http://archive.canonical.com/ubuntu/pool/partner/s/sun-java6/ - you'll need an *.orig.tar.gz, a *.dsc, and a *.debian.tar.gz file for the version you've picked. I used a 6.26 version from Natty.

Grab a latest (currently 6u30) "bin" Linux packages for each architecture from http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html (Note: needs Javascript!).

dpkg-source -x *.dsc
cd sun-java6-6.26/
rm *bin
mv ~/Downloads/jdk-6u30-linux-i586.bin jdk-6u30-dlj-linux-i586.bin
mv ~/Downloads/jdk-6u30-linux-x64.bin jdk-6u30-dlj-linux-amd64.bin


Edit debian/rules, comment out the section following the comment 'check if the sources are the "same"'.

dch -v 6.30

(add in a stub changelog entry - this is just so it realises what version it's building)


cd ../sun-java6-6.30
dpkg-buildpackage -b -uc


Then you should be left with workable packages matching the last Ubuntu-released ones, but with a newer JRE/JDK.

Note #1: I haven't so much as installed these packages, it's just theory. It built, it ships - I mean, hey, it's New Year's Eve! ;-)
Note #2: This won't include a working web plugin - I pulled the build packages from after 6.26-1natty1, which was the last release with a working web plugin.

We're pleased to announce that your proposal(s) has/have been ACCEPTED for LCA2012.

<SNIP>

---
 IPv6 Dynamic Reverse Mapping - the magic, misery and mayhem
---

So - wow! I'll see you there :)

But still, screwups happen. Here's some tips on what to do to try to ensure you aren't caught out in the cold.

Tip 1: Have multiple servers.

Without a doubt, this is the biggest tip about DNS. Designed in from the beginning was an assumption that you'd have multiple nameservers for a given zone. So... have them!

Put them as far apart as you reasonably can - different hosts, different networks, different power. The more they share, the more risk you're in.

Countertip: Hosting your DNS server only over your ADSL link.

Tip 2: Do backups.

Pretty standard sysadmin fare. RAID isn't a backup, and neither is a slaved nameserver.

Tip 3: Nameservers must all agree.

You know how kids will sometimes ask their parents the same question independently, hoping for a different answer? It's important that the parents always give the same answer, and it's downright vital that your nameservers do too. Don't let them get out of sync!

Typically, zone transfers fix all your woes here, but do make sure they're working.

Tip 4: Test your changes directly against all nameservers.

It's just a small change, right? What could go wrong? Lots! So test each server individually. If one doesn't update, maybe you have a problem that you need to fix. (Or maybe it's just a bit laggy - it happens). "dig" is your friend.

Countertip: Not realising until too late that you're breaking Tip 3

Tip 5: Make your NS records match your glue.

If you've told your domain registrar that your nameservers are ns1.example.org and ns2.example.org, then make sure you put that in your zone file too - all sorts of wacky caching issues can ensue when you don't.

Tip 6: If you use a CNAME record, don't use anything else.

CNAMEs are a really convenient way of saying "www.example.org is really webserver.example.org". You can't then say "But www has an MX of foo.example.org" or "www is also a subdomain with nameservers at ...".

That'd be contradictory, because you've already said with the CNAME that it's really webserver.example.org. It can't be both, if it's both then it's actually something different altogether and needs its own records.

Relatedly, don't point a CNAME at anything other than a plain hostname - Don't try to CNAME www.example.org to example.org, it'll just break stuff.

Tip 7: Don't firewall out DNS queries to your nameserver.

No, really. The whole internet needs to be able to look up domain names, not just some of it, not just most of it. (You're excused if it's a private nameserver, of course!).

Counterpoint: Using bogon filters on nameservers and ignoring genuine queries.

Bonus Tip: Monitor your servers.

If you're running DNS servers in production, monitor them so you know that you haven't lost one. Once it's all set up right, you really can lose one without noticing.

An easy win here is to normalise the numbers you call. In South Australia, a number listed as "(08) 5550 1234" can be dialled as either 0855501234 or 55501234 - the "08" prefix is optional, since it merely clarifies the area code.

Since you don't want to have to reconcile against both forms (that just makes things messy), let's clean it up so it always appears with the leading '08':

;; Handle 08XXXXXXXX calls by default, as this is our "native" area code here in SA.
exten => _XXXXXXXX,1,Goto(08${EXTEN},1)


Easily done.

Another common issue is to 'answer' an outbound call (either explictly with Answer() or implicitly with something like Playback()) before actually dialling. Answering the call too early means your billsec - which you're potentially basing your customer billing on - is higher than it should be!

If you want to play a message back to the caller before the call itself is made, use Progress() to indicate that you're doing it as the call progresses, and that the call isn't actually connected yet.

A final example is much less common - when dialling one number actually calls another; you have several options available with different tradeoffs. For this example, pretend that the PSTN-looking number 0855501234 should actually be delivered to the VoIP-looking number 0870101234. (Maybe it's a cheaper internal-use-only equivalent number, and people forget to use it?)

This example will have your CDR showing the original number (the PSTN number that was dialled, rather than the number that was actually called). It assumes your upstream SIP provider is named vsp1


exten => 0855501234,1,Dial(SIP/0870101234@vsp1)


This example will have your CDR showing the number that was really called, not the number dialled:


exten => 0855501234,1,Goto(0870101234,1)


The following will have a mix of the two - a CDR for 0870101234 for the ringing, and a CDR for 0855501234 covering the ringing and the billable call. This assumes regular_outbound is your context for outbound calls.

exten => 0855501234,n,Dial(LOCAL/0870101234@regular_outbound)


So, that's a few easy ways to make sure that your CDRs are actually worth the bits they're saved with.

Absolutely!

As an example, imagine the contexts incoming (where calls go to when they come in to Asterisk from a SIP provider) and outgoing (which allows outbound calls from internal phones). There's also an internal context to allow calls between internal phones.

[internal]
exten => s,1,WaitForExten

;Internal extensions
exten => 1000,1,Dial(SIP/alice)
exten => 1001,1,Dial(SIP/bob)
exten => 1002,1,Dial(SIP/charlie)
exten => 1003,1,Dial(SIP/jack)

[outgoing]
include => internal
;External calls
exten => _XX.,1,Dial(SIP/${EXTEN}@voipprovider)

[incoming]
; Incoming calls go to Alice, our receptionist
exten => s,1,Dial(SIP/alice)

Let's add in an override for when an employee rings in from their mobile phone - instead of having to talk to Alice, they get put back into the internal context, so they get to directly dial an extension.

Within the incoming context, add:

exten => s/0491570156,1,Playback(please-enter-the)
exten => s/0491570156,n,Playback(number)
exten => s/0491570156,n,Goto(internal)

We can make it a bit better yet. Let's allow another employee's mobile phone in, and let's share the config by moving them to a different context. We can also reuse parts of the dialplan in the new context by only using the CallerID information to override the first part of the sequence.

[incoming]
exten => s/0491570156,1,Goto(employeemobiles)
exten => s/0491570157,1,Goto(employeemobiles)
exten => s/0491570158,1,Goto(employeemobiles)
[employeemobiles]
exten => s/0491570156,1,Set(CDR(accountcode)=alice)
exten => s/0491570157,1,Set(CDR(accountcode)=bob)
exten => s/0491570158,1,Set(CDR(accountcode)=charlie)
exten => s,2,Background(please-enter-the)
exten => s,3,Background(number)
exten => s,4,WaitForExten
include => internal

Note the accountcode being set - this way, you can readily choose to allow outgoing calls from your PABX for remote employees - and it all gets tracked separately (per employee) in your call data records.

You can also just as easily use include => outgoing so that calls in from the mobiles can make calls back out to any other number (eg. if you have great international rates from your PABX, or free calls to certain numbers).

Download the slides here

Overall, I think it went pretty well - it didn't seem too obvious that I hadn't had a chance to practice the talking that went with the presentation, and only one of my demonstrations failed.

A handy hint for doing PDF exports from OpenOffice.org - Use the menu (File -> Export as PDF...) rather than the toolbar button; you get shown a dialog with a large number of options that are useful in reducing filesize. (Reduce the DPI and allow OO,o to resize the images, lower the JPEG quality).

If I have a server at home that can _only_ be accessed via 192.168.1.12 (say) - perhaps because it is a web application that rewrites all internal URLs to always go to that IP - how do I get access?

Easy. My application listens on 192.168.1.12:8888, so I'll give it exactly that:


ip addr add 192.168.1.12/32 dev lo
ssh -L 192.168.1.12:8888:192.168.1.12:8888 user@example.net


So, I add 192.168.1.12/32 as a local IP address, bind SSH to it, forward all packets for 192.168.1.12:8888 over the SSH tunnel to the gateway server (example.net for this example), then it unbundles it from the SSH stream and passes it to the real 192.168.1.12.

It only works for that one port on that one IP address (though you can add individual IPs and ports easily enough!), but the key part is that it works.

To clean up, close your SSH links and run:


ip addr del 192.168.1.12/32 dev lo


What is IPv6?

IPv6 is the next step after our current IPv4 addressing scheme - and not a lot more. Instead of the we-thought-it'd-be-ample 2^32ish addresses, we get the it'll-be-enough-this-time-for-sure 2^128ish addresses. If nothing else, it'll take us a long time to run out again!

So how do we, as sysadmins, use it?

Software developers have taken a lot of the fun out of this already. Most software - Apache, BIND, SSH, etc - have already had IPv6 support for a very long time. You can just add an IPv6 address to the "bind" or "listen" address of the vast majority of non-custom-built software and it will Just Work. Plenty of software doesn't even need that, and will bind to the IPv6 address just as soon as it exists on your system.

If you don't want to use IPv6's native autoconfiguration, statically configuring your address and gateway is a simple enough process on any vaguely modern OS. Probably Windows, too ;).

What about gotchas?

There are plenty of places this can fall down... but none that typically hurt too much.

When you add IPv6 support to a server, note that you're also adding IPv6 support for it to talk to other services. If it makes calls out to another server, it will try to do so over IPv6. So, make sure your IPv6 connectivity is good, and all of the services that your server might connect to work over IPv6 (if they have AAAA DNS records - if not, it will quickly fall back to IPv4 anyway). This is especially true if your servers chatter amongst themselves to share data (like caching resolvers and authoritative nameservers, or webservers and database servers). You'll also be at the mercy of the IPv6 support of any providers that you consume services from.

Anywhere that an IP address gets treated as "special" and not just an opaque block of text, could cause issues. For most, this won't be an issue. But, if you're storing it in a database, make sure you aren't using a short fixed-length string - IPv4 addresses max out at 15 characters but IPv6 goes to 11. I mean, 39.

Why haven't you done it yet?

There are plenty of reasons not to have IPv6 enabled your services.

The easiest answer is that everyone will need an IPv4 address to see everyone else's services, so they'll already have one to see mine. Right? Realistically, IPv4 will probably continue to be supplied behind a carrier-grade-NAT system (or "something"), and most stuff on the internet works just fine behind NAT anyway. It will be a long time before IPv4 addresses start getting dropped entirely from consumer broadband connections.

Not everyone has an IPv6-capable upstream provider. If you are with a particular hosting provider and they don't provide IPv6 connectivity, you're pretty much stuck. Find a new one (and migrate all of your hosts) or just wait it out.

Moving on, and a lot of "miscellaneous" equipment doesn't support IPv6. An up-front example of this is hardware load-balancers. Months out from the beginning of 'IPv4 exhaustion', you can buy new hardware that doesn't support hardware-accelerated IPv6. Sure, you can work around it, or buy from an alternate vendor - but really, that's pretty excessive just to support 128-bit addresses, when we all know that everyone will have IPv4 connectivity for years to come. Do your "appliance" boxes and VMs support IPv6?

The same lack of hardware applies in the consumer-space - seen very many DSL modems with native IPv6 lately? The biggest driver there will be gamers and P2P users wanting to restore end-to-end connectivity.

There are also potential security issues - how do you make sure your IPv4 and IPv6 firewalls are equivalent? Typically, they're treated as independent... changes to one but not the other can lead to breakage or security holes. (Forget to lock off SSH over IPv6 so only your corporate network can see it? Oops! Locked down IPv6 too tight, so upstream service connections over IPv6 suddenly fail? Oops!)

You're being very negative.

Yes, I am. IPv6 is crucial to the long-term health of the Internet. But its biggest driver (in most industries) is merely that it's the "new shiny", and not any real business-case. Even when there is a business case, it's typically for the network, not for the systems. IPv6 connectivity, *check*. IPv6ified webserver... come back next year. Maybe.

So, what if I do find it fun?

Seek professional help. Or, just admit you're a geek.

Some checks are more complicated again - "Does the page '/index.php' include the text 'Blog'?". A few even go so far as to simulate multiple page-loads, "clicking" along a path to ensure functionality.

Me, I'm too lazy for that. (Well, sometimes ;). A really sneaky way of checking that your app is working, is to measure the metrics around how much it's being used! How many concurrent users do you have, compared to normal? How many page/thread/post views per second do you currently have? If your site is broken in any way, these numbers are likely to drop off - even if your checks above find nothing wrong.

Let's just say that you have an average of 1000 concurrent users. If the site is broken, people won't hang around. They'll disappear, coming back later to see if you've fixed your problem - so maybe you'll only have 300 concurrent users. Set up an alert that you must have at least [say] 500 users, and alert if the number drops below that. Bingo, you have a new alert that will catch failures with very little effort on your part.

Naturally, most sites don't have flat traffic all week long. Maybe your off-peak traffic is only 400 users, and your peak traffic is 1000. There's no sensible single threshold that can alert you that there's a problem, while also tolerating the normal fluctuations that you're likely to see. The solution, for a lot of sites, is as simple as making your threshold into a sine wave. For this example, the function 300*SIN(time())+400 sits at a pretty nice threshold for my sample data:

Example concurrent-users-online statistic

How you actually tie this into your alerting system, is obviously dependent on your alerting system! Any system that allows custom scripting is likely to support this pretty readily. As a random example, I've succesfully tied it into munin's alerting by adjusting the "users.critical" config fragment to be echo'd out with a dynamic number driven from a formula similar to the above; munin re-read it on each evaluation and quite happily emailed or SMS'd me when things "weren't quite right".

False alarms are moderately rare, but certainly do happen - usually I found that things like public holidays (including in other countries :) would trigger it. It's far from foolproof, but nifty - and pretty darn easy to implement.