ΔΕΞΑΙ!

We have a boat load of servers that need to be gotten to. If you manage any number of servers, you've more than likely looked at centralized authentication.

We've been using LDAP to store account information, and kerberos for the authentication bit. We ran into two problems with this. The first problem we ran into is that we have servers still running that survived the jurrasic period. This is a problem, because stable production type operating systems from the jurassic period still run with a nice old verison of kerberos.

This brings us to the second problem. Our servers don't live in a magic happy network where everything is proper and hosts live by one name with proper reverse DNS. Our servers are loaded to the gills, and often serve more than one thing under different names. We also use esoteric host names to identify servers within the data center. Nobody like using these names, so they get friendlier aliases...

Herin lies the problem with being old. Kerberos 1.7.x totally digs aliases when used with LDAP. Old kerberos hates them and despises them. Say you have a server named sf4n136.domain.com which also known as gateway23.domain.com. Typically you would want to log into gateway23, so you'd type 'ssh gateway23.domain.com'. If your ssh is sane, it will dutifully ask the kerberos server for a service ticket for gateway23. The kerberos server will see that and say, "huh, who?". It wants you to log in with the real host name of sf4n136. To make this even more problematic, some old ssh versions patronizingly do a reverse DNS lookup after resolving the IP, and will use whatever it gets back as the hostname. Obviously that's what you really wanted, you just didn't know any better!

So what do you do? You install a kerberos server that handles aliases. You then put the aliases in the proper entry of your LDAP server database. Kerberos up to very recently required the client to specifically ask the KDC to check if the service you are requesting could be an alias. If you didn't ask, it wouldn't bother returning aliases. The problem with this approach is that old clients don't bother with all that new fangled alias garbage. They don't even know what they are. Back when dinosaurs ruled the data center, servers who called themselves by more than one name were eaten.

One option is to patch the KDC so it unconditionally returns canonicalized service entries (fancy talk for resolving aliases). Ok, so now you ask for gateway23, and the server sees that it is really sf4n136, so it just whips up a ticket encrypted with sf4n236's password, winks at you and labels it gateway23.

Here's the rub with that. The server you hand that ticket to has to figure out what the heck you're trying to use, so it can check if you are allowed to use it. It does this by looking at the service name on the ticket you handed it, and tries to decrypt the ticket with the corresponding keytab entry for that service. Unless you've setup keytab entries for every possible name that might be used to get to that server, you're sunk. Then you might as well forget aliases, and setup some insane system for keeping a jazillion keytabs up to date.

New kerberos to the rescue! 1.7.blah knows that the KDC has been doing favors for you, and doesn't really trust the name on the ticket. It knows that the real KDC is trustworthy, so if you really do know the real KDC, the ticket you gave it probably matches something in its keytab. If it can't find that particular name, it just loops through everything it has, and tries to decrypt the ticket. Once it finds something that works, it figures out what secret arrangement you had with the KDC and lets you into the corresponding service.

Still, there's another problem. ARGG!! There's always another problem. Kerberos has changed a lot since these servers where born. You can't just update kerberos, since 3/4 of the entire freekin operating system seems to depend on the old version. Here is where we are blessed. We don't use kerberos4, and all that stuff depends on kerberos4. This means when we install a newer kerberos, we can just leave bits of those old libraries laying around. When those old programs dance with the linker, they won't know the difference, maybe...

There is one bit in the kerberos5 API that changed. Fortunately, it was just an initialization function that is no longer needed. That means we can just write that function in, have it do nothing, and the program will be blissfully unaware.

So that's what I did. I rolled an RPM that works on all our versions of Centos by grabbing bits and pieces it needs from old kerberos and patching the source. Then I compiled for i386 and x86_64, threw the RPMs in our custom repo, and viola. The next day all the servers had new shiny Frankenkerberos. It's a conglomeration of old and new parts, and it's alive! Didn't even have to subject the servers to static shock.