darxus: (2014-03-27)
darxus ([personal profile] darxus) wrote2015-02-03 04:09 pm
Entry tags:

[geek] initializing DST: openssl failure / upgrading ubuntu from lucid to precise broke DNS

I upgraded my linode server from ubuntu distro lucid to precise today. I didn't have any specific needs, but lucid is getting EOLed in a couple months.

It broke my DNS server. This is what ended up fixing it:
mkdir -p /usr/lib/i386-linux-gnu/openssl-1.0.0/engines
cp -a /usr/lib/i386-linux-gnu/openssl-1.0.0/engines /var/lib/named/usr/lib/i386-linux-gnu/openssl-1.0.0/
Where /var/lib/named was where I had bind chrooted to.

I noticed DNS didn't seem to be working. I ran:

# /etc/init.d/bind9 start
* Starting domain name service... bind9

Then I checked /var/log/daemon.log, and noticed:
Feb 3 13:35:36 panic named[23991]: initializing DST: openssl failure
Feb 3 13:35:36 panic named[23991]: exiting (due to fatal error)

The useful way to find out what the problem was from there was:
strace -o named -ff named -u bind -t /var/lib/named

The "strace" command tells you, among other things, what files a program tries to read. "-o named" says to write to file "named", and "-ff named" handles forking into multiple threads, creating a file for each. "-u bind -t /var/lib/named" came from my /etc/default/bind9 :
OPTIONS="-u bind -t /var/lib/named"

"-u bind" means to run as user bind, and "-t /var/lib/named" means to run via chroot in /var/lib/named.

panic:~/tmp# grep -i ssl *
named.23906:open("/usr/lib/ssl/openssl.cnf", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
named.23906:open("/usr/lib/i386-linux-gnu/openssl-1.0.0/engines/libgost.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

So there were some ssl related files it was trying to load and not finding. openssl.cnf was apparently not required.

I tried doing this via mount -bind in /etc/fstab, but it turns out you can't do that readonly. I added this to /etc/fstab:

/usr/lib/i386-linux-gnu/openssl-1.0.0/engines /var/lib/named/usr/lib/i386-linux-gnu/openssl-1.0.0/engines none bind,ro

But that commented that it got mounted writeable. Because... bind mounting doesn't have a way to do readonly.

And so I copied it. And will, apparently, need to forever maintain that manually. There's probably a better way. Haven't figured that out yet.

I also had some linode specific problems (login prompts went away), which were fixed by enabling "xenify distro".
drwex: (Default)

[personal profile] drwex 2015-02-04 06:39 pm (UTC)(link)
Wow, your geek fu is mighty. I could mostly figure out what you did from parsing the command lines but would never have been able to do it myself.

[identity profile] darxus.livejournal.com 2015-02-04 07:13 pm (UTC)(link)
Thanks. A lot of it was just copying and pasting errors into web searches. I should've thought to strace named myself, but I didn't. And I didn't know the syntax for following threads off the top of my head.

I think this really should've been mentioned in the known issues in the release notes, or the apt-listchanges output, all of which I actually read thoroughly before upgrading.