No more systemd on server.

Some time ago I’ve tried to solve a ‘funny’ issue with PowerDNS. It refused to start on one of my backup VPS. After apt-get dist-upgrade, it first stopped to work with very meaningful ‘Failed at step ADDRESS_FAMILIES‘  error. Well, I’ve managed to solve it somehow, but then another ‘funny’ problem popped out…

Something was already listening on port 53. WTF ? I’ve not installed any other DNS server, no resolver, nothing… Guess twice, systemd programmers (once again, programmers the one that make programs, the one that don’t care about system as whole…) decided to re-implement one more service that already works well, and unfortunately the roulette stopped this time on ‘dns resolver‘. And I’ve installed it, because I’ve issued that stupid ‘apt-get dist-upgrade‘. And my perfectly working server was ruined, existing dns resolving via external NS servers has been replaced with local resolver (why, it has been running this way for years without problem – so why ???), and my PowerDNS server has been explicitly ruined first by invalid systemd startup script for PowerDNS and afterwards with systemd-resolved occupying its port. Strike ! One dist-upgrade and any DNS related functionality of my VPS was down.

Yes, it’s my fault ! I should have expected that apt dist-upgrade installs and starts some new network service, rewrites my resolv.conf, overwrites already fixed systemd start config… It has been doing that for years or ? Well, I’ve not seen something like that in last 15 years, but it must be normal.

And the best thing about this ? I’ve tried to get rid of systemd from my debian installation and I failed and I killed my VPS. It’s not booting anymore 😉

Lessons learned:

  • Don’t use systemd on server – IT’S UNPREDICTABLE, you never know where the roulette ball stops as next (systemd-httpd, systemd-filesystemd, systemd-libc, systemd-bash, systemd-ldapd, systemd-smtpd, systemd-docker…) and which service will be killed next
  • Don’t install Debian 8 on VPS, when there is no Gentoo option. Start with Debian 7 and upgrade incrementally until systemd tries to sneak in
  • Backup your backup VPS more regularly, even if it’s only a backup server only one monthly backup is not enough. Systemd can strike anytime !