How to get MediaWiki running after upgrade from Debian 9 to Debian 10

My wiki on wiki.mkcs.at is a LXC container which was so far running Debian 9 and its host is also running Debian 9. I already set up a new host running Debian 10 (buster) and of course the containers running on it should run the same OS version.

After I copied the container from the old to the new host, everything was still working fine. But after I upgraded to Debian 10, suffering began, followed by some hours of tinkering. I would like to safe you that time and share my experiences.

The first service that failed to start was Apache. This could be seen directly in the output of apt full-upgrade:

apache2.service: Failed to set up mount namespacing: Permission denied

apache2.service: Failed at step NAMESPACE spawning /usr/sbin/apachectl: Permission denied

apache2.service: Control process exited, code=exited, status=226/NAMESPACE

As I read here, these two lines in /lib/systemd/system/apache2.service fix that problem:

PrivateTmp=false
NoNewPrivileges=yes

These commands then start Apache:

systemctl daemon-reload

systemctl start apache2.service

 

The next service that failed to start was mariadb and this was the one that was hard to fix. dpkg --list | grep -i mariadb showed that mariadb wasn't installed at all, so I installed it using apt install default-mysql-server. However, after installation, it did not run and systemctl status mysql showed

mariadb.service: Failed to set up mount namespacing: Permission denied

mariadb.service: Failed at step NAMESPACE spawning /usr/bin/install: Permission denied

mariadb.service: Control process exited, code=exited, status=226/NAMESPACE

The error was similar to the one I had with Apache but I could not quickly find a simple solution. Many search results on Google pointed to solutions with Proxmox or LXD that helped me to slightly understand the cause of the problem (somehow, AppArmor is preventing access) but not a solution. As I have seen in the output of dmesg, mariadb was not the only application that did not work:

audit: type=1400 audit(1621946328.497:5424): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=4240 comm="(install)" flags="rw, rslave"

audit: type=1400 audit(1621946331.481:5425): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=4241 comm="(vnstatd)" flags="rw, rslave"

On my LXC wiki page I once noted to add a line "lxc.aa_profile = unconfined" if strange permission problems occur in the container, but LXC then refused to start the container:

confile.c: parse_line: 2312 Unknown configuration key "lxc.aa_profile"

So, the option must have renamed in LXC 3. This page showed the right option:

lxc.apparmor.profile = unconfined

And finally mariadb was working again.

The next challenge was to get MediaWiki running. It first complained that it was missing "mbstring" and "xml" modules, so I installed them using apt -y install php-mbstring php-xml. Then I figured out that php7.3 and php7.3-mysql were not installed and I installed them too (apt -y install php7.3 php7.3-mysql). Next, MediaWiki complained it could not connect to the database:

no viable database extension found for type 'mysql'

The problem was that two versions of the mysql library were installed, as dpkg -S mysqlnd.so showed:

php7.0-mysql: /usr/lib/php/20151012/mysqlnd.so
php7.3-mysql: /usr/lib/php/20180731/mysqlnd.so

dpkg --list | grep "php7.0" showed that several packages belonging PHP 7.0 were still installed, so I removed them using apt remove php7.0 php7.0-cli php7.0-common php7.0-json php7.0-mysql php7.0-opcache php7.0-readline.

Finally I tested my wiki by logging in, modifying an article and see if it occurs in the "Recent changes" page. Everything worked smoothly. 😀

SSL_ERROR_RX_RECORD_TOO_LONG after upgrading Apache on Debian

Recently I upgraded a Debian server from Debian 8 (jessie) to Debian 9 (stretch) which also caused Apache to be upgraded from version 2.4.10 to 2.4.25. After the upgrade, when trying to connect to the web site using HTTPS, Firefox showed an error message with the code "SSL_ERROR_RX_RECORD_TOO_LONG". This error message is quite confusing, I thought that there would be something wrong with the certificate or the code that handles TLS. After searching a little bit in the Web, I figured out that Apache is actually serving HTTP on port 443 which doesn't work of course. The page that helped me was

https://community.letsencrypt.org/t/ssl-error-rx-record-too-long-on-debian-9-apache-2/123371

The command apachectl revealed that Apache was only aware of the virtual host configuration to serve on HTTP:

apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:80                   x.y.z (/etc/apache2/sites-enabled/000-default.conf:3)

It's interesting that Apache can't detect this misconfiguration somehow. Who wants to run a HTTP server on port 443 anyway?

Once I was aware of the problem, I tried to find out why it didn't read the virtual host configuration using TLS. The filename was /etc/apache2/sites-enabled/001-default-ssl and it worked fine with Apache 2.4.10 on Debian 8. So I took a look at /etc/apache2/apache2.conf and made a diff with the old configuration file. Then I saw this:

< IncludeOptional sites-enabled/*.conf
---
> Include sites-enabled/

While the old configuration told Apache2 to just read any file in sites-enabled/, the new configuration only processes files with ".conf" at the file name's end (which is clever IMHO). So what I had to do was to remove all the wrong links in sites-enabled, rename the configuration files in sites-available and add them again using "a2ensite". Finally, my web site was working again using HTTPS.

NetBSD 7 and XFCE in VirtualBox; lighttpd in NetBSD

As I found out in my last post about NetBSD 7 RC 3, there have been some problems with it and I wanted to try it out in VirtualBox. Finally, I have done that yesterday and I saw that NetBSD 7 performs quite well in VirtualBox. However, at first it was also not possible to set another background picture. The reason for that was that "xfdesktop" - which manages the desktop in XFCE - wasn't installed (which I have expected it to do when issued "pkgin install xfce"). The package's name in NetBSD is " xfce4-desktop". I tried to let it install by pkgin, but that failed, which showed me some lines in /var/db/pkgin/pkg_install-err.log:

---Nov 06 20:40:55: installing xfce4-desktop-4.12.3...
pkg_add: no pkg found for 'tdb>=1.2.10', sorry.
pkg_add: Can't install dependency tdb>=1.2.10

I installed that package "tdb" and afterwards, xfce4-desktop could be installed.  Since that moment it's possible to set another background picture in XFCE.

Other IMHO useful packages I installed:

firefox icedtea-web xpdf nmap iftop libreoffice-5* xfce4-screenshooter xfce4-weather* xfce4-cpugraph* xfce4-battery* xfce4-systemload-plugin gnome-screensaver gnome-audio gnome-backgrounds gnome-system-monitor gnome-system-tools xfce4-thunar sysupgrade sudo

For some experiments I also installed: lighttpd tuxpaint

Most of these packages worked, except gnome-system-monitor. gnome-about worked after I installed "py27-gtk". It's wise to start gnome-screensaver-settings once to setup the screen saver.

lighttpd didn't work after installing it. Unfortunately, pkgin doesn't copy any files necessary to actually run lighttpd (like apt-get does on Debian). This can be done with these commands:
cp -v /usr/pkg/share/examples/rc.d/lighttpd /etc/rc.d/
cp -v /usr/pkg/share/examples/lighttpd/lighttpd.conf /etc/lighttpd/

/etc/lighttpd/lighttpd.conf should be reviewed. Using

service lighttpd start

lighttpd can be started. Using

mkdir -p /srv/www/htdocs

it is possible to create the directory where files are being searched, unless this setting has not been changed in lighttpd.conf (see variable "server.document-root"). In my case, lighttpd now only serves files via IPv6, which can be seen if "netstat -f inet6 -a" is executed:

Active Internet6 connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp6       0      0  *.http                 *.*                    LISTEN
tcp6       0      0  *.x11                  *.*                    LISTEN
tcp6       0      0  *.ssh                  *.*                    LISTEN
tcp6       0      0  *.sunrpc               *.*                    LISTEN
udp6       0      0  fe80::1%lo0.ntp        *.*                  
udp6       0      0  localhost.ntp          *.*                  
udp6       0      0  2001:470:1f0b:2e.ntp   *.*                  
udp6       0      0  fe80::a00:27ff:f.ntp   *.*                  
udp6       0      0  *.ntp                  *.*                  
udp6       0      0  *.*                    *.*                  
udp6       0      0  *.1023                 *.*                  
udp6       0      0  *.sunrpc               *.*                  

If "inet" is used instead of "inet6", no "http" is listed.

First look at NetBSD 7 RC 3 and XFCE

The NetBSD team released RC 3 of NetBSD 7 this week and I was just too curious about its progress, so I downloaded both the i386 and x86_64 ISOs. I set it up in a virtual machine that I virtualised using Qemu 2.0.0 on my Xubuntu 14.04 system. First I had some trouble with the installer. I unaccidently selected a wrong option and then cancelled with Ctrl+C. Although I finished the installation normally, the system didn't boot correctly but dropped me on a shell with the hint, that /etc/rc.conf would contain RC_CONFIGURED=NO. I had no clue at that time how to solve that, especially how to mount the root filesystem read/write. In the meantime I found out that it isn't that hard: "mount -u -w /" would do the trick.

However, I just started the installation again (which is really fast and takes just a few minutes) but I had yet another problem: When trying to setup pkgin, that failed because it said it couldn't download something from "http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/7.0_RC3/". In fact, that directory doesn't exist. I realised that there is path to "7.0" (without the "_RC3") and changed the path accordingly. The installer warned me that at least one package was build for RC 2 rather than RC 3 but I decided to accept this and the installation went fine.

After installing the "usual suspicous" packages (bash, nano), I installed XFCE and was very delighted that it made a huge jump from version 4.6 in NetBSD 6 to 4.12 in NetBSD 7. A first trial was a little bit disappointing: The performance was quite poor, the mouse pointer disappeared and appeared again continuously and there was a background with vertical black and white stripes in contrast to what was set up in the settings dialog.

Figure 1: XFCE 4.12 in NetBSD 7 RC 3 with wrong background

To demonstrate this, I made a short, two minutes video.

If I have the change to do it, I will try installating NetBSD 7 RC 3 inside VirtualBox and see if it performs better there.

"No space left on device" while executing apt-get/dpkg: it's just inodes...

Today I wanted to install some software using "apt-get install", when it suddenly reported that there was no space left on a device. df -h reported me, that my root file system was almost full, so I looked what I could do and finally moved some big files to another partition. Although my system had then plenty enough free disk space, "apt-get" kept reporting low disk space. So I asked DuckDuckGo and Google for help and got it: Apparently even a file system with enough free space can be full - if it ran out of inodes! df -i then showed me, that almost 100% of all inodes were used. So I was curious what caused those troubles. On [1] I found a quite useful command which I even "pimped" a little bit:

$ for i in /*; do echo $i has `find $i | wc -l` files; done

which outputs something like

/bin has 153 files
/boot has 262 files
and so on.
Or use

$ for i in /*; do echo `find $i | wc -l` files in $i; done

if you would like to have the numbers first:

153 files in /bin
262 files in /boot

Piping the output to "sort -n" does sorting, but you have to wait for the command to complete:

$ for i in /var/*; do echo `find $i | wc -l` files in $i; done | sort -n
1 files in /var/crash
1 files in /var/local
...
600 files in /var/log
11066 files in /var/lib

Using that command, I found out that "/usr" has many files, so I replaced "/*" by "/usr/*" and figured out that /usr/src was the really bad directory on the file system. I didn't notice yet that removing a Linux kernel doesn't remove the according headers. This must be done manually. To see all currently installed header packages, it's possible to use

$ dpkg --list | grep linux-headers

Those packages have really huge numbers of files! For Linux 3.2.0-80, there are two directories in /usr/src: "linux-headers-3.2.0-80" and "linux-headers-3.2.0-80-generic" which together have 22042 (!) files! And I had 28 of those on my system...

Conclusion: After "apt-get remove linux-headers-3.2.0-24 linux-headers-3.2.0-25-generic linux-headers-3.2.0-26-generic..." I had 167098 free inodes, which should be safe for the upcoming weeks. In future, I will monitor this value to avoid troubles.

[1] http://www.ivankuznetsov.com/2010/02/no-space-left-on-device-running-out-of-inodes.html

Moving Active Directory integrated DNS zones from Windows 2000 Server to bind9

Since some weeks I am in the act of moving services from my virtual Windows 2000 server to my Ubuntu 12.04 machine. The first step was moving all files out of it so they are saved on my Ubuntu machine.

This post is about moving the Active Directory integrated DNS zones from Windows 2000 to bind9. First, a little bit about my current setup: 192.168.1.1, named speedy, offers DHCP and DNS services, while 192.168.1.2, named gspdc, offers Active Directory services and therefore also acts as a DNS server. Requests belonging the AD integrated domain (greatsoft.local) are forwarded from speedy to gspdc. This KB article from Microsoft shows an easy way to get all AD integrated DNS zones in files so that they could be processed by bind9. The easy trick is to simply convert those zones to "primary zone files": Afterwards, they are saved in the default DNS directory (that is normally %WINDIR%\System32\DNS). To get them to Ubuntu, I use SMB, so I first activated sharing the directory as "dns". On Ubuntu, I mounted the share:

root@speedy:/mnt # mount -t cifs //192.168.1.2/dns temp -o username=Administrator

Then I copied the files to /etc/bind9:

root@speedy:/mnt # cp -v temp/*dns /etc/bind

To make the database file names compliant with the standard of bind9, I renamed all of them and gave them a "db." prefix like all other database files have.

Until that moment, I used forwarding in bind9 to get DNS requests for *.virtual, *.greatsoft.local (AD integrated) and reverse DNS requests belonging 192.168.0.0/16 being resolved by Windows 2000.

root@speedy:/etc/bind # cat named.conf.gspdc
zone "greatsoft.local" {
    type forward;
    forwarders { 192.168.1.2; };
};

zone "virtual" {
    type forward;
    forwarders { 192.168.1.2; };
};

zone "168.192.in-addr.arpa" {
    type forward;
        forwarders { 192.168.1.2; };
};

That's not necessary any more: now the zone files are available in /etc/bind. In that directory, I edited named.conf and commented out the reference to named.conf.gspdc. The replace these zones, I created three files: named.conf.greatsoft.local, named.conf.virtual and named.conf.192.168.1.x.subnet which have contents like the following:

root@speedy:/etc/bind # cat named.conf.greatsoft.local
zone "greatsoft.local" {
        type master;
        file "/etc/bind/db.greatsoft.local.dns";
};

In contrast to the previous situation, I now have separate files for every /24 of 192.168.0.0/16, so reverse DNS queries for 192.168.1.0/24 are resolved by a different file than those for 192.168.2.0/24. I think that's easier to manage, because otherwise the zone database file could get too large.

Next task it to add those new named.conf.* files to the main configuration file, named.conf:

include "/etc/bind/named.conf.192.168.1.x.subnet";
include "/etc/bind/named.conf.greatsoft.local";
include "/etc/bind/named.conf.virtual";

Now it's time to restart bind9 by "service bind9 restart" and troubles finally begin. A simple "nslookup www.mkcs.at.virtual 192.168.1.1" just returns "** server can't find www.mkcs.at.virtual: SERVFAIL". But a "host -t TXT test.virtual" returns the text that I entered for it once (just for test purposes, as the name suggests), the same is true for my test entries for "test.test.virtual" (type A) and "eee1.test.virtual" (type AAAA). Using "grep named /var/log/syslog | tail -30" I found an interesting line:

/etc/bind/greatsoft.local.dns:33: gc._msdcs.greatsoft.local: bad owner name (check-names)

That is line 33 of file greatsoft.local.dns:

gc._msdcs               600     A       192.168.1.2

It's yet unclear to me why this causes an error. However, that entry was used by Active Directory, which I am about to deactivate, so I can relinquish entries like that. After uncommenting this line and the following lines that belong to that entry "gc._msdcs", I use

service bind9 reload

to tell bind9 it should reload its configuration and voilà, it works (tested from a client PC in my network):

michael@michimain1404:~$ host -t A www.mkcs.at.virtual
www.mkcs.at.virtual is an alias for w3mkcs.greatsoft.local.
w3mkcs.greatsoft.local has address 192.168.1.64

That's it! The final steps in deactivating that old (virtual) system will be shutting it down, (finally!) uninstalling VMware and then move it to a safe place in case I still need some data from it.

If anyone has suggestions for me how to improve that process, just let me know, I'd be curious!

Rusch Wanduhr von IKEA mit eigenem Ziffernblatt

Vor vielen Jahren hatten wir uns mal beim IKEA die Wanduhr "Rusch" gekauft. Seit Dezember 2012 hing sie in unserem Badezimmer. Die öfters Mal feuchte Luft hat jedoch scheinbar dem Ziffernblatt nicht so gut getan. In den letzten Wochen wurde das Ziffernblatt immer mehr spröde und fing nach und nach an, sich aufzulösen.

Daher habe ich kurzerhand in LibreOffice Draw ein neues Ziffernblatt entworfen, was ganz einfach war: Der Durchmesser des Ziffernblatts beträgt 18 cm, also habe ich in Draw einen Kreis mit 18 cm Durchmesser gemacht, mit einem radialen Farbverlauf gefüllt und die Ziffern von "1" bis "12" in Schriftgröße 44 entsprechend positioniert. In die Mitte habe ich noch ein liebes Foto von Tobias und Jana gegeben. Dann noch ausgedruckt und ausgeschnitten: Fertig war das Ersatzziffernblatt. :-)

Nun war es noch notwendig, dieses in die Uhr zu bekommen: Auf der Rückseite der Uhr sind 3 Einkerbungen, die man mit einem Schlitzschraubendreher leicht öffnen kann, sodass sich die Verdeckung der Uhr ablöst. Tobias hat mir dann geholfen, das alte Ziffernblatt so gut wie möglich rauszubekommen. Es waren nur mehr hunderte Brösel. ;-) Dann habe ich vorsichtig zuerst den Sekundenzeiger, dann den Minutenzeiger und zum Schluss den Stundenzeiger abgenommen und zur Seite gelegt. Das neue Ziffernblatt habe ich mittig positioniert und dann einfach hineingedrückt. Dann habe ich die Zeiger wieder auf die entsprechende Vorrichtung gedrückt, die Abdeckung raufgedrückt, die Batterie wieder eingesetzt und die Uhr gestellt.

Und so sieht das Ganze aus:

Nun haben wir im Badezimmer wieder eine aktuelle Uhrzeit. :-)