DAVx5 CalDAV may break with Unicode symbols in Horde/Kronolith syncs.

If you get user complaints about broken CalDAV syncs with Horde, there’s many places to look at. In one particular instance, an event was created from travelling app Transportr into the stock android Calendar app. Through the DAVx5 sync app, the user wanted to push these events to Horde’s SabreDAV interface – and from there, also sync it to his desktop email solution, Mozilla Thunderbird.

However, his sync application told him about an error. The server administrator saw a 500 status code in the server log. - user [24/Jan/2021:18:48:26 +0000] "PUT /horde/rpc/calendars/user/calendar~KL14jYhCMpbet4ecYzAg1bn/2bace303-f0d8-4df6-9652-baa5fb9e86c4.ics HTTP/1.1" 500 892 "-" "DAVx5/3.3.8-ose (2021/01/13; dav4jvm; okhttp/4.9.0) Android/11

The root cause was actually not in the software code but in the MariaDB database configuration. The calendar entry from Transportr included some Unicode icon characters like a fast train and some arrows. These characters are part of the standard unicode encoding, utf-8.

Now you might wonder: New installations of mysql and MariaDB default to a character set they call utf-8 since 2010 or so. This shouldn’t be an issue. However, what they call utf-8 is not what you would expect.

Some years ago, TV sets which did not support the full HD resolution were marketed as „HD ready“. In some sense, the default character set is „unicode ready“ at best. The default data type saves on disk space by encoding a subset of utf-8 into up to three bytes. While this supports most natural language characters, it is only a fraction of what Unicode can offer. Database manufacturers are well aware that this is not something you should run nowadays that unicode icons like the hamburger are all over the place is user generated content. The mysql manual even says:

„Please use utf8mb4 instead. Although utf8 is currently an alias for utf8mb3, at some point utf8 is expected to become a reference to utf8mb4. To avoid ambiguity about the meaning of utf8, consider specifying utf8mb4 explicitly for character set references instead of utf8.“

Now that’s what I did. First I changed horde’s database encoding to utf8mb4 in conf.php:

$conf['sql']['protocol'] = 'tcp';
$conf['sql']['charset'] = 'utf8mb4';

Then I changed the mysql server’s and client’s default charset:

cat /etc/mysql/my.cnf
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
default-character-set = utf8mb4
default-character-set = utf8mb4
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

I have obviously stripped a lot of file content not relevant to our story. After reloading the database, all new connections should use the real utf8 encoding and new tables should be created with the new standard. But what about existing content? We need to convert all tables and all their text-like columns, varchars, mediumtexts etc.

First, it’s backup time – better safe than sorry.

Then, let’s find all tables in our db server and feed them conversion commands.

mysql --database=horde -B -N -e "SHOW TABLES" | awk '{print "ALTER TABLE", $1, "CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"}' | mysql --database=horde

Do this in a downtime or ensure HA some way or other. It will take some time. Some sources suggest it might be sensible to also rebuild the tables with the optimize command. I am not very convinced, but it won’t harm.

mysqlcheck --auto-repair --optimize --user="hidden" -p --databases mysql horde

After this, repeat the sync test. It should work this time.

2 Kommentare

  1. Stephan Richter

    Wenn man jetzt noch anfangen würde, utf-8 unter Windows sinnvoll zu unterstützen…

    1. Ralf

      Hi Stephan,

      „man“ davx5 oder „man“ Windows oder „man“ horde oder wer „man“?



Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.