bookmark_borderA wicked problem from the past

In the last few evenings there was great despair. Trying to solve one problem would reveal yet another one. As a result, I hesitated to release some of the changes immediately. I don’t want to make people suffer, having to deal with new improved problems of the day whenever they run an update on their horde installations. I’m glad feedback on the mailing list has improved quite a lot, sometimes coming with patches and suggestions and even people telling me things start to just work. That’s great. But whenever you see the light, there’s a new challenge on the horizon. Or rather something very old lurking in the shadows.

Break it till you make it

So recently we felt confident enough to switch our local installation from a frozen state to the latest version of the wicked wiki and the whups bug tracker. We also updated the PHP container, the database version and the likes. This turned into a great opportunity to test the rollback plan. 🙄 Issues were cascading.

Bullets you might have dodged

Generally having both the composer autoloader and the horde autoloader work together creates some friction. This is complicated when the exact order in which they are initialized is not the same in all scenarios. As reported previously, Horde’s apps are not strictly conforming the PSR-0 autoloading standard while the libraries mostly do. Newer releases of the apps autoload using a class map instead of PSR-0 logic. But that class map was not restricted enough in the first iterations. Thus you might have the canonical locations of classes in the autoloader, but also the version in the /web/ dir or, in case of custom hook classes, the version from the /var/config/ directory. Torben Dannhauer suggested to restrict the rules and we will do that with the next iteration. The first attempt to fix this unfortunately broke overriding the mime.local.php config file. An update is in the git tree and will be released as a version later this week. But I’m glad we ran into this as it revealed another dark secret

We part, each to wander the abyss alone

Turned out the wicked wiki software carried a little library Text_Wiki in its belly. Hailing from PHP 4 days, it’s archaic in its structure and its way of implementing features. Parts of the test suite predate phpunit and the way it’s packaged is clearly not from our times. This library also exists as a separate package horde/text_wiki. Which also exists as pear/text_wiki both in the classic PEAR archive and the modern github repository. While the base package does not exist in packagist, the extension driver for mediawiki format does. Odd enough. It’s really a shame the software from the old PEAR ecosystem is slowly fading away because of its ossification. They all share ancestry and they were all largely written and maintained by the same set of people but they are all different in subtle ways.

Text_Wiki works great when it works. But it’s a real treasure trove of incompatibilities with modern PHP versions. Over the years, unicode support has evolved along with the strictness of PHP’s core. While I am very much willing to contribute back any changes to the official pear version, I have my doubts if they will be accepted or handled at all.

Rising again like the phoenix

I really, really did not want to deal with any of this. Creating a fourth version out of three splinters feels like madness. But what can you do?
The result is an upcoming version of horde/text_wiki and horde/wicked which is a radical break. The new text/wiki has no PSR-0 lib/ content. It’s all PSR-4, it’s namespaced, every single class name and file name has changed. A new test suite has been started but only a tiny fraction of unit tests have been ported so far. The different extension drivers for mediawiki and tiki dialects have been incorporated into the base package as I did not want to maintain a hand full of packages.

The whole thing couples as little to horde as possible. It has its own exceptions based on the PHP builtin exception tree, so no dependency on horde/exception. I even stripped the dependency on Pear_Exception as there is no value in keeping that nowadays. Anything which ties this revamped library into the horde ecosystem now lives in horde/wicked. Extra markup rules which render horde portal blocks into a wiki page. Adapters querying the database if a linked page already exists or must be created. Dynamic links created from the registry and application states. None of this glue belongs into the text_wiki library.

Many incompatibilities with modern PHP have been solved. Often this was a matter of mixing and matching bits from the three different splinter versions out in the wild. Some of it is just a chore to do. Of course, the pear XML files are gone and the composer file is fully compliant with packagist and most recent composer. At least this part has been contributed back already.

Dawn, finally

It will be another few evenings until the new horde/wicked and the new horde/text_wiki are ready for a tagged release, probably along with some of the changes to other libraries I explained above. There’s probably something that will break and need fixing. But that shouldn’t block progress for too long.

bookmark_borderProxy Mode in horde-installer-plugin

Horde 5 relied on the PEAR ecosystem ripping apart their packages and putting everything where it needs to be. But PEAR is a thing of the past. Horde 6 integrates with the composer installer using an additional piece called the horde/horde-installer-plugin. This composer plugin links together pieces from the installed composer packages into the right places and adds auto-generated configuration to make sure everything is found even though stuff is NOT mixed together the way PEAR did. This was a good starting point early in Horde 6 development when we took great pains not to break the previous distribution installs (which relied on PEAR or PEAR like behaviour) and installs based on horde-git-tools.

However, by now this is much less a priority and we are absolutely willing to move forward and make things less complicated, potentially breaking and changing code that relied on these older schemes. Much less code and assets are going to be exposed through the /web/ folder.

Proxy Mode vs Symlink Mode

Version 2.7.0 of horde/horde-installer-plugin introduces a new switch to the horde:reconfigure command:

composer horde:reconfigure --mode=proxy
  • In proxy mode composer will not symlink whole apps to the web dir but only create folders and files outside lib/, src/, test/ and a fairly large exclusion list. The endpoint php files will forward requests to the actual code in /vendor dir. This should reduce code duplication and makes it more obvious where related files need to be looked up
  • A new constant HORDE_CONFIG_BASE is written to the autogenerated registry snippets.
  • The composer autoloader is directly loaded by the proxy php files before delegating to the actual file in the vendor dir.
    If you see unexpected class not found messages when testing proxy mode, please upgrade your horde/core package.

Currently proxy mode is a one-off. However, in an upcoming version the last used mode will be persisted to the composer.json file and reused on subsequent commands without an explicit mode option. Proxy mode is supposed to become the default in horde-installer-plugin 3.0.

Benefits of proxy mode

Proxy mode is the first step towards clarifying the intended architecture. Until recently much of our detection logic needed to behave differently if called through a file in the webroot or the same file but in its original vendor/horde/$app location. The new proxy files always perform the same way:

  • First load the composer autoloader because the file always knows where the autoloader is relative to its own path.
  • Then load the actual file in the vendor/horde/$app location
  • Horde’s own internal PSR-0+ autoloader would only be loaded after composer’s autoloader. It hits action less and less often and will probably be removed from the default setup altogether.

This means the actual class files can rely on __FILE__ pointing to a predictable scheme and much less guess work and protection code is needed. On the other hand, it reveals some hidden problems and complications which developers can now fix before making this new mode the default. Unexpected lowercase class names (as found in whups and Horde Forms), complicated chicken/egg issues in the Registry, duplicate class names and parts of the error handling broken by subtle changes in newer PHP versions can now be seen and fixed.

As a consequence of spotting bugs and loading issues as I go along, starting with version 2.7.1 the composer plugin will check if a file var/config/autoload-extra.php exists. If it does, it will be included after the composer autoloader. This allows selectively hardcoding some class files until fixed versions of apps become available – or to actively undefine something for tests.

bookmark_borderUnit Testing Horde 6

Some years ago I reasoned about upgrading unit tests from ancient PHPUnit 4 to then-recent PHPUnit 9.
Back then Horde’s unit test suite would use a Horde_Test class inheriting from the actual phpunit test cases. Even then I was fairly certain that this approach was not very practical in the long run.
Why extending PHPUnit might be wrong – ralf-lang.de

Turns out this is right. With many new or refactored components I started to use plain PHPUnit 9 and only resorted to Horde_Test when I strictly needed it. This simplified things a lot. The Horde organisation houses a dozen or so apps, almost 200 library repositories and some auxillary stuff. Having Horde_Test in the inheritance hierarchy makes any upgrades awkward. Need to be more strict about signatures for a new PHP release? Fix it all at once. Want to adopt a newer phpunit framework? Do it wholesale. Hook into some internal PHPUnit mechanism? They told us not to and they use their freedom to break internal classes very liberally. Tough luck.

So instead I am mostly abandoning the Horde_Test_Case base class and everything that comes with it.

  • PHPUnit hooks right into the composer autoloader. No more AllTests.php, autoload.php or bootstrap.php
  • I use the PHPUnit version I see fit. If I rarely update a component and it’s supposed to run on old PHP 7.4 through PHP 8.4, then I am fine when the tests work with PHPUnit 10.
  • If I develop something bleeding edge which does not need to work on PHP 8.2 and older, I simply start out at PHPUnit 12.
  • If I want to abstract out a test helper which is useful beyond the limits of an individual repo, I try to avoid building new bases classes. Instead I build it as a utility

Many libraries which get a wholesale upgrade no longer depend on Horde/Test. This is good because Horde/Test pulls in too many implicit dependencies anyway.

This comes at a price though. Our custom test runner relied on a specific structure of horde tests. The internal quality check pipeline in the horde-components tool also relied on a specific phpunit version and library layout. These are now broken and I won’t try that approach again.

So the new layout looks fairly standard.

  • Tests are in the /test/ subdirectory
  • Unit tests which are intended to run on most platforms are in /test/unit. Tests which require external resources, expensive calculations or rarely used PHP extensions get their own lowercase test suite directory
    /test/ldap/
    /test/pam/
    /test/kolab/
    /test/integration/
  • The horde-components tool autodetects these test suites and sets up the appropriate PSR-4 autoload-dev rules in the composer.json file
  • Just call “phpunit” to run the test suites.