bookmark_borderHorde 6: Return of the Git Tree

Over the last few weekends, Horde 6 code has been merged back from the Maintaina fork and from separate contributions to the former Horde development version, “master”. It was time to upgrade the development tool chain.

Back in the Horde 5 days, there was a utility called git-tools developed by Michael Rubinsky. It would checkout all horde library and application repositories of the github organization into a development directory – also called the git tree – and link into into another structure that resembled a regular Horde 5 installation – called the web tree. This required to hardcode some configurations what could otherwise be auto-detected but it allowed to organize the horde code base in a developer friendly flat structure and provide a ready to run test scenario that reflected the latest code changes even before they were committed to the official repositories. In the days of composer, half of what git-tools does is already covered by composer itself. The git-tools shell already had an integration for the horde-components tool which manages releases, change log, housekeeping tasks and some more.

It was obvious that I wanted a similar functionality for the composer-based setups. It took a while to get this right. The good news is I achieved that. Bear with me for another paragraph of incremental learning or skip to the next section with the results.

It’s getting better all the time

Originally I integrated a generator for “vcs” type repositories into the composer.json generation code. But these vcs repositories are relatively slow. This is OK for one or two apps or leave packages but it’s really really slow when you have a list of 80 or 120 dependencies each pulled from a separate vcs repository.

The next iteration involved generating a satis repository as a standin to releasing code into packagist. The satis repository would be updated by any commit or accepted pull request in any of the serviced repositories. Generating the satis repository is not particularly fast but I managed to scope refreshs to the individual component that needed updating. Reading from satis however is way faster than the vcs approach. Keeping an installation up to date with the latest development commits became much more viable. I also figured I could replace an already installed dependency with an actual git repository and composer would update the autoloader as needed. This works as long as you don’t change the autoloader rules, i.e. upgrade a package from PSR-0 to PSR-4.

Back to #1

Checking out individual libraries as root components and doing integration tests in the satis setup worked OK when the focus was on isolating individual pain points and solving bugs. For any development spanning changes on multiple libraries, it did not work out too well. I ended up implementing a new command.

horde-components github-clone-org

This will checkout a developer tree of git repositories containing apps, libraries or themes.

total 752
drwxr-xr-x 188 root root 4096 Jan 12 11:04 ./
drwxr-xr-x   4 root root 4096 Dec  4 17:44 ../
drwxr-xr-x   9 root root 4096 Jan 11 07:50 ActiveSync/
drwxr-xr-x   9 root root 4096 Jan 11 07:50 Alarm/
drwxr-xr-x   8 root root 4096 Jan 11 07:50 ApertureToAnselExportPlugin/
drwxr-xr-x   9 root root 4096 Jan 11 07:50 Argv/
drwxr-xr-x   9 root root 4096 Jan 11 07:50 Auth/
drwxr-xr-x   7 root root 4096 Jan 11 07:50 Autoloader/
drwxr-xr-x   7 root root 4096 Jan 11 07:50 Autoloader_Cache/
drwxr-xr-x   7 root root 4096 Jan 11 07:50 Backup/
drwxr-xr-x   7 root root 4096 Jan 11 07:51 Browser/
drwxr-xr-x   9 root root 4096 Jan 11 07:51 Cache/
drwxr-xr-x   9 root root 4096 Jan 11 07:51 Cli/
drwxr-xr-x   8 root root 4096 Jan 11 07:51 Cli_Application/
drwxr-xr-x   8 root root 4096 Jan 11 07:51 Cli_Modular/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Compress/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Compress_Fast/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Constraint/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Controller/
drwxr-xr-x  11 root root 4096 Jan 11 07:52 Core/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Crypt/
drwxr-xr-x   7 root root 4096 Jan 11 07:52 Crypt_Blowfish/
drwxr-xr-x   6 root root 4096 Jan 11 07:52 CssMinify/
drwxr-xr-x   8 root root 4096 Jan 11 07:52 Css_Parser/
...
drwxr-xr-x   7 root root 4096 Jan 11 08:17 Xml_Element/
drwxr-xr-x   7 root root 4096 Jan 11 08:17 Xml_Wbxml/
drwxr-xr-x   7 root root 4096 Jan 11 08:17 Yaml/
drwxr-xr-x  14 root root 4096 Jan 11 07:50 agora/
drwxr-xr-x  17 root root 4096 Jan 11 07:50 ansel/
drwxr-xr-x  21 root root 4096 Jan 11 07:51 base/
drwxr-xr-x  13 root root 4096 Jan 11 07:51 beatnik/
drwxr-xr-x   7 root root 4096 Jan 12 21:33 bundle/
drwxr-xr-x  12 root root 4096 Jan 11 07:51 chora/
drwxr-xr-x  15 root root 4096 Jan 13 16:54 components/
drwxr-xr-x  11 root root 4096 Jan 11 07:52 content/
drwxr-xr-x   2 root root 4096 Jan 11 08:02 dev.horde.org/
drwxr-xr-x   3 root root 4096 Jan 11 08:02 dns/
drwxr-xr-x  15 root root 4096 Jan 11 08:03 folks/
drwxr-xr-x   6 root root 4096 Jan 11 08:03 git-tools/
drwxr-xr-x  12 root root 4096 Jan 11 08:03 gollem/
drwxr-xr-x   9 root root 4096 Jan 11 08:03 groupware/
drwxr-xr-x  13 root root 4096 Jan 11 08:03 hermes/
drwxr-xr-x   6 root root 4096 Jan 11 08:08 horde-installer-plugin/
drwxr-xr-x   7 root root 4096 Jan 11 08:08 horde-support/
drwxr-xr-x   9 root root 4096 Jan 11 08:10 horde-web/
drwxr-xr-x  10 root root 4096 Jan 11 08:10 hylax/
drwxr-xr-x   7 root root 4096 Jan 11 08:12 iPhoto2Ansel/
drwxr-xr-x  14 root root 4096 Jan 11 08:11 imp/
drwxr-xr-x  14 root root 4096 Jan 11 08:12 ingo/
drwxr-xr-x   3 root root 4096 Jan 11 08:12 internal/
drwxr-xr-x  15 root root 4096 Jan 11 08:12 jonah/
drwxr-xr-x  12 root root 4096 Jan 11 08:12 klutz/
drwxr-xr-x  13 root root 4096 Jan 11 08:12 kolab_webmail/
drwxr-xr-x   9 root root 4096 Jan 11 08:12 koward/
drwxr-xr-x  18 root root 4096 Jan 11 08:13 kronolith/
drwxr-xr-x  13 root root 4096 Jan 11 08:13 luxor/
drwxr-xr-x  17 root root 4096 Jan 11 08:13 mnemo/
drwxr-xr-x  18 root root 4096 Jan 11 08:14 nag/
drwxr-xr-x   9 root root 4096 Jan 11 08:14 operator/
drwxr-xr-x  13 root root 4096 Jan 11 08:14 passwd/
drwxr-xr-x  11 root root 4096 Jan 11 08:14 pastie/
drwxr-xr-x  10 root root 4096 Jan 11 08:15 sam/
drwxr-xr-x  13 root root 4096 Jan 11 08:15 sesha/
drwxr-xr-x  11 root root 4096 Jan 11 08:15 shout/
drwxr-xr-x  13 root root 4096 Jan 11 08:15 skeleton/
drwxr-xr-x   8 root root 4096 Jan 11 08:16 timeobjects/
drwxr-xr-x  14 root root 4096 Jan 12 09:08 trean/
drwxr-xr-x  16 root root 4096 Jan 11 08:16 turba/
drwxr-xr-x  11 root root 4096 Jan 11 08:16 ulaform/
drwxr-xr-x  15 root root 4096 Jan 12 11:06 vilma/
drwxr-xr-x  10 root root 4096 Jan 11 08:17 webmail/
drwxr-xr-x  18 root root 4096 Jan 11 08:17 whups/
drwxr-xr-x  15 root root 4096 Jan 11 08:17 wicked/

This tree does not look like a composer based installation and would not easily work in a web browser. I did not want to reinvent composer with its autoloader and other benefits so I wrapped it into an installer. This installer creates a new copy of the horde/bundle base project and registers all the other libraries as a special type of composer repository “path”.

 /srv/git/horde/components/bin/horde-components -c ~/horde-testme.conf.php install
[  INFO  ] Installation directory is missing: /srv/www/testme
[   OK   ] Created installation directory: /srv/www/testme
Creating a "horde/bundle" project at "../../www/testme"
Installing horde/bundle (dev-FRAMEWORK_6_0)
  - Installing horde/bundle (dev-FRAMEWORK_6_0): Mirroring from /srv/git/horde/bundle
Created project in /srv/www/testme

When running the composer install command on this prepared setup, composer will not download the horde components from packagist or github but will use your local checkout. Only external dependencies are still downloaded from the web.

# composer install
No composer.lock file present. Updating dependencies to latest instead of installing from lock file. See https://getcomposer.org/install for more information.
Loading composer repositories with package information
Updating dependencies
Lock file operations: 92 installs, 0 updates, 0 removals
  - Locking horde/alarm (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/argv (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/auth (dev-FRAMEWORK_6_0 as 3.0.0alpha7)
  - Locking horde/autoloader (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/browser (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/cache (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/cli (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/cli_modular (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/compress (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/compress_fast (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/constraint (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/controller (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/core (dev-FRAMEWORK_6_0 as 3.0.0alpha17)
  - Locking horde/crypt_blowfish (dev-FRAMEWORK_6_0 as 2.0.0alpha4)
  - Locking horde/css_parser (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/cssminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/data (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/date (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/dav (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/db (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/exception (dev-FRAMEWORK_6_0 as 3.0.0alpha4)
  - Locking horde/form (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/group (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/hashtable (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/history (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/horde (dev-FRAMEWORK_6_0 as 6.0.0alpha7)
  - Locking horde/horde-installer-plugin (v2.5.5)
  - Locking horde/hordectl (v1.0.0alpha4)
  - Locking horde/http (dev-FRAMEWORK_6_0 as 3.0.0alpha8)
  - Locking horde/http_server (dev-FRAMEWORK_6_0 as 1.0.0alpha2)
  - Locking horde/icalendar (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/idna (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/image (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/injector (dev-FRAMEWORK_6_0 as 3.0.0alpha11)
  - Locking horde/javascriptminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/listheaders (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/lock (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/log (dev-FRAMEWORK_6_0 as 3.0.0alpha9)
  - Locking horde/logintasks (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/mail (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/mime (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/mime_viewer (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/nls (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/notification (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/pack (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/perms (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/prefs (dev-FRAMEWORK_6_0 as 3.0.0alpha7)
  - Locking horde/routes (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/rpc (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/secret (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/serialize (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/sessionhandler (dev-FRAMEWORK_6_0 as 3.0.0alpha3)
  - Locking horde/share (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/stream (dev-FRAMEWORK_6_0 as 2.0.0alpha5)
  - Locking horde/stream_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/stream_wrapper (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/support (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/template (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/test (dev-FRAMEWORK_6_0 as 3.0.0alpha7)
  - Locking horde/text_diff (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/text_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha4)
  - Locking horde/text_flowed (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/token (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/translation (dev-FRAMEWORK_6_0 as 3.0.0alpha3)
  - Locking horde/tree (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/url (dev-FRAMEWORK_6_0 as 3.0.0alpha6)
  - Locking horde/util (dev-FRAMEWORK_6_0 as 3.0.0alpha8)
  - Locking horde/vfs (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/view (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/xml_element (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking horde/yaml (dev-FRAMEWORK_6_0 as 3.0.0alpha5)
  - Locking pear/archive_tar (1.4.14)
  - Locking pear/console_color2 (0.1.2)
  - Locking pear/console_getopt (v1.4.3)
  - Locking pear/console_table (v1.3.1)
  - Locking pear/pear (v1.10.14)
  - Locking pear/structures_graph (v1.1.1)
  - Locking pear/xml_util (v1.4.5)
  - Locking php-extended/polyfill-php80-stringable (1.2.9)
  - Locking psr/container (2.0.2)
  - Locking psr/http-client (1.0.3)
  - Locking psr/http-factory (1.0.2)
  - Locking psr/http-message (2.0)
  - Locking psr/http-server-handler (1.0.2)
  - Locking psr/http-server-middleware (1.0.2)
  - Locking psr/log (3.0.0)
  - Locking sabre/dav (4.6.0)
  - Locking sabre/event (5.1.4)
  - Locking sabre/http (5.1.10)
  - Locking sabre/uri (2.3.3)
  - Locking sabre/vobject (4.5.4)
  - Locking sabre/xml (2.2.6)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 92 installs, 0 updates, 0 removals
  - Installing horde/horde-installer-plugin (v2.5.5): Extracting archive
  - Installing horde/util (dev-FRAMEWORK_6_0 as 3.0.0alpha8): Symlinking from /srv/git/horde/Util
  - Installing horde/translation (dev-FRAMEWORK_6_0 as 3.0.0alpha3): Symlinking from /srv/git/horde/Translation
  - Installing horde/exception (dev-FRAMEWORK_6_0 as 3.0.0alpha4): Symlinking from /srv/git/horde/Exception
  - Installing horde/compress_fast (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Compress_Fast
  - Installing horde/cache (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Cache
  - Installing horde/stream_wrapper (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Stream_Wrapper
  - Installing horde/support (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Support
  - Installing psr/log (3.0.0): Extracting archive
  - Installing php-extended/polyfill-php80-stringable (1.2.9): Extracting archive
  - Installing horde/constraint (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Constraint
  - Installing horde/log (dev-FRAMEWORK_6_0 as 3.0.0alpha9): Symlinking from /srv/git/horde/Log
  - Installing psr/container (2.0.2): Extracting archive
  - Installing horde/injector (dev-FRAMEWORK_6_0 as 3.0.0alpha11): Symlinking from /srv/git/horde/Injector
  - Installing horde/controller (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Controller
  - Installing horde/crypt_blowfish (dev-FRAMEWORK_6_0 as 2.0.0alpha4): Symlinking from /srv/git/horde/Crypt_Blowfish
  - Installing horde/url (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Url
  - Installing horde/css_parser (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Css_Parser
  - Installing horde/cssminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/CssMinify
  - Installing horde/text_flowed (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Text_Flowed
  - Installing horde/secret (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Secret
  - Installing horde/idna (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Idna
  - Installing horde/text_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha4): Symlinking from /srv/git/horde/Text_Filter
  - Installing horde/stream_filter (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Stream_Filter
  - Installing horde/stream (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Stream
  - Installing horde/mime (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Mime
  - Installing horde/mail (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Mail
  - Installing horde/listheaders (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/ListHeaders
  - Installing horde/nls (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Nls
  - Installing horde/date (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Date
  - Installing horde/icalendar (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Icalendar
  - Installing horde/browser (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Browser
  - Installing horde/data (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Data
  - Installing horde/hashtable (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/HashTable
  - Installing horde/db (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Db
  - Installing horde/history (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/History
  - Installing horde/view (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/View
  - Installing horde/vfs (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Vfs
  - Installing horde/tree (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Tree
  - Installing horde/token (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Token
  - Installing horde/text_diff (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Text_Diff
  - Installing horde/serialize (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Serialize
  - Installing horde/xml_element (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Xml_Element
  - Installing horde/group (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Group
  - Installing horde/perms (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Perms
  - Installing sabre/uri (2.3.3): Extracting archive
  - Installing sabre/xml (2.2.6): Extracting archive
  - Installing sabre/vobject (4.5.4): Extracting archive
  - Installing sabre/event (5.1.4): Extracting archive
  - Installing sabre/http (5.1.10): Extracting archive
  - Installing sabre/dav (4.6.0): Extracting archive
  - Installing psr/http-message (2.0): Extracting archive
  - Installing psr/http-factory (1.0.2): Extracting archive
  - Installing psr/http-client (1.0.3): Extracting archive
  - Installing horde/http (dev-FRAMEWORK_6_0 as 3.0.0alpha8): Symlinking from /srv/git/horde/Http
  - Installing pear/pear (v1.10.14): Extracting archive
  - Installing pear/xml_util (v1.4.5): Extracting archive
  - Installing pear/structures_graph (v1.1.1): Extracting archive
  - Installing pear/console_getopt (v1.4.3): Extracting archive
  - Installing pear/archive_tar (1.4.14): Extracting archive
  - Installing horde/template (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Template
  - Installing horde/share (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Share
  - Installing horde/sessionhandler (dev-FRAMEWORK_6_0 as 3.0.0alpha3): Symlinking from /srv/git/horde/SessionHandler
  - Installing horde/prefs (dev-FRAMEWORK_6_0 as 3.0.0alpha7): Symlinking from /srv/git/horde/Prefs
  - Installing horde/pack (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Pack
  - Installing horde/notification (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Notification
  - Installing horde/compress (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Compress
  - Installing horde/mime_viewer (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Mime_Viewer
  - Installing horde/logintasks (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/LoginTasks
  - Installing horde/lock (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Lock
  - Installing horde/javascriptminify (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/JavascriptMinify
  - Installing psr/http-server-handler (1.0.2): Extracting archive
  - Installing psr/http-server-middleware (1.0.2): Extracting archive
  - Installing horde/http_server (dev-FRAMEWORK_6_0 as 1.0.0alpha2): Symlinking from /srv/git/horde/Http_Server
  - Installing horde/cli (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Cli
  - Installing horde/autoloader (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Autoloader
  - Installing horde/auth (dev-FRAMEWORK_6_0 as 3.0.0alpha7): Symlinking from /srv/git/horde/Auth
  - Installing horde/alarm (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Alarm
  - Installing horde/core (dev-FRAMEWORK_6_0 as 3.0.0alpha17): Symlinking from /srv/git/horde/Core
  - Installing horde/dav (dev-FRAMEWORK_6_0 as 2.0.0alpha5): Symlinking from /srv/git/horde/Dav
  - Installing horde/rpc (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Rpc
  - Installing horde/image (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Image
  - Installing horde/form (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Form
  - Installing horde/argv (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Argv
  - Installing horde/horde (dev-FRAMEWORK_6_0 as 6.0.0alpha7): Symlinking from /srv/git/horde/base
  - Installing horde/yaml (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Yaml
  - Installing horde/cli_modular (dev-FRAMEWORK_6_0 as 3.0.0alpha5): Symlinking from /srv/git/horde/Cli_Modular
  - Installing horde/hordectl (v1.0.0alpha4): Extracting archive
  - Installing horde/routes (dev-FRAMEWORK_6_0 as 3.0.0alpha6): Symlinking from /srv/git/horde/Routes
  - Installing horde/test (dev-FRAMEWORK_6_0 as 3.0.0alpha7): Symlinking from /srv/git/horde/Test
  - Installing pear/console_color2 (0.1.2): Extracting archive
  - Installing pear/console_table (v1.3.1): Extracting archive
81 package suggestions were added by new dependencies, use `composer suggest` to see details.
Generating autoload files
Applying /presets for absent files in /var/config
Looking for registry snippets from apps
Writing app configs to /var/config dir
Linking app configs to /web Dir
Linking javascript tree to /web/js
Linking themes tree to /web/themes
1 package you are using is looking for funding.
Use the `composer fund` command to find out more!

This composer environment works just like a regular installation. When you install turba, kronolith or passwd through composer, it will end up linking these apps and their library dependencies from the development tree.

Composer provides an option to copy files from the repositories rather than link the files. This would allow creating archives with artifacts for distribution packaging. The horde-components tool does not yet provide a switch to generate the necessary tweak to the repository files.

bookmark_borderTesting Horde 6 in Windows WSL Ubuntu

Install a recent ubuntu/apache/mod-php setup into your local Windows WSL to get going.
(This is a dump – It will involve into an article someday)

Assuming you have installed WSL2 for Windows 10 or Windows 11

This is Ubuntu for WSL in the MS Store: Ubuntu – Microsoft Apps
Install via CLI or Store UI
Run Ubuntu in WSL:
wsl -d Ubuntu

Now Upgrade, add the php repo, install some software

apt update ; apt dist-upgrade
sudo add-apt-repository ppa:ondrej/php
sudo add-apt-repository ppa:ondrej/nginx
apt install php8.2 php8.2-fpm php8.2-curl php8.2-dom composer gh php8.2-mbstring php8.2-sockets nginx

Add nginx config for passing php-fpm and for recognizing .php

File: /etc/nginx/sites-available/base.php82.horde

server {
        listen 80;
        root /var/www/base.php82.horde/web;
        index index.php index.html index.htm;
        server_name base.php82.horde;
        location / {
                try_files $uri $uri/ =404;
        }
        location ~ \.php$ {
                include snippets/fastcgi-php.conf;
                fastcgi_pass unix:/run/php/php8.2-fpm.sock;
        }
}

Run these commands to activate PHP for our horde testing environment:

cp -ar /var/www/base.php82.horde/vendor/horde/horde/config/conf.php.dist /var/www/base.php82.horde/var/config/horde/conf.php
cd /var/www/base.php82.horde/

Open your Windows Notepad in “Administrator” mode and edit file C:\Windows\System32\Drivers\Etc\Hosts

Add new lines at the bottom and save

127.0.0.1 base.php82.horde
127.0.0.1 components.php82.horde

By now you can enter into your browser: http://base.php82.horde and get to a message saying
Please create a /var/www/base.php82.horde/var/config/horde/conf.php file and then run ‘composer horde-reconfigure’ to activate Horde

Starting with a default horde config

Start with a default horde config

cp -ar /var/www/base.php82.horde/vendor/horde/horde/config/conf.php.dist /var/www/base.php82.horde/var/config/horde/conf.php
cd /var/www/base.php82.horde/
composer horde-reconfigure

bookmark_borderMaintaina’s Horde 6 goes upstream

It’s been a while. It’s been much too long actually. Let’s look forward though.
I recently started to onboard into FOSS development again. People have been asking for PHP 8.2 support with Horde but capacity to deliver that was very limited. But we’re getting there.

I used the “Maintaina” fork to deliver fixed and upgraded customized versions of Horde beyond what was allowed in Horde’s master branch. It’s time to move forward though. Maintaina used to target PHP 7.4 to PHP 8.1, composer2 based install from a custom satis repo and some quite invasive changes to CalDAV/CardDAV support. Most prominently, Maintaina introduced library level compatibility with several PSRs (container, http middleware, logging) and the all new horde/http_server component.

At the moment I am importing the FRAMEWORK_6_0 branches and the alpha releases based on this branch to horde’s upstream repository. So far I have imported enough libraries to make the Horde Base application barely install from packagist.

Get the current state of affairs

Your magic carpet is:

composer create-project horde/bundle .

Originally I wanted to mimic maintaina’s setup with a separate satis server as a QA stage before release to packagist. I re-considered and dropped the separate satis server. Every update of the development branches and every tagged release is consumed via packagist. I will have to fix some of my tools and workflows to reflect that. The tagged alpha releases still use maintaina’s satis server. The upcoming releases won’t.

Onboarding Procedure


I leave the FRAMEWORK_5_2 and master branches mostly untouched. FRAMEWORK_6_0 is the new default branch on github for anything I am handling. I only edit other branches if they block packagist’s import. Usually I also rebase FRAMEWORK_6_0 on any latest commits of master, but in some cases I only cherry-picked from master branch. In some cases, some entries of the changelog between 2022 late and now (october 2023) might be missing. Pull Requests, bug reports and patches welcome.

What’s next

Before I move forward with the actual applications, I want to make sure the necessary infrastructure is in place. I need to fix some aspects of the FRAMEWORK_6 version of composer. The workflow files in each repo need some review, too. Does PHPUnit and PHPStan still pass? Can we improve management of 130+ repo’s workflows?

Finally https://dev.horde.org should index from packagist, not from the satis server we originally planned to use.

What will become of maintaina repos?

The maintaina repos have had direct contributions from some trusted maintainers from the company I used to work for. They service some customers out of these repos and the related SATIS server so I won’t actively break it. However, with the move to Horde upstream, maintaina has served its purpose for me and I will not actively support the fork anymore. I suggest once everything is ported to upstream, maintaina should be archived. I will need to consult other stakeholders of this fork and the satis server.

Maintenance cost

Over the lifetime of the fork I have explored and applied various strategies for keeping the effort in check. Still, a fork of 100+ repos and the accompanying infrastructure for testing and deploying is a major burden which detracts from actually developing and maintaining code. I am glad I can save on this now and actually contribute to Horde directly in a way that doesn’t slow down activity too much. We now have the chance to speed up the cycle of feedback and releases. I hope this attracts some occassional and regular contributors.

bookmark_borderWSL2 openSUSE Tumbleweed ssh-agent

ssh-agent does not persist over sessions in WSL2 with current openSUSE Linux and other common distributions. Linux native bashrc solutions don’t work here.
Let’s use keychain instead

zypper in keychain

Then let’s add some snippet to .bashrc and remove any eval ssh-agent lines

vim .bashrc

# Once per key to load by default
/usr/bin/keychain -q --nogui $HOME/.ssh/id_rsa
/usr/bin/keychain -q --nogui $HOME/.ssh/id_host_www.ralf-lang.de

# Only once
source $HOME/.keychain/$HOST-sh

Then open a new window. It will ask you to enter your passphrase once. Enter any additional windows, it won’t ask you

bookmark_borderSatis is now a Composer Plugin.

Satis is the lightweight, static repository generator for the composer package manager. It enables admins to provide PHP software packages inside airgapped CIs, OS packaging environments and restricted data centers.

Back in August I added a plugin mode to satis to make it work as a regular composer plugin. While working on it, I also fixed some preexisting issues found by static analysis and made it compatible with the recent composer versions 2.3 and 2.4.

This week, the upstream maintainers merged my contribution. I feel a bit satis-fied 😉

Why make it a plugin?

When looking under the hood, it is easy to see that satis always has been some awkward kind of plugin or extension. It literally sits on top of an internal copy of composer. It hooks into its internals in not quite transparent ways, it uses its class and interface organs for its own vital functions. You might call it a parasite that attaches to composer’s body for its own needs. There are downsides to this approach. The first is that you need a private copy of composer. The second is that any refactoring of composer internals likely breaks satis compatibility. That happened some time ago when composer 2.3 and 2.4 were released and not for the first time. Composer has a maturing plugin API with nice, well-defined integration points. It provides some means to overload or amend core functionality but it also provides messaging between core and plugins. I only did the bare minimum work to make satis hook into the plugin API and not break the standalone mode. When installed as a dependency, package resolution will ensure that the API versions used by satis matches the API versions provided by the underlying composer.

I don’t quite understand… What is the benefit?

By itself, this change provides little benefit. It is a feature enabling feature.

  • Satis can be further refactored to make compatibility bread less often
  • Satis can send and receive events from composer or other composer plugins. This enables running satis as part off a custom command. Think passing unit and integration tests of a project and then conditionally updating the staging or canary package repository.
  • Satis’ schema could be amended to make a project’s root package also function as an instruction to build a repository of all dependencies with almost zero configuration. Add this to a workflow or add a collaborator plugin that handles the necessary push/release and you have a powerful tool for CI and developer laptop alike.

But as I went along, I also re-aligned satis with the latest breaking changes inside composer 2.3/2.4. This will benefit users who do not care about the whole plugin story.

What’s next?

With satis 3.0-dev merging this initial change, the next steps are obvious, but not urgent.
Making the new plugin mode play nice with the latest composer was already easier than fixing the standalone mode. Satis still has an internal, now updated dependency copy of composer which is only run in standalone mode.

Standalone mode should be refactored to be just a thin wrapper around composer calling into its satis plugin. Keeping intrusion into composer internals to the bare minimum to hide the builtin commands and re-brand it as satis, this would make breakage on upcoming updates much less likely. Eventually, we can maybe stop carrying around a source code copy of composer at all.

Finally, there is reaping the benefits. I want to leverage composer/satis functionality inside the horde/components tool. Rolling out new versions of horde stuff could be so much easier.

Resources

bookmark_borderLet’s have a date. Revised horde/date ideas

It’s bad but not as bad as you think.

The way we write dates is very different among cultures and technologies. Even countries of the same language family might have totally different notions where to put the year, where to put the month, where to put the day when writing down a date in numbers. Apart from the order, we might use hyphen -, dot . or slash / to mark the sections. Leading zero yes or no. It gets much worse when writing out a date, like “Vlad was born on the 13th of June” or “Monday, October 10 I will have a barbecue”. Mix in different languages and different ways to abbreviate Monday as “Mo”, “Mon”, “M”.
The PHP-native ways to deal with date math and date formatting are a little quirky and best wrapped in some code that hides away the nasty details but exposes useful capabilities.

The horde/date library has been around for ages and it does its job. Due to changes in the PHP 8.1+ engine, it requires some internal re-engineering. As it is also due for a conversion to namespaces and PSR-4, I decided to think about the interface a little more. These are rough ideas and I am still figuring what will be fun and safe to work it and what is possibly over-thinking it.

It’s DateTime all right.

PHP provides internal data types to store and manipulate a date: DateTime and DateTimeImmutable. Horde_Date behaves similar to DateTimeImmutable but implements a lot of intricate math on its own. Still, it uses DateTime for some conversions. It makes sense to delegate most mutation math to the DateTimeImmutable class and make it also hold most state. No need to manually manage minutes, hours, days of weeks etc in most cases. Preferring Immutable safes us some clone magic.

Wrap it up or extend it?

There are two possible approaches to dealing with the builtin type: Either extend it or hide it in a wrapper. The third option would be to implement the interface but this is not possible.
Extending any of the two DateTime types would be handy as it is the lingua franca between different library ecosystems. We could simply inject our Horde Date type into any library that uses it. There are down sides to this approach though. While DateTimeInterface is reasonably small and mostly useful, DateTime and DateTimeImmutable expose a lot of functionality. Some of it is awkwardly named. Some of it would have to be wrapped in extending to avoid inconsistencies in our own object. Some of it possibly clashes with own naming schemes and blocks us from using preferred signatures. Some of it might not fit into our own notion of what belongs where. There is also a risk of exposing different functionality based on PHP versions. This is undesirable. Extending is not the way to go. Hiding away the DateTimeImmutable object and exposing it explicitly might seem a little verbose. It offers some interesting applications.

Clock Date – Now is the right time.

A clock date type always emits now. Asking it again at a later point will yield a later time without having to manipulate the object. This is useful for tracing duration of processes or for emitting status messages. We can also make this clock the second element of a time span. Its duration will automatically expand. We can add a Stop method to the clock which will return a regular, fixed date for further processing.

No Date – When it’s not right for you.

Sometimes we cannot rely on a date being present in the input. It might be optional. It might be required by the current data model but used to be optional or malformed. Traditional options would be expressing the non-date as null value, throwing an exception or silently assuming no date means now. These are appropriate, good solutions in many cases. Sometimes you may prefer to have the “no date” information behave a little like a date – until it reaches a point where it cannot. For example, a “not a date” can be formatted many ways. It can be serialized to the backend, provided the backend can deal with it. It can however not be calculated, mutated or cast to a DateTimeImmutable. I am not yet sure how to handle this. Maybe it should be confined to Formatters and Readers.

In The Zone.

PHP provides an own finite list of Time Zone identifiers. Wrapping PHP’s timezone objects allow dealing with well known but unsupported timezone names. We can map them to known names. We can safe the originally provided name for later usage. We might not carry a PHP timezone at all but signal the other parts of the library that some custom code must be applied.

In Good Format.

There are a plethora of ways to express a given date. There are three builtin date renderers in PHP, IntlDateFormatter (not always installed), strftrime (deprecated) and DateTimeInterface::format (English only). You might add your own. Each has its own dependencies, arguments, requirements.
It is much simpler if there is a Formatter type. Implementations can just configure it and load it with a date. Consumers have a simple interface to work with them regardless of how they are implemented.This also allows to keep the dependency footprint of the core date library low and makes adding more output formats very easy. The same is true for reading data. Reading values from various formats should not be the Date object’s concern. Another object should turn arbitrary string, integer or other data into dates – including legacy Horde_Date objects.

bookmark_borderModernizing horde/text_diff

If you ever read a github pull request or similar extension proposal, you will likely have seen side by side comparisons of the original and the changed file. You may also have seen some text format that highlights only differences and a little context but hides the unchanged rest of the file. Both of these formats are called Diff, named after the popular diff and patch utilities dating back to ancient Unix times. The git diff command does something very similar. The horde/text_diff library and its ancestor, the pear/text_diff library, are tools to generate and format such difference information for different usage scenarios.

Apart from Horde’s internal usage in its repository viewer, horde/chora, and its wiki software, horde/wicked, the tool is also used by external parties. WebSVN maintainer Michael O. approached me because he wanted to use a PHP 8.2 ready version of horde/text_diff to substitute an older component which did not do the job. Michael has been very helpful in getting me started, pointing me to some issues to solve and also providing his own solutions in some parts. The result is a conservative update of horde/text_diff that will run in the upcoming versions of PHP without causing any trouble. But this is only where I started.

Breaking bad habits

A closer look at the internal structure of the library showed that it deserved a major overhaul. The solution was to refrain from a verbatim upgrade to namespaces and the likes but to actually change some things. This meant breaking backwards compatibility. I go to great lengths to keep a conservative drop-in version of everything I touch in the lib/ folder. Sometimes it is just an interface or wrapper, sometimes the new and old code do not really share a lot.

I began with adding type hints to most methods. Targetting PHP 8.1+ for the src/ path allows to use union types and intersection types. A lot of knowledge hidden in phpdoc comments is moved into actual code and makes it more robust.

Exploring the code for base classes and interfaces, I noticed that some things I did not like.

Method signatures did not add up

Some method signatures did not add up. Depending on the type of Diff Engine, the diff() method would take different types of arguments. The interface was mixing the specifics of how the diff engine is set up with the command to create a list of operations objects. Loading the engine is now separated from running it. The running method is now always called in the same way.

Internal dependency creation

The Diff, Threeway Diff and Mapped Diff utilities all created their diff engine internally. To do this, they needed a very flexible constructor that allowed passing whatever is needed to set up the actual engine. That was bad enough but they also did it in different ways. The Differs’ constructor now only accepted a pre-constructed engine. For convenience, I created a factory which would take over the responsibilities originally assigned to the differs’ constructor: Building a differ from input and if no specific differ was chosen, selecting one by some priority logic. In the end it turned out that the Differ does not need the engine at all but rather needs the product of the Engine: A set of operations to transform document A to document B. Born is the OperationList class. I did not want to just pass an array. I added a small static method as a named constructor. It frees the actual constructor of too many responsibilities and allows to keep the interface clean and strict.

More explicit type juggling

Creation if diffs contains some interesting math. The algorithms use a lot of short variable names and operations that make sense if you know the underlying theory but otherwise look like garbage. I added some explicit conversions between string and integer and made some changes to ensure a number zero or an empty string is not mistaken for a “false” or “null” value which would have another meaning.
Overall it is now much easier for static analyzers to spot any issues.

Dual stacks have a price tag

Essentially maintaining two different sets of the library comes with some cost. One must ensure that unit tests targeted at the newer platform are not run when testing for compatibility with the older platform. The conservative lib/ is ready to run PHP 7.4 code but the modern version in src/ must be transpiled to be run in PHP 7.4 – I do that on release but still it is another aspect that needs minding. As I also use automated tools for some upgrade tasks, I need to ensure I do not upgrade the lib/ path. The price is worth it as I cannot convert the code base at once and I want to provide a good development experience to all who are caught in between maintaining an older release or creating new code. I am in that spot myself. Essentially it allows me to run two conflicting major versions of some critical libraries and pick the right one for different sub systems. The need will go away as code is gradually migrating towards the newer implementations. At some point a next major version will drop the conservative path. Anybody interested is free to maintain the older major version and keep using it.

Upcoming work

I consider the external interface of the newer horde/text_diff implementation fairly stable by now. Internally, however, there is a lot of room for improvement. Some functionality should move out of the base classes and into separate traits – which the base classes will use. Some getters should be added and used, preparing to move some public variables to internal state in a next major release. The new OperationList gets unloaded to plain arrays in several places – It needs to learn some tricks without degrading into a glorified array. None of this should stop early adopters from using the new code base. None of this is supposed to break any user code.

Out of scope for now

There are some items which I decided to postpone for now. One thing which bothers me is the amount of dependencies. While a dependency on horde/exception makes sense, it pulls in horde/translation for no good reason. Horde\Util is pulled in but really only used in two places: A horde/string call which could be reduced to a direct call to an internal function and one call to a helper for handling temporary files. That helper should maybe live in its own library, nicely decoupled from unrelated utilities. There was a reason why they were packed together but it is no longer relevant.

Also, some functionality is missing in the Xdiff-based engine. Most distributions do not even offer php-xdiff, including my own development platform. I will add that feature once I get it into CI and into the development setup. I do not want to delay other items to do that right now. Patches welcome 🙂

bookmark_borderCode Generators: Bad, worse & ugly

Code generators have been invented and forgotten at least four times in software history. They have an appeal to developers like the sun to Daedalus’ son. Let’s not be Icarus, let’s keep them generators at a distance and watch them carefully.

Whenever a language, framework or paradigm forces developers to do the same thing over and over and over and over again they will try to get rid of that repetition. One approach is to automate writing code. It is not pretty but it saves time to concentrate on more interesting and useful things. Seasoned and reasonable software developers have resorted to that solution and many inexperienced have followed. Outside very narrow use cases I think generated code should better not exist at all.

Valid use case

Generated code and other kinds of boilerplate code are valid where avoiding them is not practical. This is often true for entry points. Depending on language, it might not be trivial for a the first piece of running code to find

  • its configuration
  • its collaborators, base classes, dependencies
  • useful data from previous runs

I have written a long, long piece on two-stage autoloaders and other two-stage bootstrapping topics and I keep rewriting it, breaking out separate topics. It is still unreleased for a reason. Any two-stage process that splits automated detection or definition of artifacts from the production run that uses them is essentially code generation. Avoiding it might be possible but impractical. Some level of repetition cannot be avoided at all and is best automated.

Another valid use case is generating code from code or transpiling. Nothing is wrong with that.

Unfortunate Use Cases

There are other use cases that should be avoided. Your framework follows convention over configuration so making magic work requires having some artifacts in the right places. Even if they have no natural place in your specific solution they are needed for technical reasons so you copy/paste or auto-generate the minimum sufficient implementation and make it fit. This is something to look for. Often there are ways around it. Another case is limits of the underlying language. You evolved from using magic properties and methods to implementing type safe, explicit equivalents but now you have to re-invent the type specific list or container type and you automate it. Bonus points if your ORM tool requires it. If your language does not support generics or another templating method, you are stuck between repetitive, explicit code and weakly typed magic. You end up using a code generator. Hopefully at one point somebody is annoyed enough and ventures to bring generics into your language. That would be the better solution but it is likely out of scope for your day to day work.

Stinking unnecessary use cases

Beyond that you are likely in the land of fluff where things just happen while they can and lines of code are generated just because it is customary. This is a foolish thing best avoided. Granted, automating code is better than hand-writing it. It does however not mean the code should exist at all. If you have no specific reason to repeat code, it is likely a design smell. This is not new, the Cunningham wiki had this thought a decade or more ago. Likely they were not even the first to recognize it. Refactoring, abstraction, configuring the specifics can help reduce the necessity for repetitive code.

My programming tools were full of wizards. Little dialog boxes waiting for me to click “Next” and “Next” and “Finish.” Click and drag and shazzam! — thousands of lines of working code. No need to get into the “hassle” of remembering the language. No need to even learn it. It is a powerful siren-song lure: You can make your program do all these wonderful and complicated things, and you don’t really need to understand.

Ellen Ullman: The dumbing-down of programming, Salon, May 1998 https://www.salon.com/1998/05/12/feature_321/

Let us take the input to a code generator and make it the input to abstracted, ready to run code instead. We will know when it is not practical, not performant or not possible. Then code generation is a blessed thing. Otherwise it is a sin.

bookmark_borderWhy you should develop for latest, greatest

Developers sometimes choose not to use the latest available language features that would be appropriate to tackle a problem for fear of alienating users and collaborators. This is a bad habit and we should stop doing that. Part of the solution are transpilers. What are transpilers, where are they used and what is the benefit? Why should we consider transpiling all our code?

I cut this piece from an upcoming article that is way too long anyway. I made this new article by reusing and reshaping existing text for a new audience and frame. You are reading a new text that first was built as a part of another text. – Yes. This was transpiling: Rephrasing an input, including externally supplied, derivative or implicit facts about it to an output that generally expresses the same. Excuse me, what? Let me go into some details.

Transpiling: Saying the same but different

In software, transpilers are also known as source to source compilers. They take in a program written in one language and write out a roughly equivalent program for another language. The other language may be another version or dialect of the input language or something entirely different. Don’t be too critical about the words: transpilers are just like all other compilers. Source code is machine-intelligible, otherwise it could not be compiled. Machine code is intelligible by humans, at least in principle.

Preprocessors are transpilers

A preprocessor is essentially a transpiler even if it does not interpret the program itself. The C language preprocessor is a mighty tool. It allows you to write placeholders that will be exchanged for code before the actual C compiler touches it. These placeholders may even have parameters or make the program include more code only if needed. Concatenating many source files into one and minifying these by stripping unnecessary whitespace can also be seen as a primitive form of transpiling.

Coding Style Fixers are transpilers

Automatic tools that edit your source code are transpilers. They might only exchange tabs for four space characters or make sure your curly braces are always in the same place or they may do much more involved stuff. For example php-cs-fixer transforms your technically correct code written in plain PHP into technically correct code in standards-conforming plain PHP. One such standard is PSR-2, later deprecated in favor of PSR-12 and PER-1 – these are all maintained by the PHP FIG. Software projects may define their own standards and configure tools to transpile existing code to conform to their evolving standards.

Compilers are transpilers

A compiler is a transpiler. It takes in the source code and builds a machine-executable artifact, the binary code. It might also build a byte code for some execution platform like Java’s JVM. It might build code for a relatively primitive intermediate language like CIL or a machine specific Assembly Language. Another compiler or an interpreter will be able to work with that to either run the software or turn it into a further optimized format. These transformations are potentially lossy.

Decompilers are transpilers

Earlier in life I used tools like SoftICE that would translate back from binary machine instructions to Assembly Language so that I could understand what exactly the machine is doing and make it do some unorthodox things. Compiling back from Machine Code to the machine-specific Assembly Language is technically possible and lossless but the result is not pretty.

Lost in translation

When humans rewrite a text for another target audience, they will remove remarks that are unintelligible or irrelevant to the new audience. They may also add things that were previously understood without saying or generally known in the former audience. Transpilers do the same. When they transpile for machine consumption, they remove what the machine has no interest in: Whitespace, comments, etc. They can also replace higher concept expressions by detailed instructions in lower concepts. Imagine I compiled a program from assembly language into binary machine code and then decompiled back to assembly language. Is it still the same program? Yes and no. It is still able to compile back into the same machine program. It does not look like the program I originally wrote. Any macro with a meaninful name was replaced by the equivalent step by step instructions without any guidance what their intention is. Any comments I wrote to help me or others navigate and reason about the code are lost in translation. The same is true anytime when we translate from a more expressive higher concept to a lower concept. Any implicit defaults I did not express now show up as deliberate and explicit artifacts or the other way around, depending on tool settings.

Lost in Translation but with Humans

You may know that from machine translated text. Put any non trivial text into a machine translator, translate it from English to Russian to Chinese to German and then back to English. In the best case, it still expresses the core concept. In the worst case it is complete garbage and misleading.

Another such thing are Controlled Languages like Simple English, Français fondamental, Leichte Sprache, etc. They use a reduced syntax with less options and variations and a smaller selection of words. Some like Aviation English or Seaspeak also try to reduce chance for fatal ambiguity or mishearing.

These reduced languages are supposedly helpful for those who cannot read very well, are still learning the language or have a learning disability. They may also enable speakers of a closely related foreign language to understand a text and they generally cater to machine translation. For those who easily navigate the full blown syntax and vocabulary and can cope with ambiguity and pun, simplified language can be repetitive, boring and an unnecessary burden to actual communication. Choosing a phrase or well known roughly fitting word over a less used but more precise word is an intellectual effort. Reading a known specific word can be easier on the brain than constructing a meaning from a group of more common words. Speaking to an expert in a language deliberately evading technical terms may have an unintended subtext. Speaking to a layman in lawyerese or technobabble might not only make it hard for them to understand you but also hard for them to like you. Readers will leave if I make this section any longer.

Useful application

Now that everybody is bored enough, let’s see why it is useful and how good it is.

Upgrading Code to newer language versions

You can use a transpiler to upgrade code to a newer version of the language. Why would you want that? Languages evolve. New features are added that allow you to write less and mean the same. Old expressions become ambiguous by new syntax and features. Keywords can be reserved that previously weren’t. Old features become deprecated and will finally stop working in later versions. A transpiler can rewrite your code in a way that it will run in the current and next version of a language. It can also move meta information from comments or annotations into actual language code.

    // Before
    /**
     * @readonly
     * @access private
     * @var FluffProviderInterface Tool that adds bovine output
     */
    var $fluffProvider;
    /**
     * Constructs an example
     * 
     * @access public
     *
     * @param IllustrableTopicInterface A topic to explain by example
     * @param bool $padWithFluff        Whether to make it longer than needed
     * @param int  $targetLength        How long to make the article
     *
     * @return string The Article
     */
    function constructExample($topic, $padWithFluff=true, $targetLength=3000)

Now that’s what I call a contrived example. Code might look like this if it was originally written in PHP 4 and later enhanced over the years, only using new expressiveness where needed. While it technically runs, it is not how we would possibly write it today.

    // After
    /**
     * @var FluffProviderInterface Tool that adds bovine output
     */
    private readonly FluffProviderInterface $fluffProvider;
    /**
     * Constructs an example
     * 
     * @param IllustrableTopicInterface A topic to explain by example
     * @param bool $padWithFluff        Whether to make it longer than needed
     * @param int  $targetLength        How long to make the article
     *
     * @return string The Article
     */
    public function constructExample(
        IllustrableTopicInterface $topic,
        bool $padWithFluff=true,
        int $targetLength=3000
    ): string

That could be the output of a transpiler. It takes meta information from controlled language in the comments and uses the advanced grammar of the improved PHP language to express them.
In other words, the upgraded code has turned instructions for the human or for external tools into instructions that the language can actually enforce at runtime. Before it helped you understand what to put in and what to expect out. Now it forbids you from putting in the wrong things and errors if the code tries to give back anything but text.

It may drop comments that are already expressed in the actual code. Some project standards suggest to drop @param and @return altogether to make the code more consise to read. I am a little conservative on this topic. A documentation block may be removed if it does not contain any guidance beyond the code. There is no need to rephrase “this is the constructor” or “The parameter of type integer with name $targetLength tells you how long it should be”. But sometimes things deserve explaining and sometimes the type annotations exceed what actual the language expresses. Intersection types are PHP 8.1+. PHP 8.2 can express “return this class or false but not true” while before the language only allowed “This class or a boolean (either true or false)”. Annotations can be read by tools to work with your code. As demonstrated, a transpiler can use them to rewrite your code to a more robust form. Static analyzers can detect type mismatch that can lead to all sorts of bugs and misbehaviours. Documentation generators can strip away the actual code and transform the comments and structural information into something you can easily navigate and reason about. Code including high concept and documentation is first and foremost for humans. Adapting it for machines often means dumbing it down.

Downgrading to an older platform or language version

Code can be transformed in the other direction, stripping or replacing advanced expressiveness to make the older runtime understand the code. This is very popular with Frontend Developers: Both Javascript and CSS are usually no longer shipped the way they are written. A variety of type safe and advanced languages exist that are not even intended to be run in their source form but compiled down to a more or less modern standard of JavaScript, then minified to the smallest valid representation. Possibly variable and function identifiers are changed to avoid them colliding between unrelated software loaded into the same browser. In other languages, we are used to develop against a target baseline and only use the features it provides, plus annotations for concepts it does not support. We choose the baseline by deciding on the lowest platform we want to or have to support. This is jolly insane and I mean it in a nice way.

Imagine we create a book for small children. We will first create a compelling story, lovely characters and possibly some educational tangent using our words and our thoughts, the level of abstraction we are fluent in and the tools we can handle. We finally take care to adapt wording, level of detail and difficult concepts to fit the desired product.
We don’t write to the agent, the publisher or the printing house in baby english. So why should we use anything less than our own development environment supports? It is not healthy. Outside very special situations or for the joy of it, we generally don’t work with one hand tied to the back, using antiquated tools and following outmoded standards.

This Catapult resembles the state of the art centuries ago. Shooting it is fun. For any other purpose it is the wrong tool.
This Catapult resembles the state of the art centuries ago. Shooting it is fun. For any other purpose it is the wrong tool.

If we cater to the lowest assumable set of capabilities at development time, we limit ourselves in a costly way. We cannot benefit from the latest and most convenient, i.e. effortless and reliable set of tools. We are slower than we could be, we will make more mistakes and it will exhaust us more than needed.

Provided our production pipeline from the development laptop or container to the CI are able to work with the latest tools, we can use them.

Deliver using a transpiler

The source branch should always target your development baseline, tools as modern as you can come by. Delivery artifacts, i.e. released versions, should deviate from the source distribution anyway:

  • Why should you ship build time dependencies with a release?
  • Why should you ship CI recipes or linter configurations with a release?
  • Depending on circumstances, shipping the unit tests might be useful or waste.
  • You would not normally ship your .git directory, would you?

Adding a transpiler step is just another item, just another reason. Transpiling to your lowest supported baseline is not really different from zipping a file, editing a version string or running a test suite to abort faulty builds before they ship. But still, it is not perfect. The shipped code will run on the oldest supported environment but it will miss many runtime benefits of newer versions. This is especially true if your library is a build time dependency of another project. In the best scenario, a build for a fairly recent but reasonable platform expectation exists and another build for an well-chosen older target exists. Both need to run through the test suite and ideally the older build will pass the test suite both when actually run on the old platform and when run on an upgraded platform. There are some details, edge cases and precautions needed to make this feasible and reliable. This will be detailed in an upcoming article which just shrank by a good portion.