bookmark_borderCan Horde’s internal API use PSR-7 Messages?

The Horde Inter-App system has been around since Horde 3.

Horde Inter-App messages are addressed by a two part string. The first part, followed by a slash character, is called the API. The second part after the slash is called the method. Registry can delegate complete APIs to an application or a list of complete API/method strings. In the latter format, a certain API/method combination can be assigned to one app even if the API in general is assigned to another app. The API is implemented by a class $application_Api in file Api.php inside each horde application. That application class methods and their signatures are the methods exposed by the Inter-App API. There are some meta arrays controlling further details but let’s ignore them. All but a few APIs only take arrays and primitives (string, number, bool) as parameters and issue them as return types. This is because the RPC layer eventually receives and emits HTTP messages, which are just text. Only those Inter-App API methods which are meant to be strictly internal will consume and emit PHP Objects.

PSR-7 messages and PSR-15 handlers/middlewares are an interop standard. They do not make a lot of assumptions about the underlying implementation. They have been used to implement REST solutions as well as old style server-driven dynamic websites. The request objects contain an URI to a resource and will eventually result in a response object. In between there is usually a broker piece called a router, which analysis URI and other request parameter to assign it to the proper implementation code or chain of code pieces, called middlewares. Anywhere in that chain, an answer is created and sent back to the caller. The request and response bodies are streams of text or binary data, as the request and response are essentially text messages.

At first glance, this is an easy match. The Registry mediates between the message sent by the caller and the code which handles it. We could call it the router. The API class with its methods could be seen as a set of handlers. The PSR-7 ServerRequest object represents a HTTP request, but it also allows arbitrary attributes attached to the actual request data. These attributes may be any PHP value, including objects.

There are some details to keep in mind though.

The inter-app API has little definition on a data contract. It predates PHP method parameter types and return types. In traditional code, users could feed just about anything into the inter-app API and the implementation would need to guard against any value expected or unexpected. Inter-App API just assumes the caller is eligible to call. Authentication is delegated to the existing PHP session or to the RPC setup, authorization control must happen in the called code. That may lead to bloated, repetitive code in the implementation.

As each app has only one API class file, it is not currently possible to implement two different methods on different APIs if the same app handles it. If you have two different APIs clients/get and contracts/get and both are implemented in the same app, they will end up in the same code path. The way around it is with calls like clients/getClients and contracts/getContracts, but this is just ugly.

The rampage/routes/http_server stack can easily discern a GET /clients/ call and a GET /contracts call, but it only works inside a specific app. Setting up a separate API set of routes, we can easily have calls abstracted from the implementing app. The system of handlers and middlewares allows to delegate authentication and authorization checks outside of the actual implementation of an endpoint. This reduces repetitive boilerplate. One big issue remains. As of now, Inter-App can return native PHP objects to the caller. PSR-7 messages allow Attributes on the ServerRequestInterface but there is no equivalent in the ResponseInterface. Inter-App can carry objects (for internal calls) or serialisation-friendly nested arrays until it hits the RPC layer. This layer will turn it into a text structure, say XML or JSON. How would we do that in ResponseInterface implementations? How would that make the implementation reusable for a REST interface, app-internal AJAX or other code?

A vision of convergence

Bringing together new capabilities and existing system participants is tricky. A new RPC and Inter-App system should integrate with the old interface, it should not just stand beside it. Having two different Inter-App layers would be confusing, abandoning the old one right now would be unnecessary stress on developers’ time budgets.

As an inter-app user, I want to use $registry->call(‘method’, [params1, param2]) or $registry->api->method($param1, $param2…) as I did before.

As an RPC user, I want to call \Horde_Rpc::request('api/method', $params, $options) as I did before. I do not care what happens in the background.

As an application developer exposing an API, I do not want to give up Api.php right now, it has to work with the new stack as good or bad as it did before.

As a developer of new apps and features, I want to leverage extended capabilities. I want to be able to implement two distinct APIs using the same method names. I want to be able to return native objects, even serialisation-unfriendly ones with php resources, along with a serialisation-friendly message to use in RPC, Rest or other HTTP use cases. I do not want to be restricted to two levels of API/method. I want to re-use middleware I already built for the frontend AJAY.

As a distributed app developer, I want to define API resources and have them served either internally or by external microservices transparantly over http requests.

For the future, I would like some degree of introspection and possibly some guidance on allowed or required request parameters.

As an integrator, I want to securely communicate with only partially set up horde instances to finish or upgrade setups by firing HTTP requests.

Implementation approach

The horde-deployment project includes a route from webroot/api/ to a global API router managed by the horde base app. This API router is first populated by the Registry and then supplemented by a config/routes.api.php file in each registry app.

Regardless of the calling context, a cascade of middlewares sets attributes for the called API/route, the parameters, the resolved implementation and the outcome of already happened authentication checks. The implementation is either an adapter middleware calling an app’s Api class or an actual implementation middleware/stack. It will write the return values and other state into an attribute digested by the bottom of stack. In case of Inter-App, a token response is returned and the actual data structures are taken from the handler and returned to the caller. Real RPC backends generate appropriate headers and stream body for response. The response can possibly be processed further as it returns back to top of stack, for example gzip compressed or logged or trigger metrics updates.

bookmark_borderMove config out of webroot for Horde 6

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

  • Default assumptions in the Horde registry for where apps are located were wrong
  • Javascript from library packages needed to be web visible and Horde needed to know their path and webroot
  • Custom Themes location and webroot
  • Composer2 would wipe custom configuration during software updates
  • Composer2 would wipe the static dir during software updates

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

  • Point to /web/$app/ for an app’s fileroot
  • Fix the webroot assumption, too from $webroot/horde/$app to $webroot/$app
  • Make horde search for themes in /web/themes/$app/ rather than /web/horde/$themes and /web/horde/$app/$themes/
  • Make horde load horde and library javascript files from /web/js/horde rather than /web/horde/js

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation

These items remain in the config/ dir:

  • default registry.php file
  • default prefs.php file
  • routes.php file
  • conf.xml definition file
  • other defaults files like backends.php, nls.php
  • a stub for hooks (maybe we can get away without it)

These items reside outside the webroot:

  • conf.php file
  • registry.local.php
  • routes.local.php (if any, I have never seen it used)
  • horde.local.php
  • other overrides like prefs.local.php, backends.local.php
  • registry.d/ snippets
  • conf.d/ snippets
  • all vhost-specific overrides like conf-www.example.com.php, registry-www.example.com.php, prefs-www.example.com.
  • Hooks Files. (We need to ensure correct autoloading though)

Support for existing installations

There is currently no software level support for upgrading a Horde 5 installation from pear or git-tools to a Horde 6 installation. There is no Horde 6 release by Horde Upstream yet. Installations of the maintaina horde fork are treated as existing installations of Horde 6 for the purpose of this document.

To allow a smooth transition, horde-installer-plugin will be changed in this way:
For each file in the presets dir, check if it exists in the old and new location.
Normally, the old dir should not hold a copy of the file unless it is delivered by the package itself.
If the new /var/config/ folder is writeable, put the file there. Create subdirectories as necessary.
Finally, symlink all files from /var/config to corresponding old location, even if they are not from preset.

The logic for reading configs should prefer the new location, if available, but fall back to the old location. This must be addressed in some code paths in horde/core and possibly also somewhere in the individual apps.
Writing files should always go to the new location, though in theory the symlink should take care of that.

Outlook

Moving configuration out of the web-visible and composer-owned area is a major step forward. Future developments may reduce the web-visible surface of a horde installation even further.

Implementation progress

  • https://github.com/maintaina-com/horde-deployment as of 1.0.0 includes the var/config/ dir
  • https://github.com/maintaina-com/horde-installer-plugin as of 2.2.1 contains a working implementation. Configs from /presets/ are written to /var/config/ and symlinked to their traditional locations.

bookmark_borderHorde Authentication: Revisited

Horde Authentication as it is today has been devised more than a decade ago and it is surprisingly complex. Use cases and requirements have evolved. In a previous article, I introduced basic authentication and authorization needs and how they can be overwhelmingly complex to get right.

I will not go into details of how it currently works. Jan Schneider has done a very good series on this topic:

These articles written back in 2011 still reflect how things are done.

To sum it up:

  • There are different concepts of being logged in “to horde” and being logged in to a specific app
  • Horde uses Authentication Drivers for the initial login
  • Authentication Drivers offer different capabilities with impact on the ability to list or search users, create new users, change password etc.
  • Initial login can be either explicit through a redirect to login screen or implicit, called “transparent authentication”
  • While returning Horde sessions are transparent, resource backends like IMAP, LDAP, FTP etc may still need an explicit login per request.
  • Login sets up some orthogonal aspects of Horde like
    • presentation type (Dynamic, Traditional, Mobile, Minimal) – some restrictions and defaulting logic apply
    • Display Language and Localisation – also with some defaults of its own
    • Theme selection

Matters are complicated by Horde’s modular nature and a plethora of configuration options. Accessing the same installation through two different virtual hostnames may yield different authentication backends and different defaults or freedom in selecting themes.

Back when these design choices were made, nobody expected modern complexities to be relevant for a web application:

  • Normal users wanting to integrate desktop applications through CalDAV
  • Use of external trust providers and identity solutions like SAML, OpenID Connect
  • Prevalence of API use cases where other applications interact with Horde
  • Single Page Applications and Mobile Apps interacting with Horde like a remote system
  • Use of scope-reduced throwaway tokens to authenticate for limited purposes
  • Share token based guest access to specific resources

Let’s inspect some use cases.

I may want to authenticate my session by an external ID provider but still have some means of searching and listing other users when granting permissions to my resources.

I may want to rate-limit login attempts with bad passwords regardless of the authentication backend’s capabilities.
The same is true for enforcing a maximum passsword age.

When I make a REST API call, I want to explicitly avoid any state carried over from a previous request. Any session management is unwelcome overhead. Any logic dealing with presentation is just stealing response time.

Some APIs may accept either a Basic Authentication via username/password or some token ID.

Seeing the current number of unread mails on a portal page is welcome, but having to do an expensive imap login when I actually just look at the calendar is less desirable. More so, when the permission system does not even allow me to access the email component and the imap login is failing on each API call.

It seems important to isolate the different concerns from each other and break the mighty authentication logic into smaller, independent parts:

  • A User Repository for searching and listing known users
  • Actual validity check of a set of credentials, resulting in identification
  • User creation and deletion
  • Renaming existing users
  • Change Password to a chosen new password
  • Reset Password without choice

The implementation of these concerns should be strictly isolated from other aspects

  • Application Bootstrap: Setup of an environment, localization, specific view
  • normalization of a provided username, enforcing lowercase, adding a default domain or other
  • How the credential is delivered: The LDAP or SQL backend must not care if the password comes via HTML Form, HTTP Header or some CLI option.
  • Authorization related aspects like checking password age, bad login count, temporary or permanent blacklist of usernames, if the user is known as a super administrator
  • Access to backend resources.
  • Session expiration
  • Explicit logout

A UI login should only ever result in a login to an external technology like LDAP, IMAP, remote web resources etc if the current context actually uses these resources.

Recently I introduced the horde/http_server component which allows a middleware-based approach to these topics.
We can neatly move the different authorization filters into stacks of middlewares doing one thing at a time or adding a conditional middleware. We can reuse predefined stacks for common scenarios. We can delegate the composition of a plethora of options to some factory and configuration and keep our actual drivers very lean and our bootstrap procedure as concise as possible.

In one of the upcoming articles, I will provide some example cases.

bookmark_borderAuthentication & Authorization is complex

Could there be any more straight forward topic than authentication & authorization? The user provides user name and password and clicks “login”, the backend checks if credentials are valid. Invalid credentials are not authorized, valid credentials are authorized and identified (authenticated). End of story. Right? Well… in many cases, it’s not that trivial.

As a user, I want to be informed if I have to change my password soon.

As a security officer, I want accounts blocked for some time after a certain amount of failed login attempts. I also want passwords to expire after a certain time. I also want login sessions to expire if client IP address or browser identity changes.

As an integrator, I want to enhance the system to digest certificates, Shibboleth, SAML or OpenID Connect, Bearer Tokens, JWTs or even Kerberos Tickets.

As a support person, I want to silently normalize user login names, lowercase them or append domain names to login names.

As a usability consultant, I want to leverage the user database for UI, make user names searchable and browseable.

As a site administrator, I want to be able to filter out certain or most users from the backend even if they provide valid credentials or block login for all but a few users, i.e. for site maintenance.

As an innovator I want to join user bases from two different authentication sources and possibly migrate them on next login seamlessly, without them noticing.

As a sales person, I want to allow a limited guest user experience prior to login rather than force everbody to the login screen.

As a returning user, I want to transparently log in through a remember me cookie, but maybe make the application aware of that limited trust, asking for real login for sensitive operations.

As a developer, I want to be flexible and allow any combination of criteria. Users may login with a global password or a purpose-limited token, I want them to use a second factor like TOTP if they have set up one but pass if they haven’t, unless I don’t allow it.

For business reasons, I want to rate limit API access per hour, per day and per month with individual thresholds each.

As an auditor, I want each request’s authorization process to be logged for evidence.

There is a lot more to consider, but I will stop here.

Authentication is any means of making sure of a requester’s identity. The most common practice in computers asking for a username (identity) and a password (proof). Another common practice is asking some external authority we trust, an Identity Provider. When we look at a person’s passport to compare his photo or fingerprint with his actual face or finger, this involves the same aspect: The name written on the passport for identity. The photo or fingerprint for proof. And, implicitly, we trust the party who created the passport (Identity Provider) and maybe have some means to check the integrity of the document. But if somebody has no passport but a driver’s license, a club membership card, we might instead use this for verification.

Transparent Authentication is a special case where we can identify the user without explicitly interacting with him. This can be achieved whenever the request carries credible identifying information like a pre-established cookie, a certificate, a passport token whose integrity can be validated or other methods.

Authorization is any decision making if a requester has access to a resource. Simply being authenticated might not be enough. Guests without authentication may be eligible for certain areas of your application, but maybe not if they are from a certain IP range or country. A person may be too young or too old to use a certain facility. A person may be old enough to buy alcohol but cannot currently present a sufficient document to proof this. On the other hand, the bearer of a ticket may be eligible to visit some concert, with no interest in his actual identity. A software user may need to both be authenticated and part of a certain privilege group “administrators” to access a configuration screen.

Both requirements can be linked to each other, as well as all those aspects mentioned above.

In another article I will look at how Horde does it and discuss if this approach is still right for modern use cases.

bookmark_borderHorde’s HTTP component goes PSR

This weekend, I gave the horde/http component a some major redesign. See how things escalated. Oh my.
My minimum goals were namespacing, PSR-4 (Revised Autoloading Standard) and some minor, schematic adjustments. The final result is quite different. I ended up implementing PSR-7 (HTTP Message Interface), PSR-17 (HTTP Factories) and PSR-18 (HTTP Client). The code largely complies with PSR-12 (Extended Coding Style Guide) and thus, implicitly, PSR-1 (Basic Coding Standard). I am sure you will find some deviations and issues, so I welcome any Pull Requests against my repo. You will find the new code in /src/. The original, incompatible implementation is untouched and resides in /lib/. They can coexist as they are so different (namespaced vs unnamespaced, among others).

This is not a total rewrite. I could leverage most of the existing code base with some tweaks. This work would not exist without the foundation by Chuck Hagenbuch and all the contributions from the different Horde maintainers over the years. You will also notice similarities to other php PSR-7/18 implementations out there. I checked out Guzzle, httplug/php-http and some others. It was a great learning experience and I will not pretend I am not influenced by it.

As with all my modernisation activities, I made use of features allowed by PHP 7.4. This excludes Constructor Property Promotion and, sadly, Union Types as both are PHP 8. Union Types have been relegated to phpdoc annotations or check methods. Please mind most of the PSR’s target compatibility with PHP releases older than PHP 7.4 and thus do not sport return types or scalar type hints. I followed these signatures where applicable.

One major change between the old codebase and the new one is clients and Request/Response classes.
In the old implementation, there would be one client but different Request/Response implementations using different backing technologies like pecl/http, fopen or curl. The new implementation moves transport code to clients implementing PSR-18. Optionally, they can be wrapped by a Horde\Http\HordeClientWrapper which exposes the PSR-18 itself, but otherwise mimics the old Horde_Http_Client class.

Horde/http is used by very different parts of Horde, including the horde/dav adapter to SabreDAV, various service integrations (Gravatar, Twitter, …), the horde/feed library and application code all over the place. I intend to upgrade those use cases to the new implementation. I am looking forward to criticism or acceptance of that approach.

The goal of this project is more far-reaching. While Horde 4 and Horde 5 already had horde/controller, they made very limited use of it. In my non-public projects, I relied heavily on controllers and I made several attempts at improving the way controllers are set up in horde/core. However, I always felt the results were clunky and not really what I wished to achieve. While horde/controller knows prefilters and postfilters, these are not easy to use and there are few examples. While doing research, I made up my mind. I want to replace Controller/Prefilter/Postfilter with their PSR-15 (HTTP Handlers and Middleware) equivalents. Controllers will be Handlers, Pre/Postfilters will be Middlewares. Together they will be stacks. Authentication, Authorization, Logging etc will be relegated to Middlewares. There will be a default stack to mimic the default controller behaviour in Horde 5 (Be authenticated or be relegated to the login page). You will be able to define application-specific default stacks or request-specific stacks. As Middlewares are a public standard, we might be able to leverage middlewares existing out in the wild or attract microframework users to some horde built middlewares. I want to make it easier for coders to build horde apps without relearning everything they needed to learn for laravel, laminas or symfony. I also want to make it easier for everyone to cooperate. Horde is among the oldest framework vendors, predating most of PEAR and Zend. I think we still have some bits to offer.

Missing Bits:

  • While I did some implementation of UploadedFileInterface, it is still quite basic
  • UploadedFileFactoryInterface is missing as I have not yet built the server side use cases
  • Unit Tests need to be adapted to the new code base. Is there some PSR Acid Test out there?
  • I began implementing the ext-http (PECL_HTTP) backend but stopped as I am unsure about it. That extension is in version 4 now and still services version 3, but we have backends for versions 1 and 2. I need to learn more about it and decice if it makes sense to invest into that aspect.

bookmark_borderMaintaina Horde: Fourth Of July Additions

I packaged some more exotic horde apps and libraries for use with the composer installer and the FRAMEWORK_6_0 codebase. This was mostly formal conversion work, no actual testing was done. Some of these items might not even be very useful to most administrators. Nevertheless, I want to close the gap between what is available in the horde git-tools install and what can be installed through the horde installer plugin for composer.

These apps are now available:

  • refactor (refactoring utility)
  • trean (bookmarks)
  • ulaform (Forms app)
  • sesha (H4ish inventory app)
  • skeleton (template application)
  • sam (Spam Assassin Integration app)
  • operator (phone call details reader)
  • pastie (simple H4ish pastebin app)

The libraries were previously not available for H6 and now have seen their alpha release:

  • horde/thrift
  • horde/service_urlshortener
  • horde/service_vimeo
  • horde/service_facebook
  • horde/service_twitter
  • horde/scribe
  • horde/reflection
  • horde/pgp
  • horde/rampage
  • horde/oauth
  • horde/kolab_resource
  • horde/pdf
  • horde/lens
  • horde/openxchange
  • horde/mongo
  • horde/memcache

I wish you all a joyful fourth of july.

There are still some items missing, including the Klutz cartoon reader app and some Kolab related libraries.

bookmark_borderNo more master branch in maintaina horde

I am changing the default branch in all maintaina-com/ horde repos to be the FRAMEWORK_6_0 branch. In a separate next step, the master branch on these repos will be removed.

This is not due to any semantic discussion of technical terms like “master”. I am simply acknowledging reality. At the moment I cannot hope to get all those changes merged into horde’s proper master branches. I do incorporate their advances into maintaina but it is not going to the tight feedback loop I would wish to have.

I have no use for a branch pretending to be something different than or beyond the FRAMEWORK_6_0 venture. I had README’s in all those master branches warning against using them. It is way better to simply get rid of them now.

bookmark_borderHorde/Rdo ORM: PSR-4 and BC Breaks

Summary: Horde/Rdo ORM got upgraded for Namespaces. User code conversion is straight-forward. Backward Compatibility is limited.

If you ever wondered, RDO stands for Rampage Data Objects. This has been on my list for quite long, but it took some time to get it right. The horde/rdo library is horde’s Object Relational Mapping (ORM) solution. It allows you to store objects into sql databases or retrieve them without writing SQL. Originally, it has been written by Chuck Hagenbuch way back in the PHP 4 days and I have been a heavy user for years. If you know Laravel’s Eloquent, Doctrine, Hibernate, ActiveRecord, nHydrate or Dapper, this one is Chuck’s “as light as possible” implementation of the concept. Colleagues from B1 Systems have been users and contributors over the years. However, it has long been time to rethink Rdo in the light of newer capabilities of PHP 7.4 or even PHP 8. This time is now.

But you should not fear upgrading. First, the library still keeps the unnamespaced PSR-0 code, at least for the time being. Second, there is a straight-forward upgrade path for existing users.

Horde_Rdo_Base -> Horde\Rdo\Base
Horde_Rdo_Mapper -> Horde\Rdo\BaseMapper
Horde_Rdo_Factory -> Horde\Rdo\Factory
Horde_Rdo_List -> Horde\Rdo\DefaultList
Horde_Rdo_Iterator -> Horde\Rdo\DefaultIterator
Horde_Rdo_Query -> Horde\Rdo\DefaultQuery
Horde_Rdo_Exception -> Horde\Rdo\RdoException
Horde_Rdo:: -> Horde\Rdo\Constants::

It’s about as easy as it looks. Converting an application took me a few minutes. You might have noticed the names do not exactly match. Some names were not practical to simply turn into namespaced classes. In other cases, I turned class names into interface names. I found myself implementing the same enhancements over and over in multiple projects and I found myself wishing there was an easy way to do some others.

Less Boilerplate Mappers and Entities

Rdo is much more fun than some other ORMs as it comes with very little configuration. The library autodetects properties from the table columns. Datetime fields are automatically cast to Horde_Date objects. By default, Rdo tries for a convention over configuration approach for mapping table names, mappers, entities etc. Unfortunately, for most of my projects, that default does not fit too well to the class structure and file layout I choose. But still, implementing a new pair of mappers and entities takes two files and only two or three settings I ever need to think about.

Most often, subclassing the base mapper and the base entity is the right thing to do. But sometimes, you do not really care. If all you ever do to your entity is call ->toArray() and serialize it to json, you would be served very well by a generic entity instead. This is something on my list. I would even go one step further: If all you are changing to a mapper is subclassing and telling it the name of its database table and entity class, why subclass at all? Yes, I would want to turn the optional Factory class into something smarter. It will give you your mapper, be it an instance of the generic mapper with the right table name or something very customized.

Custom List Objects

Rdo queries always return a Horde\Rdo\List. This is a good default implementation and it makes common tasks easy. However, there are situations where you want your list of entities to be specific to an entity type or maybe a subclass of some base list class completely external to Rdo. Maybe you want to manipulate a list or merge results from two different queries.

Custom Entities
Sometimes the default entity implementation does not serve you well. There’s a range of things you would want.
– You want to inherit from a base class to attach behavior to your data. So you attach an interface and a trait to that foreign class to make it Rdo aware.
– You want to implement your own behaviour altogether
– You want Rdo to implement a proper repository with strongly encapsulated, less chatty domain objects. Rdo should provide a mechanism to produce those objects for you rather than having you cast or wrap Rdo\Base objects into your actual models. But it should not force you to think about such concepts before you really need them.

NoSql backends
Remember the name Rampage Data Objects? Rdo is mostly about mapping data to objects. It’s not about autogenerating the smartest SQL for the most obscure use cases. But once you have your prototype version ready, your first feedback comes in, you think about new features – and suddenly you want to support a new backend for some of your domain objects. Be it a NoSql database, a key/value store, a limited scope within a directory like LDAP or even a REST service. In a traditional horde application, you would wrap Rdo into a Driver called Driver/Rdo or Driver/Sql and implement a different backend. But what if you do not want to flip all your data to the new backend? Only the shopping list should go to the nosql backend but not the customers or the product inventory? You end up implementing individual drivers with individual backends. But you used Rdo’s relations feature … things get messy.

To achieve these capabilities, I want to make the Mapper less dominant. The formerly optional Factory gets promoted to take care of managing the right mappers, entities, backends, list types. This is what these interfaces are for. Mappers should mostly take care of mapping between an object class and a plain data format. Currently the mapper and query do too much, tightly coupled with the single mandatory list type. This will change.

Rampage Data Objects provides out of the box defaults for easy and common use cases. It gets you started really quick. We will add the capabilities needed when your application is maturing and your use cases get more demanding. This will be fun.

Backward Compatibility Breaks
The Horde\Rdo\Base* classes and their return types will be your best bet for backward compatibility. If you don’t try to use entities and mappers for side effects, you will be very safe. Factory’s constructor will change very soon. Factory should best be created from a Dependency Injector. Mappers should be created from Factory.

You should not rely on mappers exposing adapter or factory for creating other objects. Also, trying to manipulate sql session modes or transactions through Rdo’s adapter is not a good idea.

bookmark_borderWhat’s new in Maintaina Horde: Status 3/2021

  • CalDAV and CardDAV now run off SabreDAV 4 rather than SabreDAV 2
  • We now support both the Composer installer versions 1 and 2.
  • Nothing still depends on the PEAR protocol.
  • The Horde Icalendar Library now supports vCard 4. Still, importing/exporting vCard 4 or using it in CardDAV in the addressbook App Turba is not yet done. This requires good test coverage, syncing is not something I’d like to break
  • PHPUnit Tests are being upgraded from PHPUnit 4 to PHPUnit 9.
  • All libraries and groupware-related libraries are now packaged as “alpha” versions from the FRAMEWORK_6_0 branch

bookmark_borderMaintaina Horde: Now on Leap 15.2

I changed the maintaina.com Horde Images to use openSUSE Leap 15.2 instead of openSUSE Tumbleweed as a base. You should not experience any issues but I have not yet tested much. The change was necessary as I had build failures since 2021-02-17 in the github actions CI builds. It’s something about the docker used in ubuntu-latest being too old or github’s security settings being too strict. I did invest some weekend hours, but could not really solve it. I intend to switch back to tumbleweed at some point.

Update July 2021: I can now build tumbleweed-based images again.