bookmark_borderCustom iCalendar data in Kronolith & Nag

The iCalendar exchange format is everywhere in Horde’s calendar (kronolith) and tasks (nag) apps. It is offered for manual import/export. It is the centerpiece of the CalDAV synchronisation protocol and various APIs of these apps. The format also plays a role in email invitiations and updates sent via iTip. It is a very powerful and well-accepted format. This is, unfortunately, a little bit of a problem. Fortunately, we also have a solution

The iCalendar format was originally defined by the IETF. They released RFC 2445 back in 1998. While Horde still understands that format, a newer version 2.0 was defined in RFC 5545 in 2009. An extension to the 2.0 standard was released in 2016: RFC 7986 defines additional attributes needed to describe related conferencing software resources or other technical aspects.

Server and client software has always been free to add custom attributes to the different components of the standard. For example, Horde Kronolith stores and reads a special attribute X-HORDE-ATTENDEE to mark event attendees who have a horde user attached. Mozilla uses attributes like X-MOZ-LASTACK and MS Exchange comes with a long list of custom attributes, among them X-CALEND. Understanding these attributes when processing a icalendar file is optional. However, servers are expected not to silently drop attributes they do not understand.

Up until now, both Kronolith and Nag did support a wide range of attributes, but far from all. Events are reconstructed from calendar backend contents. Unsupported data simply gets lost. As it is technically impossible to know all attributes anybody out there may use, the opposite definition was needed: An attribute is an “other” attribute if it is not in the list of attributes the app already handles. Both apps got a catalog of attributes they understand and a property for storing everything else. Support for actually storing and retrieving this data is currently limited to the SQL backend. I do not currently invest time in Kolab support, maybe I will never do that.

Also, while I took care to ensure that normal editing via the UI will not break the “other” data, I did not cross check with scenarios where ActiveSync is involved. Please report bugs as you find them.

CalDAV event Storage

For kronolith, an optional storage of unmodified caldav events as received has been added during development. I am not sure if I will keep this or remove it again. It is useful for debugging various support scenarios but under normal operation, it is not required. It may be useful for creating a cache of external caldav calendars though.

Extend to Turba Addressbook

Turba Addressbook uses a similar data format vcard and implements the sister protocol CardDAV. Things are indeed a little more complicated. There are three relevant versions of vcard 2.1, 3.0 and 4.0 and multiple extension RFCs with additional contact attributes exist. Worse, there is a host of X- attributes from popular desktop addressbook vendors, Evolution, Kontact, Thunderbird, Android, MS Exchange/Outlook… There are at least two different ways to represent contact groups and the whole thing is a little messy. But the real problem lies with Turba’s highly configurable nature. At least two backends need to be upgraded: Both SQL storage and LDAP play a key role in typical Turba use cases. The exact layout of each addressbook can be different even inside the same installation. The list of vcard attributes considered as natively supported needs to be calculated for each addressbook and additional attributes maybe need to be stored outside of the specific backend. This will take some time and consideration.

Still, Turba’s tendency to forget what it does not care for needs to be addressed. Otherwise users risk losing attributes on sync.

bookmark_borderTurba Addressbook (II) – Architecture

Welcome back to our mini series on Turba.

Part I covered all the features and integrations provided by Turba.
Part II gives a dive into implementation, code structure etc.
Part III will consist of proposals for a changed architecture.

In the first chapter we looked at Turba’s features, APIs, Protocols. In the current installment, I want to present the concepts and structure of the code.

Turba is among the oldest horde applications. As such, it contains parts from various stages of Horde’s development.
Basically, it’s a layered architecture, but not fully fleshed-out or fully separated.

  • Presentation layer
  • Application Logic Layer
  • Storage/Backend layer

This is plugged together with some framework-provided integration points

  • with the sync services and inter-app API, RPC
  • with the portal/blocks service
  • with the Backup API
  • with the Content Tagger

Presentation Layer.

Turba provides both a desktop UI and a mobile UI. The following is mostly about the desktop UI. Blog de culturismo total fitness de lee hayward: debes probar estos entrenamientos musculares híbridos primobolan depot inicio – culturismo femenino y control de la natalidad, entrenamientos de culturismo femenino youtube – la casa de juegos del kama sutra negro.

Turba’s UI is organized into client pages rather than using a controller/route approach. This means, user visible URLs include files with a .php suffix.
The client pages build the UI, but are also API endpoints in a very traditional sense, catch interaction from forms or buttons as get variables and trigger actions in the backend
A representative example:

https://github.com/horde/turba/blob/master/search.php#L155
try {
$share = Turba::createShare(strval(new Horde_Support_Randomid()), $params);
$vid = $share->getName();
} catch (Horde_Share_Exception $e) {
$notification->push(sprintf(_("There was a problem creating the virtual address book: %s"), $e->getMessage()), 'horde.error');
Horde::url('search.php', true)->redirect();
}

Typically the top part of each client page is initializing the application, catching request and environment/session variables, setting up the business objects.
The middle part usually orchestrates actions depending on present or missing parameter scenarios.

The lower part will actually output the Horde Topbar, utility javascript, and the actual page content.

A rather extreme example is the data import/export part: https://github.com/horde/turba/blob/master/data.php
It contains mappings, attribute filtering, a longer cascade of if’s and switch statements and after about 360 lines, the actual UI logic starts.

This might sound messier than it really is. Actual functionality is mostly factored out into separate classes and the UI uses both some View classes and a form library.
Turba’s UI is heavy in forms and tables for the very reason that editing and displaying a highly configurable addressbook with hundreds of fields of data is
very crud-like by nature.

The Turba UI uses three types of helpers to compose the UI:

  • View classes
  • The Forms library
  • HTML templates with PHP snippets

View Classes

The View classes are very similar to the horde/view library and replicate some of its functionality, but are not using or inheriting from it.

Forms Library

Turba is a prominent user of the horde/forms library.
This utility allows to dynamically compose forms or multiple fields, check for internal/formal validity of entered values, missing mandatory values, etc.
It couples both a readonly and editable representation with a lot of processing logic.
Forms relies mostly on server-generated HTML with small parts of javascript injected for usability improvements.
The JavaScript snippets may utilize PrototypeJs and Scriptaculous, two formerly popular mainstream libraries.

Templates

The HTML templates provide most of the actual presentation apart from the forms – though they also provide some HTML forms to drive interaction.

Mobile View

Turba provides a read-only, touch friendly mobile phone presentation based on jQuery Mobile. This is completely separate from the rest of the turba UI.

Application Logic Layer and Backend Layer.

These two layers are closely tied together so it makes sense to discuss them as one.
From a problem domain perspective, it would make sense to expect these items:

  • Multiple addressbook sources or backends. These are individual configurations using drivers. Multiple sources on the same ldap driver can represent different directories or different views on one.
  • Addressbooks
  • Addressbook entries or contacts
  • Groups which are both entries and contain entries.

Turba’s logic layer consists of

  • Representations of actual addressbook entries via the Object and Object_Group classes.
  • A collection of reusable static functions in the Turba class.
  • Parts of the base backend driver.
  • Exporters/Importers for formats like LDIF and vCard.
  • specialised forms which deal both with presentation and state transformations.

Turba Objects and Groups are the common ground here. The objects glue together the actual data from the driver, files from VFS, permissions managed by the driver, the object’s change history and tags in the tagger app.
Groups are the only subtypes of Objects or Entries. Other subtypes suggested by the vcard standard like organisations or locations get no special treatment. Groups act as virtual addressbooks or views on addressbook data.
Turba’s groups only work with turba-accessible contacts. They cannot reference external contacts from other sources.

The addressbooks don’t really show up as entity objects. They are arrays of passive data managed by different parties.
This makes part of the logic a little hard to reason about and to setup unit tests.

A lot of Turba’s logic is data transformation. Backends have a native representation of data as SQL columns, LDAP objects, etc… as well as possibly native key names.
A person’s name may be a column “object_lastname” in one backend, but an attribute “givenName” in another one.

This is complicated by a highly configurable list of fields each backend can hold.
A driver transforms these native formats into a uniform format, transform date strings into date objects, handle blobs etc…
Both the driver dependent format and the “turba” format are hashes. It is up to the driver to actually generate a list object containing individual addressbook entry objects.

Another functionality delegated to the driver is deriving TimeObjects from addressbook data. TimeObjects are really not objects but hashes, representing anniversaries or birthdays.

Permission management in turba is using three distinct approaches:

  • The permission system allows restricting who can view or edit a certain addressbook
  • One addressbook source, usually the SQL db, can be configured as using the Shares system. In this case, the user can delegate access to addressbooks to other users or groups or make them world readable
  • The backing technology can restrict the user further. For example, the LDAP driver can be used to either bind using a service credential or using the user’s credential. Different users could have completely different data presented based on LDAP ACLs.

In the next article of this series, I am going to propose some modernisation approaches for Turba and discuss how they bring benefits in maintaining or extending the software vs being a tedious refactoring exercise.