bookmark_borderDAVx5 CalDAV may break with Unicode symbols in Horde/Kronolith syncs.

If you get user complaints about broken CalDAV syncs with Horde, there’s many places to look at. In one particular instance, an event was created from travelling app Transportr into the stock android Calendar app. Through the DAVx5 sync app, the user wanted to push these events to Horde’s SabreDAV interface – and from there, also sync it to his desktop email solution, Mozilla Thunderbird.

5 programa efectivo de entrenamiento de culturismo – programa de entrenamiento de culturismo de 5 días estore de culturismo halotestin szkani hombre sin mangas con capucha chaleco de fitness culturismo largueros entrenamiento camisetas sin mangas.

However, his sync application told him about an error. The server administrator saw a 500 status code in the server log.

1.1.2.11 - user [24/Jan/2021:18:48:26 +0000] "PUT /horde/rpc/calendars/user/calendar~KL14jYhCMpbet4ecYzAg1bn/2bace303-f0d8-4df6-9652-baa5fb9e86c4.ics HTTP/1.1" 500 892 "-" "DAVx5/3.3.8-ose (2021/01/13; dav4jvm; okhttp/4.9.0) Android/11

The root cause was actually not in the software code but in the MariaDB database configuration. The calendar entry from Transportr included some Unicode icon characters like a fast train and some arrows. These characters are part of the standard unicode encoding, utf-8.

Now you might wonder: New installations of mysql and MariaDB default to a character set they call utf-8 since 2010 or so. This shouldn’t be an issue. However, what they call utf-8 is not what you would expect.

Some years ago, TV sets which did not support the full HD resolution were marketed as “HD ready”. In some sense, the default character set is “unicode ready” at best. The default data type saves on disk space by encoding a subset of utf-8 into up to three bytes. While this supports most natural language characters, it is only a fraction of what Unicode can offer. Database manufacturers are well aware that this is not something you should run nowadays that unicode icons like the hamburger are all over the place is user generated content. The mysql manual even says:

“Please use utf8mb4 instead. Although utf8 is currently an alias for utf8mb3, at some point utf8 is expected to become a reference to utf8mb4. To avoid ambiguity about the meaning of utf8, consider specifying utf8mb4 explicitly for character set references instead of utf8.”

Now that’s what I did. First I changed horde’s database encoding to utf8mb4 in conf.php:

$conf['sql']['protocol'] = 'tcp';
$conf['sql']['charset'] = 'utf8mb4';

Then I changed the mysql server’s and client’s default charset:

cat /etc/mysql/my.cnf
[server]
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
[client]
default-character-set = utf8mb4
[mysql]
default-character-set = utf8mb4
[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

I have obviously stripped a lot of file content not relevant to our story. After reloading the database, all new connections should use the real utf8 encoding and new tables should be created with the new standard. But what about existing content? We need to convert all tables and all their text-like columns, varchars, mediumtexts etc.

First, it’s backup time – better safe than sorry.

Then, let’s find all tables in our db server and feed them conversion commands.

mysql --database=horde -B -N -e "SHOW TABLES" | awk '{print "ALTER TABLE", $1, "CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"}' | mysql --database=horde

Do this in a downtime or ensure HA some way or other. It will take some time. Some sources suggest it might be sensible to also rebuild the tables with the optimize command. I am not very convinced, but it won’t harm.

mysqlcheck --auto-repair --optimize --user="hidden" -p --databases mysql horde

After this, repeat the sync test. It should work this time.

bookmark_borderHeads Up: Cannot login as horde admin anymore?

Please be careful: Horde has introduced a new default setting $conf[‘auth’][‘lowercase’] which acts like the auth hook used to do.

This might lead to issues when your default admin account is named “Administrator”.

It is actually a very useful setting. It ensures that any prefs and other profiles also work with case-insensitive backends regardless of how you type the user name.

How to fix?

a) Lowercase all admin names in your $conf[‘auth’][‘admin’] settings in conf.php

b) disable $conf[‘auth’][‘lowercase’] by setting it to false value.

bookmark_borderPEAR down – Taking Horde to Composer

Since Horde 4, the Horde ecosystem heavily relied on the PEAR infrastructure. Sadly, this infrastructure is in bad health. It’s time to add alternatives.

Everybody has noticed the recent PEAR break-in.

A security breach has been found on the http://pear.php.net webserver, with a tainted go-pear.phar discovered. The PEAR website itself has been disabled until a known clean site can be rebuilt. A more detailed announcement will be on the PEAR Blog once it’s back online. If you have downloaded this go-pear.phar in the past six months, you should get a new copy of the same release version from GitHub (pear/pearweb_phars) and compare file hashes. If different, you may have the infected file.

While I am writing these lines, pear.php.net is down. Retrieval links for individual pear packages are down. Installation of pear packages is still possible from private mirrors or linux software distribution packages (openSUSE, Debian, Ubuntu). Separate pear servers like pear.horde.org are not directly affected. However, a lot of pear software relies on one or many libraries from pear.php.net – it’s a tough situation. A lot of software projects have moved on to composer, an alternative solution to dependency distribution. However, some composer projects have dependency on PEAR channels.

I am currently submitting some changes to Horde upstream to make Horde libs (both released and from git) more usable from composer projects.
Short-term goal is making use of some highlight libraries easier in other contexts. For example, Horde_ActiveSync and Horde_Mail, Horde_Smtp, Horde_Imap_Client are really shiny. I use Horde_Date so much I even introduced it in some non-horde software – even though most functionality is also somewhere in php native classes.

The ultimate goal however is to enable horde groupware installations out of composer. This requires more work to be done. There are several issues.

  • The db migration tool checks for some pear path settings during runtime https://github.com/horde/Core/pull/2 Most likely there are other code paths which need to be addressed.
  • Horde Libraries should not be web readable but horde apps should be in a web accessible structure. Traditionally, they are installed below the base application (“horde dir”) but they can also be installed to separate dirs.
  • Some libraries like Horde_Core contain files like javascript packages which need to be moved or linked to a location inside another package. Traditionally, this is handled either by the “git-tools” tool linking the code directory to a separate web directory or by pear placing various parts of the package to different root paths. Composer doesn’t have that out of the box.

Horde already has been generating composer manifest files for quite a while. Unfortunately, they were thin wrappers around the existing pear channel. The original generator even took all package information from the pear manifest file (package.xml) and converted it. Which means, it relied on a working pear installation. I wrote an alternative implementation which directly converts from .horde.yml to composer.json – Calling the packages by their composer-native names. As horde packages have not been released on packagist yet, the composer manifest also includes repository links to the relevant git repository. This should later be disabled for releases and only turned on in master/head scenarios. Releases should be pulled from packagist authority, which is much faster and less reliant on existing repository layouts. https://github.com/horde/components/pull/3

To address the open points, composer needs to be amended. I currently generate the manifests using package types “horde-library” and “horde-application” – I also added a package type “horde-theme” for which no precedent exists yet. Composer doesn’t understand these types unless one adds an installer plugin https://github.com/maintaina-com/installers. Once completed and accepted, this should be upstreamed into composer/installers. The plugin currently handles installing apps to appropriate places rather than /vendor/ – however, I think we should avoid having a super-special case “horde-base” and default to installing apps directly below the project dir. Horde base should also live on the same hierarchy. This needs some additional tools and autoconfiguration to make it convenient. Still much way to go.

That said, I don’t think pear support should be dropped anytime soon. It’s the most sensible way for distribution packaging php software. As long as we can bear the cost involved in keeping it up, we should try.

bookmark_borderhorde trustr – A new horde CA app step by step

Trustr is my current project to create a simple certificate management app.
I decided that it is just about the right scope to demonstrate a few things about application development in Horde 5.

I have not made any research if the name is already occupied by some other software. Should any problems arise, please contact me and we will find a good solution. I just wanted to start without losing much time on unrelated issues.

My goals as of now:

– Keep everything neat, testable, fairly decoupled and reusable. The core logic should be exportable to a separate library without much change. There won’t be any class of static shortcut methods pulling stuff out of nowhere. Config and registry are only accessed at select points, never in the deeper layers.
– Provide a CLI using Horde_Cli and Horde_Cli_Application (modeled after the backup tool in horde base git)
– Store to relational database using Horde_Db and Horde_Rdo for abstraction
– Use php openssl extension for certificate actions, but design with future options in mind
– Rely on magic openssl defaults as little as possible
– Use conf.xml / conf.php for any global defaults
– Show how to use the inter-app API (reusable for xml-rpc and json-rpc)
– Showcase an approach to REST danabol ds in Horde (experimental)

The app is intended as a resource provider. The UI is NOT a top priority. However, I am currently toying around with a Flux-like design in some unrelated larger project and I may or may not try some ideas later on.

Initial Steps: Creating the working environment

I set up a new horde development container using the horde tumbleweed image from Open Build Service and a docker compose file from my colleague Florian Frank. Please mind both are WIP and improve-as-needed projects.


git clone https://github.com/FrankFlorian/hordeOnTumbelweed.git
cd hordeOnTumbelweed
docker-compose -f docker-compose.yml up


This yields a running horde instance on localhost and a database container.
I needed to perform a little manual setup in the web admin ui to get the DB to run and create all default horde schemas.

Next I entered the developer container with a shell
docker exec -it hordeOnTumbelWeed_php_1 bash

There are other ways to work with a container but that’s what I did.

 

Creating a  skeleton app

The container comes with a fairly complete horde git checkout in /srv/git/horde and a clone of the horde git tools in /srv/git/git-tools

A new skeleton app can be created using

horde-git-tools dev new --author "Ralf Lang <lastname@b1-systems.de>" --app-name trustr

The new app needs to be linked to the web directory using

horde-git-tools dev install

Also, a registry entry needs to be created by putting a little file into /srv/git/horde/base/config/registry.d

cat trustr-registry.d.php

<?php
// Copy this example snipped to horde/registry.d
$this->applications['trustr'] = array(
'name' => _('Certificates'),
'provides' => array('certificates')
);

 

This makes the new app show up in the admin menu. To actually use it and make it appear in topbar, you also need to go to /admin/config and create the config file for this app. Even though the settings don’t actually mean anything by now, the file must be present.

I hope to follow up soon with articles on the architecture and sub systems of the little app.

bookmark_borderHorde starts Crowdfunding for IMP Multi-Account feature: Funded after a week

Michael Slusarz of Horde LLC started a crowdfunding experiment: He offered a 3000 $ project at crowdtilt.com to back up development of the IMP multi-account feature. Multi-Account support allows users to manage multiple mail boxes within one horde account. The feature is meant to replace Horde 3’s fetchmail feature which has not been ported for Horde 4 and 5 because technically, it’s not desirable anymore.

Michael Slusarz: The old fetchmail functionality is not coming back. It simply doesn’t work coherently/properly in a PHP environment with limited process times (and is non-threaded).
The replacement MUST be the ability to access multiple accounts within a single session. But this is not a trivial change

After Slusarz started the fundraising campaign, long-time supporters and users of horde contributed funds.

Currently, after three days, more that 80% of funding have been raised. About 500 US $ are still missing. The change is not trivial and probably going into IMP 6.2 for Horde 5.

As mentioned previously, this is a multi-week project, at least from a project planning perspective. And that doesn’t include the bug-fixing that is likely to be significant, given the fact that this is 1) an invasive UI change and 2) is involving connections to remote servers.

That being said – this is something I personally would *really* like to see in IMP also, so I am willing to provide a discount and prioritize this over some other activities I am currently involved in.
[..]
* This won’t be available for IMP 6.1. This will go into 6.2, at the earliest.

The Horde IMP Webmailer is among the most popular webmail applications in the world. It is shipped with most widespread linux distributions like openSUSE and Debian and has been used to drive webmail and groupware applications for large-scale userbases all over the world.

Currently, Horde 5 / IMP 6 is integrated into the cpanel administration product.

Update: After roughly a week, by 2013-08-14 the crowdfunding tilted: 3090 USD had been contributed.

http://lists.horde.org/archives/imp/Week-of-Mon-20130812/055265.html

http://lists.horde.org/archives/imp/Week-of-Mon-20130812/055265.html
I proudly get to make the announcement that the IMP Multiple Accounts
feature has been fully funded, as we reached the funding goal on
Crowdtilt this afternoon: http://tilt.tc/Evs2

I wanted to take the opportunity to thank all of the contributors:

– Simon Wilson
– Luis Felipe Marzagao
– Ralf Lang
– Digicolo.net srl
– Elbia Hosting
– Thomas Jarosch
– Andrew Dorman
– Henning Retzgen
– Michael Cramer
– Harvey Braun
– SAPO/Portugal Telecom
– Matthias Bitterlich
– Allan Girvan
– Bill Abrams
– Markus Wolff
– CAIXALMASSORA (Jose Guzman Feliu Vivas)
– Wolf Maschinenbau AG (Samuel Wolf)

It feels good to put a definite milestone into the enhancement ticket:

http://bugs.horde.org/ticket/8077

Should be able to start on this soon… hopefully tomorrow. Still
undecided on which branch I’m going to do development in but I will
post information to the dev@ list once I decide. Those that
contributed may get status updates.

Once again, thanks to everyone for supporting the Horde Project. Not
only was this an interesting experience from my standpoint (hopefully
others as well), but now we will soon get a feature that is obviously
desired by a large portion of the user base.

michael

bookmark_borderHorde Configuration: How to move passwd app into settings menu

The passwd app is a password management/changing utility for Horde which normally lives in a menu “my account” in the toolbar.

While it has been released and is in production use at many sites, it is also under development to expand and improve the module.Bankdrukken training + kogelvrije schouders bouwen – Lee Hayward’s totale fitness bodybuilding drostanolone kratom-effecten op fitness, pre-workout en bodybuilding.

Passwd provides fairly complete support for changing passwords via Poppassd, LDAP, Unix expect scripts, SMB/CIFS (under unix), Kolab, ADSI, Pine, Serv-U FTP, VMailMgr, vpopmail, SQL passwords and other more complicated setups.For a certain horde 5 installation, I needed to move the passwd app under the gearwheel/settings menu and out of the toolbar. After setting up passwd to work correctly, I added only one line to the registry.local.php file:

<?php
$this->applications['passwd']['menu_parent'] = 'settings';
?>

Everything you put into the menu labelled “settings” automatically appears in the gearwheel menu.

Fitness gym workout beste bodybuilding workout routines beschikbaarheid van primobolan depot cable rope triceps curl – jefit oefendatabase – beste app voor training, fitness, lichaamsbeweging en bodybuilding voor android en iphone beste software voor het volgen van trainingen.

bookmark_borderHorde 5 enters release cycle – First Alpha of Sesha Inventory App available

All’s well that ends well. During the last few days, the first alpha releases of the Horde Framework and some of its core apps hit the announcement list. Horde 5 sports a completely revamped user interface which allows a much tighter integration of the portal dashboard, ajax mode applications like the IMP Webmailer or traditional mode applications (whups ticketing etc).

Horde 5 has a completely renewed user interface (Image shows the IMP 6 webmailer)

While IMP, turba and the Ingo Mailfilter are already available as alpha packages, the calendar (kronolith) is not yet done.

However, today Horde release the first alpha version of the sesha inventory app. I have been working on sesha and related packages since horde 4, but things dragged on.

Sesha allows to organize any kind of items in a searchable inventory. First you have to define properties like age, weight, length or location of an item.

A defined group of properties makes up a category, something like an inventory type: Books, DVDs, network interfaces or computer monitors all have very different sets of properties. With sesha, there is no limit on the things you can put into your catalog. Just create categories of properties and finally add stock.

Sesha has been released under the GNU General Public License and may be used free of charge.

bookmark_borderHorde 5 Preview: Sesha Inventory App 1.0 and updated Rdo library

Dear folks, I am very pleased to announce:
The Sesha Inventory application is ready for Horde 5 and it is in good shape. Sesha is a simple inventory keeping application which originally developed by Bo Daley and Andrew Coleman on Horde 3. The product was never officially released but it went into production at several sites. Sesha release cycle can now start together with the Horde 5 Alpha release cycle.

Sesha inventory can be configured to hold any number of stock categories with any number and type of attributes.
Like the original version, Sesha for Horde 5 can provide its stock categories as ticket queues for the horde ticketing application whups.
There are a lot of plans and ideas for upcoming versions but for this time the focus was on finishing a releasable product.There are no surprises for existing users of Horde 3 based sesha. Most work happened invisibly under the hood:

  • The Horde_Template library was exchanged by new Horde_View code
  • A migration script for database was added
  • Users can keep their original Horde 3 Sesha tables and data.
  • The sql backend driver was completely reworked into a driver based on the Horde_Rdo ORM library The new Driver Api provides enhanced search capabilities but the current frontend doesn’t make use of it. I do not plan to add any features to the classic view but start working on an Ajax view once the Horde 5 Redesign is completed. This may ship with Sesha 1.1 later on.
  • Object oriented code has replaced complicated hashes in many places

The Horde Rdo library is the new work horse inside Sesha. Rdo means Rampage Data Objects and is a lightweight ORM layer by Horde founder Chuck Hagenbuch. It maps database tables to PHP Objects. This is similar to the ActiveRecord pattern. Each database row can be turned into one Rdo item. For Sesha and another – non-public – software project, some enhancements went into the Rdo library for Horde 5:

  • Rdo now provides a caching factory or root object which speeds up creation of mapper objects
  • Methods for add, removing or checking many-to-many relations have been added
  • A number of edge case bugs have been fixed

I think the Horde 5 release cycle will start with alpha1 releases sometime in May. I know we’re a little late but it’s worth the wait.
That said, I welcome any early testing or updates of the language files. Provided everything works as expected, Sesha will be shipped with Horde 5 for OpenSUSE 12.2

bookmark_borderHorde 5 is coming / Horde 3 support ends

The spring 2012 release of the Horde Application Suite and Framework will probably be called Horde 5. In a recent discussion the majority of developers agreed on a new major revision for some changes that some view as minor backward compatibility break. Currently planned features include:

  • New standard UI for “traditional view”
  • Move of Ajax code from specific apps to a common framework
  • Release of a small inventory management app (sesha)
  • complete configuration via UI (likely)
  • Webmail: Write support for smartphone view
  • Calendar: Resource calendar support for ajax view

At the same time, Horde 3 will no longer receive any support. Horde 3 has been around since 2005 and really has reached its end of life.

Since the Horde 4 release, The Horde 3 family of applications has only received critical bugfixes and security updates, the last being released this february. You should really consider updating to Horde 4 – the transition from Horde 3 to Horde 4 has been tested and done by numerous people and the transition from Horde 4 to Horde 5 should run smoothly as both releases are PEAR based.

I have already removed all things horde3 from OpenSUSE-Factory. OpenSUSE 12.2 will not ship Horde 3 any longer. Depending on packaging progress, openSUSE 12.2 will very likely ship Horde 5 or the most recent Horde 4 release. Horde 4 maintainence will continue.

Horde 3 Packages in the server:php:applications repository (see here) will be available at least until openSUSE 12.1 runs out of maintainence. I won’t give these much attention though. Please also note Eleusis Password Manager will be dropped with currently no planned replacement.

bookmark_borderHorde Config: How to fill dropdowns with application data with configspecial

Horde provides system wide customisation and configuration of applications through php configuration files. These files can be edited by hand or written from an administrator config UI. This ui is automatically generated from a file called conf.xml located in your $application/config/ directory.

The config xml allows dropdowns, multiselect fields, tick boxes, radio buttons and even conditionally adding or removing a field or inserting a valid php expression.

For example a  dropdown box in the horde base application’s config is generated by this snippet:

<configenum name="use_ssl" quote="false" desc="Determines how we generate
  full URLs (for location headers and such).">2
   <values>
    <value desc="Assume that we are not using SSL and never generate https
    URLs.">0</value>
    <value desc="Assume that we are using SSL and always generate https
    URLs.">1</value>
    <value desc="Attempt to auto-detect, and generate URLs
    appropriately">2</value>
    <value desc="Assume that we are not using SSL and generate https URLs only
    for login.">3</value>
   </values>
  </configenum>

This is all nice but what if you need to provide application data rather than static values? The answer is configspecial

<values>
    <configspecial application="turba" name="sources" />
</values>

How does that work?

<configspecial> calls the horde api. the “application” part tells you which application’s api to call. You can either reference an application by its registry name (horde, imp, kronolith, turba…) or by its api name (horde,mail, calendar, addressbook)

What’s the difference? When you call turba, you get turba. When you call addressbook, you can hook into whatever application provides addressbook. For example, spam handling and ticket queues have been implemented by multiple applications. You can even implement your own handlers for any existing api.

The called application must have a method configSpecialValues() in its lib/Application.php class file. This method gets called and its only parameter is the “name” property from the xml. In our example it’s “sources”. This method will return an array of source names to use in your config screen.

    /**
     * Returns values for <configspecial> configuration settings.
     *
     * @param string $what  The configuration setting to return.
     *
     * @return array  The values for the requested configuration setting.
     */

    public function configSpecialValues($what)
    {
        switch ($what) {
        case sources:
            try {
                $addressbooks = Turba::getAddressBooks(Horde_Perms::READ);
            } catch (Horde_Exception $e) {
                return array();
            }
            foreach ($addressbooks as &$addressbook) {
                $addressbook = $addressbook['title'];
            }

            $addressbooks[''] = _("None");
            return $addressbooks;
        }
    }

Et voila – you have a list of addressbooks to choose from.