Concrete CMS includes a data building system called Express. Express builds data objects out of data attributes, the same Concrete CMS attributes that can be attached to pages, files and users. In Express, these attributes are used to configure and populate Express objects. The Express objects can be listed and can have relationships.

Sounds great, but before you jump in, some words of caution. I only use Express for quickly creating small and simple lists of data. I avoid object relationships. I avoid objects of more than just a few attributes. I avoid using it for data that is frequently updated. I avoid using it for business critical data. I avoid using it for anything but the simplest forms. With those limitations imposed, Express can be convenient.

Durability

My first issue with Express is that it is not robust. Should updating an express object be interrupted for any reason it can leave an object's attributes in an inconsistent state. Once broken, the underlying data is extremely difficult if not impossible to repair manually and certainly beyond the ability of the average user.

Compare Express to the alternative of a database table. The data is tangible and robust. You know exactly where it is. A row in a table either exists or it does not. Updating a row is atomic. If data does need fixing, it doesn't take much expertise to find it and update it using an admin tool such as phpMyAdmin.

Complexity

The second issue is that working with Express is complicated. There is a big learning curve for a developer wanting to do work with Express. There are no tools for viewing or editing it like you would a spreadsheet or a table in a database.

Perhaps this is a matter of familiarity. As a developer I have years of experience writing SQL for databases. But for Express I need to work with the relationships between classes of php objects and the many tables they map to in the database. The key difference is that documentation and education on SQL outnumbers documentation and education on Express by thousands if not millions of pages.

Performance

The third issue is that performance of accessing and writing Express objects does not scale. Every time an additional attribute is added to an Express object, the database reads/writes involved in accessing that object increase proportionally.

Its not the number of entities but the complexity of individual entities that doesn’t scale. Each attribute of an Express object involves separate database actions for at least an attribute key and an attribute value. An entity with 10 attributes involves 10 x that. An entity with 20 attributes involves 20 x that. So read and write speed decreases with the size of entities. An entity with 20 attributes also has 20x the opportunity for breakages to skew data.

Compare that to using database records directly. Adding an item involves one database action for the entire record. If you double the number of columns in the record, its still 1 database action, just more data in it. Updating a single database record is atomic. So much simpler and so much less opportunity to go wrong. If you need to see your data you can login with phpMyAdmin. If you need to integrate your data with other software, its a simple database read. So the size of records scales, its more robust, is tangible and its easier to work with.

After I mentioned this in a Concrete partners meeting, fellow developer Jeff Paethau ran some experiments. Jeff built a simple test object in Express and a similar one using Doctrine, a data abstraction layer that works directly with SQL. Then he created and destroyed 1000 of each and timed the process.

• Express: 170.6s
• Doctrine: 4.9s
• Doctrine with batch flush: 0.4s

I would expect a native SQL query to be at least as fast and probably a little bit faster.

How did it get there?

There is an old anecdote about a traveller stopping to ask for directions. They receive the advice "Well, I wouldn't start from here".

I can understand how successive incremental developments over 15+ years ended up with the current Express architecture. From early days developers have used Concrete CMS pages (collections) as data objects with attributes. At the same time, traditional form builders used dedicated tables for storing each kind of form element. But Concrete already had attributes as general purpose data elements. The natural progression was to use attributes as the back end for a form builder.

Then the next step was attaching attributes to abstract objects rather than pages/collections. Then have the objects relating to other objects and you get to Express. So Express has built a rudimentary object database on top of an abstract attribute system built on top of an SQL database, all connected together with php.

Taken individually, each small incremental step of that development chain makes sense. 

Now take a step back to view the problem as a whole "Well, I wouldn't start from here". Wouldn't it be easier to use the MySQL database directly?

A better solution

This probably won't happen in the Concrete CMS core, there is too much vested development and use of Express. Express needs a fundamental re-think of the underlying architecture. Take the same high level dashboard page and block interfaces of Express and change the code to work directly with tables and columns of MySQL data. Make data objects map 1:1 with records in the database.

With such a change of underlying architecture many of the inherent weaknesses of Express would simply disappear and Express would have a stable foundation on which to expand future functionality. Users could take advantage of a whole load of database functionality for free that development of the current Express attribute based storage architecture can never catch up with. Make Express about providing a great interface for designing and using the existing MySQL database system, rather than getting bogged down in developing a new database system out of attributes.

Form Reform

To be clear here, my Form Reform addon was not developed as a replacement for Express data objects. Form Reform was developed as a revolutionary form builder, to be both easier to use and better than Express forms and any other form builder. The default storage mechanism of Form Reform is direct to a MySQL table. Form Reform does not provide the relational side of Express. If you need that, you can do it better and more easily with SQL.

Form Reform has many other storage mechanisms available as options, including Express data objects for those who really want them.