Data sources for examples

When experimenting and self-training with Universal Content Puller, it can be useful to have some sample data to work with. 

When just pulling page areas and text content, you can pull such from other pages on your own site, from ConcreteCMS sites, from this site or anywhere on the web.

For more complex data sources, it can help to have some ideas. The examples shown on this site have used free and publicly available data sources detailed and linked below.

In addition to describing and linking the sources, we also have some suggestions on visualising the data to help you understand how the examples connect with the data sources.

CSV files

The CSV files used in our examples, including airtravel.csv and trees.csv have been drawn from free CSV samples at people.sc.fsu.edu/~jburkardt/data/csv/csv.html.

CSV files can also be viewed in any spreadsheet application, so you can get an overview of the data, columns and headings before working on it with UCP.

RSS feeds

For RSS feeds, a reliable source with a good variety of data types is NASA at https://www.nasa.gov/content/nasa-rss-feeds. In particular, we pull the Breaking News feed.

Whilst we think of them as RSS, they are in fact XML data, so can be used as sample content for any XML related experiments.

You can view an RSS feed directly in a web browser to get an idea of the XML structure.

Databases

Two well used and publicly available databases are

  • Northwind - Salesman and sales data (SQLite .db)
  • Chinook - Music and music store data (SQLite .db - zipped)

The links are for downloads in SQLite .db format files that can be used directly with the SQLite sources. When download, you can pull the databases as UCP sources directly from the database files, or upload them to the Concrete File Manager. To use SQLite files with the File Manager, you will need to enable the file extension.db in Dashboard > System & Settings > Files > Allowed File Types.

You can also download the sources as SQL files for importing to a variety of databases including MySQL, though it would not be a good idea to install them in the same database as ConcreteCMS.

  • Northwind - Salesman and sales data (MySQL .sql)
  • Chinook - Music and music store data (MySQL .sql)

Windows, Mac and Linux systems store much internal data in SQLite database files. When you start looking, there is SQLite data everywhere. For example, on my Fedora Linux development system, Chrome stores cookies in a SQLite database at /home/john/.config/google-chrome/Default/Cookies and other web browsers also store their cookies in SQLite tables. The actual cookie values are encrypted, but the other data columns provide data to play with.

If you are working with MySQL data, you should also install phpMyAdmin to explore and understand the data you are pulling.

The equivalent utility for SQLite is phpLiteAdmin. Once installed, to view databases in phpLiteAdmin, a configuration file phpliteadmin.config.php needs to be edited to provide an array of pointers to databases.

[code language="php"]

$databases = array(
    array(
        'path'=> '/path/to/chinook.db',
        'name'=> 'Chinook'
    ),
    array(
        'path'=> '/path/to/northwind.db',
        'name'=> 'Northwind'
    ),

);

[/code]

Additional Pages

About this Sidebar

Creating a sidebar for a group of pages without messing about with stacks is an easy use-case for Universal Content Puller.

This sidebar is edited once, within the main addon page for Universal Content Puller.

It is then pulled into all UCP sub-pages using a UCP block.

The Content Source is Parent Page, set to pull the Sidebar area from 2 pages from the top. The Content Transform is Selector, set to remove container and row classes that, when unnecessarily nested, could mess up the Bootstrap grid. The Content Display is Plain, which just outputs the transformed text.

In the advanced settings, sanitization is disabled as we trust the source page and don't want to strip out any formatting or functionality from the pulled sidebar.