Problem Solving

Problems you may come across and tips on how to solve them.

Nothing is shown

What we mean here is either that the UCP block displays nothing, or that it displays an error message to do with missing content and you need help to track down why.

UCP works through a series of stages to fetch and display content, so we need to find out at which stage the content you are expecting to show has become lost. Work your way through the steps below as relevant, then apply what you learn to your block.

  1. Check the source. You will have configured UCP to pull content from somewhere, so go there and make sure it is as expected.
  2. Make a not of your current settings - either by using Export and copying the JSON, or by copying the UCP block to the clipboard, or by taking a screenshot.
  3. Disable all caches, within concrete and within UCP. When you finish, remember to turn caches on again!
  4. Debug Mode. In the Advanced tab, enable Debug mode. Basic debug is mostly to the developer console and to the concrete log. Extended debug also provides copious information to the web page. This may provide some extra clues now, or later. When you finish, remember to turn debug mode off again!
  5. Swap the Content Display to Serialize or Serialize Paginate. This will enable you to see if the transform is providing the data you expect.
  6. If no content is displayed, swap the Content Transform to Pass Through. This will enable you to see if the source is providing the data you expect.

    By now, you will know whether the problem is in the Source, Transform or Display.
     
  7. If the problem is in the Content Display, particularly if you are using a complex display, go back to your original settings (2) and now swap the Content Display for something simple to configure, such as 'Table'. What you choose here depends very much on what content you are pulling. The idea is to understand your data, learn how to show it in a simple way, then build up from that, then apply what you have learned to the actual display you want to show.
  8. If the problem is in the Content Transform, leave the display as Serialize or Serialize Paginate and work with your transform settings until you get the correct serialized data showing in the display. Then add add back the display you want to show.
  9. If the problem is in the Content Source, leave the display as Serialize or Serialize Paginate and the transform as Pass Through and work with your source settings until you get the correct serialized data showing in the display. Then build forwards by putting your required transform back in, then putting your required display back in.

When you have finished, remember to turn off debug and re-enable any caches you disabled!

Scripts in pulled content are not executing

The most likely reason for scripts simply not executing is that they are sanitized out of the pulled content. Have a look at the Advanced tab Sanitize and Niceify section and make sure Sanitize output is set to none.

Beyond that, further problems are typically related to.

  1. AssetsUniversal Content Puller will load required assets for sources inside the site. However, UCP knows nothing about assets for external sources. You may need to explicitly load assets for extrenal sources using a header extra content attribute. Take care when doing so, pulling in assets for external sources can create security issues.
  2. Race - The JavaScript runs before its container is ready. The solution is usually to delay initialising until the container is ready for it, such as the when visble trick used to make maps work.
  3. Wait - The JavaScript is waiting for an initialisation such as a 'ready' event that has already happened. The solution is usually to trigger the event, though take care that some JavaScript may not like being initialised twice.

Do indexes start at 0 or 1?

In most cases, Universal Content Puller indexes start from 1. For example, rows and columns in a table or list. This is because it is most conveniently user-friendly.

The exception is anywhere that something is sliced, where the first item has an index of 0. So slicing blocks in an area or items in a list start at 0. This is because slicing follows php rules, including negative slicing from the end of a list. To change those rules could be even more confusing than an index that starts at 0.

 

Autolinking corrupts HTML

Autolinking and sanitization are applied on the output of the content transform, before the content display begins to format the transformed content for display. If the transformed content is an array, the autolinking and/or sanitization is applied on each element.

Universal Content Puller provides two alternatives for autolinking. 

  1. The core autlinker only works with plain text and fully specified URLs beginning http://... If you let the core autolinker loose on content that is already html it will corrupt attributes such a src and href.
  2. The advanced autolinker is more sophisticated and can work with content that is already html. It is also more capable of picking up abbreviated URLS such as www.mydomain.com... and URLs with query strings. Nevertheless, it is not foolproof. Apply it to content that mixes complex html and javascript and it may try and autolink something it isn't supposed to.

Overall, any complex html content should already have links in it and should not require autololinking.

Images appear as URLs, not as images

If a content source specifies a URL or path to an image without actually making that an html image element, it will by deafult appear as a URL.

To show it as an image, you need to enable the advanced autolinker and then select to Render image URLs as images. Converting image URLs to images is only available with the advanced autolinker.

How do I style pulled content?

Universal Content Puller by default wraps all pulled content in  div and ucp-body. Within complex content, each item in a list or table is also assigned a class by UCP.

The first option for styling is in the Advanced tab of the edit dialog, where you can change the wrapper element and class. For example:

  • Change the wrapper to blockquote and ucp-body to blockquote pulled content.
  • Change the wrapper to div and alert alert-info to put it into a bootstrap info box - assuming your theme is bootstrap based.
  • For UCP Charts, there are many styling options in the display dialogue. Charts also inherit font and background styling from the page, so can be styled through block design.

Beyond that, you can declare styles for the classes UCP has added to your site theme or to a header extra content attribute, as demonstrated in Example - RSS feed.

Bars or points in charts are being cut by the axis

Each scale in Universal Content Puller Charts can be set to No ScaleShow Scale or Offset Scale. If you find that bars in the chart, points or labels are being cut by the Y axis, selecting Offset Scale for the X axis will move the Y axis out and prevent it cropping the bar or point.

Similarly, If you find that points or labels are being cut by the X axis, selecting Offset Scale for the Y axis will move the X axis down and prevent it cropping the bar or point.

For some types of charts, you can also modify the minimum value of scales to specify exactly where they cross the other axis.

Further content sources, transforms and displays

Universal Content Puller is a growing framework for adding further Content Sources, Content Transforms and Content Displays.

If you can't find what you need, please ask me and explain your requirement. It may be that you just need a few tips on how to do it with existing plugins.

I have plans to add further sources, transforms and displays to UCP. Please chack these linked documentation pages for the latest list - other documentation may lag behind them.

The pluggable architecture means that as well as extension withing UCP, third party packages and /application/ classes can be used to add pluggins. This could be as simple as copying and modifying a display to add your own formatting to creating a completely new kind of display. As long as it follows the plugin architecture and has a unique plugin name, you can create as many custom plugins as you need. Please contact me for advice.

I am also available for contract development of UCP plugins if you need me. A UCP plugin will usually be considerably cheaper to develop than a full bespoke interface addon.

AJAX loading continues to show the placeholder

When Universal Content Puller is configured to AJAX load the pulled content (in the UCP edit dialog Advanced tab), a placeholder is shown until the AJAX load is complete.

If the placeholder continues to be shown:

  1. Make sure the UCP block is not cached. See the Cache settings in the UCP edit dialog Advanced tab.
  2. Make sure no form of Sanitize output is enabled in the UCP edit dialog Advanced tab. Sanitzation could strip necessary JavaScript or data attributes needed for the AJAX loading and pagination to work.
  3. Check the AJAX load is actually happening in your browser developer console, Network tab, XHR requests.
  4. Enable the Debug setting at the bottom of the UCP edit dialog Advanced tab and check the debug trace in your browser developer console. 

Pagination with more than one UCP block on the page

If you have more than one UCP block on a page requiring pagination, you have some choices on how that behaves:

  1. Leave the block identity out of the pagination. All UCP blocks will then paginate to the same page. Sometimes this can be what you want, like all the RSS Feed examples paginating to gether in Eaxmple - RSS Feed.
  2. Include the block identity in the pagination. In this case, when one UCP block is paginated, other UCP blocks on the page will go back to their first page.
  3. Set the UCP blocks to AJAX load. When a UCP block is AJAX loaded, any pagination for that block is also AJAX loaded. So all blocks can paginate by AJAX independantly.

Pulling from big data

When you need to pull content from a big data source, you need to think carefully about the way you connect. While Universal Content Puller can pull in pretty much any kind of data, for pulling content efficiently from really big datasets there are a couple of things you need to consider.

  1. Do not use a content source that loads an entire file before extracting the data. This rules out any file based content source such as CSV files. You also need to be careful about SQLite for large data sources, even though SQLite indexes and pages the .db files, it still needs to dig into file access.
  2. Use a content source that supports filtering and pagination in the source, so once connected it only returns the data you require.

With that in mind, the best way to connect to big data sources is to use the  Any Database or Any Database With Form content sources to connect, filter and paginate from a configured database connection in /application/config/database.php as described in the documentation page Connecting to Multiple Databases.

Where an external source provides a GET API, you may also be able to make paginated queries using the URL or URL With Form content sources and the pagination options they provide.

Can UCP break my site?

Universal Content Puller won't break your site directly, but the content you pull could break the page it is pulled into.

Infinite Recursion

UCP includes built in recursion detection. Nevertheless, if you try hard enough you could create a recursive loop that pulls content within content within content... You will need to tray hard to do such, but it is theoretically possible.

Dangerous JavaScript

When pulling content from an external source, or any source you are not confident in, it is always best to start with Sanitize output set to Safe in the Advanced tab of the edit dialog. It is set that way in the default installation and unless you have a good reason to change it, is best left that way so you don't forget. JavaScript will be removed from the pulled content.

However,the converse is that where pulled content contains JavaScript that is required to execute, it won't do so unless you set Sanitize output set to None - see above.

Broken HTML

Just like an HTML block containing mis-matched or otherwise broken HTML tags could break the page it is placed on, if you pull content containing such broken tags it could break the page it is pulled into.

Imported Settings

You may have noticed that many of the Universal Content Puller examples have small button 'View settings' that pops up the settings used in that block as JSON data. You can copy and paste those settings into your own UCP blocks as a starting point for your own projects. But take care, the settings can often include things like concrete5 Page IDs or File IDs that will you will need to correct before saving the edit dialog to work with your site.

Solutions

Edit mode marker

By default UCP renders a marker in edit mode rather than the pulled content. This is so any broken content you pull cannot break concrete5 in edit mode.If you can get a page into edit mode, you can resolve any problems that break the front end view.

Page versions

If you can't get a page into edit mode, you should be able to go directly to the sitemap in the dashboard at yoursite.com/index.php/dashboard/sitemap. Once in the sitemap, click on a page to get the popdown menu and select versions. Once in versions, approve a previous page version and then delete the erroneous page version.

Panic mode

If all else fails, you can set UCP into Panic mode by using FTP or SSH to edit /application/config/generated_overrides/jl_universal_content_puller.php and set 'panic' => true. When panic is set, UCP will render all blocks as markers, even outside of edit mode. This will enable you to edit the problem block in concrete5.

Serialize

If you are unsure of content, in any UCP block you can set the Content Display to Serialize. This will output any pulled content as serialized data, so you can study it and head off any issues before committing yourself to displaying faulty content.

Omin Gallery or Universal Content Puller?

There is some overlap between the capabilities of Omni Gallery and Universal Content Puller. Both can create lists of things and display them.

  • Omni Gallery creates lists of images and displays them in ways optimised for images.
  • Universal Content Puller can create lists from pretty much anything, but the displays are optimised for lists and tables. These lists and tables may contain images, but they are not galleries or sliders.
View settings

As an aside, the above block was pulled from the page Problem Solving for Omni Gallery. Click the View settings button above for details. It is simply a matter of naming the source block, pulling the area and filtering on block name.

Whoops on install

What we mean here is an internal code exception in an addon or the core when you click the install button from the Add Functionality dashboard page or when you visit a page after installing an addon package. 

If you experience such an issue, here are a few things you can check that may resolve the problem.

  1. Php Version. Check your php version is compatible with the addon, For example, check if you are running php5.6 when an addon requires php7+. You can find the php version in the report at /dashboard/system/environment/info. As a general guide, if your site is ConcreteCMS version 8+ then aim to run on at least php7.2. (It will run on lower versions, but less efficiently)
  2. Database Entities. ConcreteCMS uses a database abstraction layer called Doctrine. Doctrine works by creating proxy classes that map between php data and the database and sometimes these get out of step or muddled into a chicken and egg dilemma. The dashboard page at /dashboard/system/environment/entities has a switch to control when the proxy classes are regenerated. Try placing this into Development mode when installing an addon package (but remember to disable development mode to speed up a live site)
  3. Cache. ConcreteCMS makes extensive use of a number of different caching mechanisms to speed up web pages. When installing a package, sometimes outdated cached data causes issues with the install. Try disabling all the caches at /dashboard/system/optimization/cache when installing an addon package (but remember to re-enable the caches for a live site).
  4. Theme. Most marketplace themes are well behaved. Nevertheless, sometimes a theme can interfere with an addon package, so try switching the site theme back to Elemental using /dashboard/pages/themes. You won't loose any content on the front of your site, but some page areas may not be shown until you swap back to your theme.
  5. Middleware. Some addons implement middleware layers that manipulate pages before and/or after the main rendering of each page. If such addons are installed, there is a possibly they could interfere with the dashboard and other addon installation. Most such middleware implementations include condition checks to ensure they don't mess with the dashboard, but perhaps you can switch them temporarily off just to be sure.

If you experience a code error on or immediately after install and need assistance, please use the Get Help link from the addon marketplace page to report the problem. On the Whoops report, click the [copy] button immediately below the error message. That will provide a stack trace you can paste into the help request and save time having to request that report later.

Search Indexing times out

Search indexing has nothing to do with UCP, its all handled by the core search index jobs. Nevertheless, the UCP block is complex and can be delayed by external sources. Should you find the Index Search Engine - All job is running into php execution limits and failing, you can reduce the batch size in the file concrete/jobs/index_search_all.php.

Unfortunately the batch size is currently hard-coded into the source. In a better world it would be a configuration parameter. Hence you need to override the job file or edit the original source. This is a case where editing the original source is not that big a deal. Just change

public $jQueueBatchSize = 50; // whatever smaller value works reliably.

Manually Installing an Addon or Theme

Perhaps you are unable to connect your site directly to the ConcreteCMS marketplace to install an addon or theme. This manual process works for all addons and themes - not just mine.

  1. In your concrete account, find the addon/theme in your Purchase History > Licenses and download. Make sure to scroll down to the most recent version before downloading.
  2. Then copy the zip to your site /packages/ directory and unzip (not /concrete/packages !!!). This is usually easiest using your host cPanel file manager.
  3. In your site, visit the Dashboard > Extend page and the theme package will be ready to install.

The process is exactly the same for addons and themes, except themes have an extra step of activating the theme after installing.

Sometimes step 3 above can run out of PHP execution time. This is most likely when installing an addon or theme that installs a large amount of sample content. You should  not run into such an issue with any of my addons or themes.

If you do run into such issues, you can run the install manually from the shell command line.

If you find yourself needing to install or update many addons or themes manually, consider my Package Magic addon.

Once Package Magic Starter is installed, through the marketplace or manually, all further installs can be handled from the site dashboard using Package Magic.

Easy Setup

  1. On this site find an example similar to what you require, click the View settings button, copy the settings JSON.
  2. On your page, add a block. On the Support tab click the button to Import settings, paste the settings JSON and import,
  3. Edit the settings to what you require. 

Additional Pages

About this Sidebar

Creating a sidebar for a group of pages without messing about with stacks is an easy use-case for Universal Content Puller.

This sidebar is edited once, within the main addon page for Universal Content Puller.

It is then pulled into all UCP sub-pages using a UCP block.

The Content Source is Parent Page, set to pull the Sidebar area from 2 pages from the top. The Content Transform is Selector, set to remove container and row classes that, when unnecessarily nested, could mess up the Bootstrap grid. The Content Display is Plain, which just outputs the transformed text.

In the advanced settings, sanitization is disabled as we trust the source page and don't want to strip out any formatting or functionality from the pulled sidebar.