Problem Solving

Problems you may come across and tips on how to solve them.

Scripts in pulled content are not executing

The most likely reason for scripts simply not executing is that they are sanitized out of the pulled content. Have a look at the Advanced tab Sanitize and Niceify section and make sure Sanitize output is set to none.

Beyond that, further problems are typically related to.

  1. AssetsUniversal Content Puller will load required assets for sources inside the site. However, UCP knows nothing about assets for external sources. You may need to explicitly load assets for extrenal sources using a header extra content attribute. Take care when doing so, pulling in assets for external sources can create security issues.
  2. Race - The JavaScript runs before its container is ready. The solution is usually to delay initialising until the container is ready for it, such as the when visble trick used to make maps work.
  3. Wait - The JavaScript is waiting for an initialisation such as a 'ready' event that has already happened. The solution is usually to trigger the event, though take care that some JavaScript may not like being initialised twice.

Do indexes start at 0 or 1?

In most cases, Universal Content Puller indexes start from 1. For example, rows and columns in a table or list. This is because it is most conveniently user-friendly.

The exception is anywhere that something is sliced, where the first item has an index of 0. So slicing blocks in an area or items in a list start at 0. This is because slicing follows php rules, including negative slicing from the end of a list. To change those rules could be even more confusing than an index that starts at 0.

 

Autolinking corrupts HTML

Autolinking and sanitization are applied on the output of the content transform, before the content display begins to format the transformed content for display. If the transformed content is an array, the autolinking and/or sanitization is applied on each element.

Universal Content Puller provides two alternatives for autolinking. 

  1. The core autlinker only works with plain text and fully specified URLs beginning http://... If you let the core autolinker loose on content that is already html it will corrupt attributes such a src and href.
  2. The advanced autolinker is more sophisticated and can work with content that is already html. It is also more capable of picking up abbreviated URLS such as www.mydomain.com... and URLs with query strings. Nevertheless, it is not foolproof. Apply it to content that mixes complex html and javascript and it may try and autolink something it isn't supposed to.

Overall, any complex html content should already have links in it and should not require autololinking.

Images appear as URLs, not as images

If a content source specifies a URL or path to an image without actually making that an html image element, it will by deafult appear as a URL.

To show it as an image, you need to enable the advanced autolinker and then select to Render image URLs as images. Converting image URLs to images is only available with the advanced autolinker.

How do I style pulled content?

Universal Content Puller by default wraps all pulled content in  div and ucp-body. Within complex content, each item in a list or table is also assigned a class by UCP.

The first option for styling is in the Advanced tab of the edit dialog, where you can change the wrapper element and class. For example:

  • Change the wrapper to blockquote and ucp-body to blockquote pulled content.
  • Chnage the wrapper to div and alert alert-info to put it into a bootstrap info box - assuming your theme is bootstrap based.

Beyond that, you can declare styles for the classes UCP has added to your site theme or to a header extra content attribute, as demonstrated in Example - RSS feed.

Further content sources, transforms and displays

Universal Content Puller is a growing framework for adding further Content Sources, Content Transforms and Content Displays.

If you can't find what you need, please ask me and explain your requirement. It may be that you just need a few tips on how to do it with existing plugins.

I have plans to add further sources, transforms and displays to UCP. Please chack these linked documentation pages for the latest list - other documentation may lag behind them.

The pluggable architecture means that as well as extension withing UCP, third party packages and /application/ classes can be used to add pluggins. This could be as simple as copying and modifying a display to add your own formatting to creating a completely new kind of display. As long as it follows the plugin architecture and has a unique plugin name, you can create as many custom plugins as you need. Please contact me for advice.

I am also available for contract development of UCP plugins if you need me. A UCP plugin will usually be considerably cheaper to develop than a full bespoke interface addon.

AJAX loading continues to show the placeholder

When Universal Content Puller is configured to AJAX load the pulled content (in the UCP edit dialog Advanced tab), a placeholder is shown until the AJAX load is complete.

If the placeholder continues to be shown:

  1. Make sure the UCP block is not cached. See the Cache settings in the UCP edit dialog Advanced tab.
  2. Make sure no form of Sanitize output is enabled in the UCP edit dialog Advanced tab. Sanitzation could strip necessary JavaScript or data attributes needed for the AJAX loading and pagination to work.
  3. Check the AJAX load is actually happening in your browser developer console, Network tab, XHR requests.
  4. Enable the Debug setting at the bottom of the UCP edit dialog Advanced tab and check the debug trace in your browser developer console. 

Pagination with more than one UCP block on the page

If you have more than one UCP block on a page requiring pagination, you have some choices on how that behaves:

  1. Leave the block identity out of the pagination. All UCP blocks will then paginate to the same page. Sometimes this can be what you want, like all the RSS Feed examples paginating to gether in Eaxmple - RSS Feed.
  2. Include the block identity in the pagination. In this case, when one UCP block is paginated, other UCP blocks on the page will go back to their first page.
  3. Set the UCP blocks to AJAX load. When a UCP block is AJAX loaded, any pagination for that block is also AJAX loaded. So all blocks can paginate by AJAX independantly.

Can UCP break my site?

Universal Content Puller won't break your site directly, but the content you pull could break the page it is pulled into.

Infinite Recursion

UCP includes built in recursion detection. Nevertheless, if you try hard enough you could create a recursive loop that pulls content within content within content... You will need to tray hard to do such, but it is theoretically possible.

Dangerous JavaScript

When pulling content from an external source, or any source you are not confident in, it is always best to start with Sanitize output set to Safe in the Advanced tab of the edit dialog. It is set that way in the default installation and unless you have a good reason to change it, is best left that way so you don't forget. JavaScript will be removed from the pulled content.

However,the converse is that where pulled content contains JavaScript that is required to execute, it won't do so unless you set Sanitize output set to None - see above.

Broken HTML

Just like an HTML block containing mis-matched or otherwise broken HTML tags could break the page it is placed on, if you pull content containing such broken tags it could break the page it is pulled into.

Imported Settings

You may have noticed that many of the Universal Content Puller examples have small button 'View settings' that pops up the settings used in that block as JSON data. You can copy and paste those settings into your own UCP blocks as a starting point for your own projects. But take care, the settings can often include things like concrete5 Page IDs or File IDs that will you will need to correct before saving the edit dialog to work with your site.

Solutions

Edit mode marker

By default UCP renders a marker in edit mode rather than the pulled content. This is so any broken content you pull cannot break concrete5 in edit mode.If you can get a page into edit mode, you can resolve any problems that break the front end view.

Page versions

If you can't get a page into edit mode, you should be able to go directly to the sitemap in the dashboard at yoursite.com/index.php/dashboard/sitemap. Once in the sitemap, click on a page to get the popdown menu and select versions. Once in versions, approve a previous page version and then delete the erroneous page version.

Panic mode

If all else fails, you can set UCP into Panic mode by using FTP or SSH to edit /application/config/generated_overrides/jl_universal_content_puller.php and set 'panic' => true. When panic is set, UCP will render all blocks as markers, even outside of edit mode. This will enable you to edit the problem block in concrete5.

Serialize

If you are unsure of content, in any UCP block you can set the Content Display to Serialize. This will output any pulled content as serialized data, so you can study it and head off any issues before committing yourself to displaying faulty content.

Omin Gallery or Universal Content Puller?

There is some overlap between the capabilities of Omni Gallery and Universal Content Puller. Both can create lists of things and display them.

  • Omni Gallery creates lists of images and displays them in ways optimised for images.
  • Universal Content Puller can create lists from pretty much anything, but the displays are optimised for lists and tables. These lists and tables may contain images, but they are not galleries or sliders.
View settings

As an aside, the above block was pulled from the page Problem Solving for Omni Gallery. Click the View settings button above for details. It is simply a matter of naming the source block, pulling the area and filtering on block name.

Whoops on install

What we mean here is an internal code exception in an addon or the core when you click the install button from the Add Functionality dashboard page or when you visit a page after installing an addon package. 

If you experience such an issue, here are a few things you can check that may resolve the problem.

  1. Php Version. Check your php version is compatible with the addon, For example, check if you are running php5.6 when an addon requires php7+. You can find the php version in the report at /dashboard/system/environment/info. As a general guide, if your site is concrete5 version 8+ then aim to run on at least php7.2. (It will run on lower versions, but less efficiently)
  2. Database Entities. concrete5 uses a database abstraction layer called Doctrine. Doctrine works by creating proxy classes that map between php data and the database and sometimes these get out of step or muddled into a chicken and egg dilemma. The dashboard page at /dashboard/system/environment/entities has a switch to control when the proxy classes are regenerated. Try placing this into Development mode when installing an addon package (but remember to disable development mode to speed up a live site)
  3. Cache. concrete5 makes extensive use of a number of different caching mechanisms to speed up web pages. When installing a package, sometimes outdated cached data causes issues with the install. Try disabling all the caches at /dashboard/system/optimization/cache when installing an addon package (but remember to re-enable the caches for a live site).
  4. Theme. Most marketplace themes are well behaved. Nevertheless, sometimes a theme can interfere with an addon package, so try switching the site theme back to Elemental using /dashboard/pages/themes. You won't loose any content on the front of your site, but some page areas may not be shown until you swap back to your theme.
  5. Middleware. Some addons implement middleware layers that manipulate pages before and/or after the main concrete5 rendering of each page. If such addons are installed, there is a possibly they could interfere with the dashboard and other addon installation. Most such middleware implementations include condition checks to ensure they don't mess with the dashboard, but perhaps you can switch them temporarily off just to be sure.

If you experience a code error on or immediately after install and need assistance, please use the Get Help link from the addon marketplace page to report the problem. On the Whoops report, click the [copy] button immediately below the error message. That will provide a stack trace you can paste into the help request and save time having to request that report later.

Search Indexing times out

Search indexing has nothing to do with UCP, its all handled by the core search index jobs. Nevertheless, the UCP block is complex and can be delayed by external sources. Should you find the Index Search Engine - All job is running into php execution limits and failing, you can reduce the batch size in the file concrete/jobs/index_search_all.php.

Unfortunately the batch size is currently hard-coded into the source. In a better world it would be a configuration parameter. Hence you need to override the job file or edit the original source. This is a case where editing the original source is not that big a deal. Just change

public $jQueueBatchSize = 50; // whatever smaller value works reliably.

Additional Pages

About this Sidebar

Creating a sidebar for a group of pages without messing about with stacks is an easy use-case for Universal Content Puller.

This sidebar is edited once, within the main addon page for Universal Content Puller.

It is then pulled into all UCP sub-pages using a UCP block.

The Content Source is Parent Page, set to pull the Sidebar area from 2 pages from the top. The Content Transform is Selector, set to remove container and row classes that, when unnecessarily nested, could mess up the Bootstrap grid. The Content Display is Plain, which just outputs the transformed text.

In the advanced settings, sanitization is disabled as we trust the source page and don't want to strip out any formatting or functionality from the pulled sidebar.