Problem Solving

Problems you may come across and tips on how to solve them.

Use the Preview button

Since v8.9.5, the UCP edit dialog has a Preview button to review the incoming data to the transform and display. This shows a dump of the data coming into those stages of UCP and is a great way of quickly checking the data is as expected or for working out which items to select to display.

Bear in mind the preview is not paginated, so for big data sources you may could end up with a lot of data in the preview.

Nothing is shown

What we mean here is either that the UCP block displays nothing, or that it displays an error message to do with missing content and you need help to track down why.

UCP works through a series of stages to fetch and display content, so we need to find out at which stage the content you are expecting to show has become lost. Work your way through the steps below as relevant, then apply what you learn to your block.

1. Preview the data. In the edit dialogue use the Preview button to check the incoming data to the transform and display. Make sure the data is as expected.
2. Check the source. You will have configured UCP to pull content from somewhere, so go there and make sure it is as expected.
3. Make a note of your current settings - either by using Export and copying the JSON, or by copying the UCP block to the clipboard, or by taking a screenshot.
4. Disable all caches, within concrete and within UCP. When you finish, remember to turn caches on again!
5. Debug Mode. In the Advanced tab, enable Debug mode. Basic debug is mostly to the developer console and to the concrete log. Extended debug also provides copious information to the web page. This may provide some extra clues now, or later. When you finish, remember to turn debug mode off again!
6. Swap the Content Display to Serialize or Serialize Paginate. This will enable you to see if the transform is providing the data you expect.
7. If no content is displayed, swap the Content Transform to Pass Through. This will enable you to see if the source is providing the data you expect.
   
   By now, you will know whether the problem is in the Source, Transform or Display.
    
8. If the problem is in the Content Display, particularly if you are using a complex display, go back to your original settings (2) and now swap the Content Display for something simple to configure, such as 'Table'. What you choose here depends very much on what content you are pulling. The idea is to understand your data, learn how to show it in a simple way, then build up from that, then apply what you have learned to the actual display you want to show.
9. If the problem is in the Content Transform, leave the display as Serialize or Serialize Paginate and work with your transform settings until you get the correct serialized data showing in the display. Then add add back the display you want to show.
10. If the problem is in the Content Source, leave the display as Serialize or Serialize Paginate and the transform as Pass Through and work with your source settings until you get the correct serialized data showing in the display. Then build forwards by putting your required transform back in, then putting your required display back in.

The data shown is not what was expected

The diagnostic process for data shown not being like you expected is similar to Nothing is shown above. The first step is to determine whether issue is with the Content Source, Content Transform or Content Display as with steps 1 to 7 above, then make further diagnostic steps as in 8, 9 or 10.

If you are working on a complex display such as a chart or multiple level list, work though a series of simpler displays to get there. Simpler displays enable you to diagnose any issues with the content transform and make sure the Content Display has the right data to start with. As many displays share similar settings, simpler displays enable you to understand some of the settings needed for more complex displays.

For example, if developing a chart you could work upwards through:

• Serialize - the raw data. If you are not seeing raw data that makes sense, the settings for the content source or transform need adjustment. Be sure you have the right data coming in to the display.
• Serialize Paginate - the raw data split into a list of items, these may roughly equate to rows or columns in your chart. Sometimes it is easier to understand the raw data when it is split into a list, so again this is often a useful step in getting the content source and content transform right before going on to a more complex display.
• Table - in most cases chart data is displaying a graphical representation of a table, so getting the right data in a table is usually also a useful step in getting your data into a chart. Bear in mind that data in a chart may often be transposed compared to the best way of looking at it in a table. Tables are often organized by row, whereas a chart is often organized by column. You want to get a table that maps easily to your intended chart, which may not be the best table for actually showing the same data to visitors.
• Chart from Table - The settings for displaying a table should enable a chart to display. The final step of using a chart display can concentrate on formatting the chart because the correct data is already there.

Also with content display, there is often more than one way to get the right data shown. For example, sometimes you can filter or reorganize data in the display, but you could also achieve a similar display by filtering, reorganizing, slicing or transposing the data in a transform.

Scripts in pulled content are not executing

The most likely reason for scripts simply not executing is that they are sanitized out of the pulled content. Have a look at the Advanced tab Sanitize and Niceify section and make sure Sanitize output is set to none.

Beyond that, further problems are typically related to.

1. Assets - Universal Content Puller will load required assets for sources inside the site. However, UCP knows nothing about assets for external sources. You may need to explicitly load assets for extrenal sources using a header extra content attribute. Take care when doing so, pulling in assets for external sources can create security issues.
2. Race - The JavaScript runs before its container is ready. The solution is usually to delay initialising until the container is ready for it, such as the when visble trick used to make maps work.
3. Wait - The JavaScript is waiting for an initialisation such as a 'ready' event that has already happened. The solution is usually to trigger the event, though take care that some JavaScript may not like being initialised twice.

Do indexes start at 0 or 1?

In most cases, Universal Content Puller indexes start from 1. For example, rows and columns in a table or list. This is because it is most conveniently user-friendly.

The exception is anywhere that something is sliced, where the first item has an index of 0. So slicing blocks in an area or items in a list start at 0. This is because slicing follows php rules, including negative slicing from the end of a list. To change those rules could be even more confusing than an index that starts at 0.

Autolinking corrupts HTML

Autolinking and sanitization are applied on the output of the content transform, before the content display begins to format the transformed content for display. If the transformed content is an array, the autolinking and/or sanitization is applied on each element.

Universal Content Puller provides two alternatives for autolinking. 

1. The core autlinker only works with plain text and fully specified URLs beginning http://... If you let the core autolinker loose on content that is already html it will corrupt attributes such a src and href.
2. The advanced autolinker is more sophisticated and can work with content that is already html. It is also more capable of picking up abbreviated URLS such as www.mydomain.com... and URLs with query strings. Nevertheless, it is not foolproof. Apply it to content that mixes complex html and javascript and it may try and autolink something it isn't supposed to.

Overall, any complex html content should already have links in it and should not require autololinking.

Images appear as URLs, not as images

If a content source specifies a URL or path to an image without actually making that an html image element, it will by deafult appear as a URL.

To show it as an image, you need to enable the advanced autolinker and then select to Render image URLs as images. Converting image URLs to images is only available with the advanced autolinker.

How do I style pulled content?

Universal Content Puller by default wraps all pulled content in  div and ucp-body. Within complex content, each item in a list or table is also assigned a class by UCP.

The first option for styling is in the Advanced tab of the edit dialog, where you can change the wrapper element and class. For example:

• Change the wrapper to blockquote and ucp-body to blockquote pulled content.
• Add responsive show/hide classes as described in Example - Resposive Tables.
• Change the wrapper to div and alert alert-info to put it into a bootstrap info box - assuming your theme is bootstrap based.
• For UCP Charts, there are many styling options in the display dialogue. Charts also inherit font and background styling from the page, so can be styled through block design.

Beyond that, you can declare styles for the classes UCP has added to your site theme or to a header extra content attribute, as demonstrated in Example - RSS feed.

Bars or points in charts are being cut by the axis

Each scale in Universal Content Puller Charts can be set to No Scale,  Show Scale or Offset Scale. If you find that bars in the chart, points or labels are being cut by the Y axis, selecting Offset Scale for the X axis will move the Y axis out and prevent it cropping the bar or point.

Similarly, If you find that points or labels are being cut by the X axis, selecting Offset Scale for the Y axis will move the X axis down and prevent it cropping the bar or point.

For some types of charts, you can also modify the minimum value of scales to specify exactly where they cross the other axis.

Tables for display or tables for charts

The display of data as a table is often a very useful step to check data and filtering before showing it as a chart.

However, the best way of showing data in a table for visitors is often not the best way of visualizing data as a prelude to showing it in a chart.

Tables intended to show data for visitors are often oriented with many rows and a few columns, where each item of data is shown as a row of the table.

Charts are usually drawn as lines or columns of data where each item is a 'column' of the chart in the x-axis.

Hence the organization of data is often transposed in a chart relative to the same data show optimally as a table.

When designing a chart, you want to get a table that maps easily to your intended chart, which may not be the best table for actually showing the same data to visitors. Your 'stepping stone' table will have a few rows that extends or scrolls off the right of the page with many columns.

Further content sources, transforms and displays

Universal Content Puller is a growing framework for adding further Content Sources, Content Transforms and Content Displays.

If you can't find what you need, please ask me and explain your requirement. It may be that you just need a few tips on how to do it with existing plugins.

I have plans to add further sources, transforms and displays to UCP. Please chack these linked documentation pages for the latest list - other documentation may lag behind them.

The pluggable architecture means that as well as extension withing UCP, third party packages and /application/ classes can be used to add pluggins. This could be as simple as copying and modifying a display to add your own formatting to creating a completely new kind of display. As long as it follows the plugin architecture and has a unique plugin name, you can create as many custom plugins as you need. Please contact me for advice.

I am also available for contract development of UCP plugins if you need me. A UCP plugin will usually be considerably cheaper to develop than a full bespoke interface addon.

AJAX loading continues to show the placeholder

When Universal Content Puller is configured to AJAX load the pulled content (in the UCP edit dialog Advanced tab), a placeholder is shown until the AJAX load is complete.

If the placeholder continues to be shown:

1. Make sure the UCP block is not cached. See the Cache settings in the UCP edit dialog Advanced tab.
2. Make sure no form of Sanitize output is enabled in the UCP edit dialog Advanced tab. Sanitzation could strip necessary JavaScript or data attributes needed for the AJAX loading and pagination to work.
3. Check the AJAX load is actually happening in your browser developer console, Network tab, XHR requests.
4. Enable the Debug setting at the bottom of the UCP edit dialog Advanced tab and check the debug trace in your browser developer console. 

Pagination with more than one UCP block on the page

If you have more than one UCP block on a page requiring pagination, you have some choices on how that behaves:

1. Leave the block identity out of the pagination. All UCP blocks will then paginate to the same page. Sometimes this can be what you want, like all the RSS Feed examples paginating to gether in Eaxmple - RSS Feed.
2. Include the block identity in the pagination. In this case, when one UCP block is paginated, other UCP blocks on the page will go back to their first page.
3. Set the UCP blocks to AJAX load. When a UCP block is AJAX loaded, any pagination for that block is also AJAX loaded. So all blocks can paginate by AJAX independantly.

Pulling from big data

When you need to pull content from a big data source, you need to think carefully about the way you connect. While Universal Content Puller can pull in pretty much any kind of data, for pulling content efficiently from really big datasets there are a couple of things you need to consider.

1. Do not use a content source that loads an entire file before extracting the data. This rules out any file based content source such as CSV files. You also need to be careful about SQLite for large data sources, even though SQLite indexes and pages the .db files, it still needs to dig into file access.
2. Use a content source that supports filtering and pagination in the source, so once connected it only returns the data you require.

With that in mind, the best way to connect to big data sources is to use the  Any Database or Any Database With Form content sources to connect, filter and paginate from a configured database connection in /application/config/database.php as described in the documentation page Connecting to Multiple Databases.

Where an external source provides a GET API, you may also be able to make paginated queries using the URL or URL With Form content sources and the pagination options they provide.

When pulling data from external sources, think about how often that data may change. If the data is not changing minute-by-minute, it is usually also a good idea to cache the data. Many UCP Content Sources have a cache option. There is also a Cache with Transform content transform that you can insert into a stage of the Pipeline transform. 

Pulling CSV and Google Sheet Data

Behind the scenes, Google Sheet data is pulled in the same way as a CSV file.

Ideally, you should design your sheets for easy use with UCP. That means making the first row a row of headings, then all subsequent rows are rows of data.

However, sometimes you need to pull in data from sheets that were created with other purposes in mind. Sheets with all sorts of text and notes you don't need before you get to the actual data. Sheets with multiple tables on a single sheet.

With such sheets, the first thing to do is to discard rows down to the first row you actually need. The CSV transform has a convenient setting for discarding the first N rows. A catch with this is that empty rows are not counted, so if the sheet has one or more empty rows before the row you want to start, they need to be subtracted from the number of rows to discard.

The CSV transform can get headings automatically from the first row of data (after any rows have been discarded). This will only work if all the headings are unique. If they are not unique, the import can be corrupted. Hence, only select 'use the first row as headings' if you have unique headings for every column in that row.

You can use the Preview button to check the incoming data to the transform and display, confirm the data is as expected and work out what rows and columns to actually display.

UCP cannot automatically discard empty columns. So you may also need to crop the rows and columns extracted. Cropping for rows or columns has an offset and length. Whilst other indexes in UCP begin at 1, cropping starts counting from 0 (zero). That is because the underlying code uses php's crop mechanism and to mess with the way it indexes would only confuse things further. An advantage is that a negative offset counts back from the end, so is good for getting the last few rows or columns of data.

If you select to transpose the rows and headings, that is only applied after rows and/or columns have been cropped.

The Elements selector in Universal Content Puller Elements shows too many elements

Universal Content Puller Elements searches for elements in all packages and in /application/elements. It then filters out some packages and elements and highlights elements declared as compatible because they are in an /universal_content_puller_elements/ subdirectory.

You can set some configuration lists in /application/config to customize what gets filtered on your site, see Fine Tuning the Element Selector.

Can UCP break my site?

Universal Content Puller won't break your site directly, but the content you pull could break the page it is pulled into.

Infinite Recursion

UCP includes built in recursion detection. Nevertheless, if you try hard enough you could create a recursive loop that pulls content within content within content... You will need to tray hard to do such, but it is theoretically possible.

Dangerous JavaScript

When pulling content from an external source, or any source you are not confident in, it is always best to start with Sanitize output set to Safe in the Advanced tab of the edit dialog. It is set that way in the default installation and unless you have a good reason to change it, is best left that way so you don't forget. JavaScript will be removed from the pulled content.

However,the converse is that where pulled content contains JavaScript that is required to execute, it won't do so unless you set Sanitize output set to None - see above.

Broken HTML

Just like an HTML block containing mis-matched or otherwise broken HTML tags could break the page it is placed on, if you pull content containing such broken tags it could break the page it is pulled into.

Imported Settings

You may have noticed that many of the Universal Content Puller examples have small button 'View settings' that pops up the settings used in that block as JSON data. You can copy and paste those settings into your own UCP blocks as a starting point for your own projects. But take care, the settings can often include things like concrete5 Page IDs or File IDs that will you will need to correct before saving the edit dialog to work with your site.

Solutions

Edit mode marker

By default UCP renders a marker in edit mode rather than the pulled content. This is so any broken content you pull cannot break concrete5 in edit mode.If you can get a page into edit mode, you can resolve any problems that break the front end view.

Page versions

If you can't get a page into edit mode, you should be able to go directly to the sitemap in the dashboard at yoursite.com/index.php/dashboard/sitemap. Once in the sitemap, click on a page to get the popdown menu and select versions. Once in versions, approve a previous page version and then delete the erroneous page version.

Panic mode

If all else fails, you can set UCP into Panic mode by using FTP or SSH to edit /application/config/generated_overrides/jl_universal_content_puller.php and set 'panic' => true. When panic is set, UCP will render all blocks as markers, even outside of edit mode. This will enable you to edit the problem block in concrete5.

Serialize

If you are unsure of content, in any UCP block you can set the Content Display to Serialize. This will output any pulled content as serialized data, so you can study it and head off any issues before committing yourself to displaying faulty content.

As an aside, the above block was pulled from the page Problem Solving for Omni Gallery. Click the View settings button above for details. It is simply a matter of naming the source block, pulling the area and filtering on block name.

Whoops on install

What we mean here is an internal code exception in an addon or the core when you click the install button from the Add Functionality dashboard page or when you visit a page after installing an addon package. 

If you experience such an issue, here are a few things you can check that may resolve the problem.

1. Php Version. Check your php version is compatible with the addon, For example, check if you are running php5.6 when an addon requires php7+. You can find the php version in the report at /dashboard/system/environment/info. As a general guide, if your site is ConcreteCMS version 8+ then aim to run on at least php7.2. (It will run on lower versions, but less efficiently)
2. Database Entities. ConcreteCMS uses a database abstraction layer called Doctrine. Doctrine works by creating proxy classes that map between php data and the database and sometimes these get out of step or muddled into a chicken and egg dilemma. The dashboard page at /dashboard/system/environment/entities has a switch to control when the proxy classes are regenerated. Try placing this into Development mode when installing an addon package (but remember to disable development mode to speed up a live site)
3. Cache. ConcreteCMS makes extensive use of a number of different caching mechanisms to speed up web pages. When installing a package, sometimes outdated cached data causes issues with the install. Try disabling all the caches at /dashboard/system/optimization/cache when installing an addon package (but remember to re-enable the caches for a live site).
4. Theme. Most marketplace themes are well behaved. Nevertheless, sometimes a theme can interfere with an addon package, so try switching the site theme back to Atomik using /dashboard/pages/themes. You won't loose any content on the front of your site, but some page areas may not be shown until you swap back to your theme.
5. Middleware. Some addons implement middleware layers that manipulate pages before and/or after the main rendering of each page. If such addons are installed, there is a possibly they could interfere with the dashboard and other addon installation. Most such middleware implementations include condition checks to ensure they don't mess with the dashboard, but perhaps you can switch them temporarily off just to be sure.

If you experience a code error on or immediately after install and need assistance, please use the Get Help link from the addon marketplace page to report the problem. On the Whoops report, click the [copy] button immediately below the error message. That will provide a stack trace you can paste into the help request and save time having to request that report later.

Search Indexing times out

Search indexing has nothing to do with UCP, its all handled by the core search index jobs. Nevertheless, the UCP block is complex and can be delayed by external sources. Should you find the Index Search Engine - All job is running into php execution limits and failing, you can reduce the batch size in the file concrete/jobs/index_search_all.php.

Unfortunately the batch size is currently hard-coded into the source. In a better world it would be a configuration parameter. Hence you need to override the job file or edit the original source. This is a case where editing the original source is not that big a deal. Just change

public $jQueueBatchSize = 50; // whatever smaller value works reliably.

Cache path exceeds...

A windows system can throw an exception about cache path length. The full message will be something like "Cache path exceeds Windows PHP MAX_LENGTH of 260 characters" and the name of the exception is WindowsPathMaxLengthException.

This arises as a combination of ConcreteCMS and php internal issues. From JtFResources v2.19.32 we have a work-round for this by introducing the configuration value max_key_length in the file application/config/generated_overrides/jtf_resources.php. On windows, this will be automatically set to a value suitable for most windows installations.

If you experience the above exception:

1. Check the version of JtFResources is v2.19.32. If not, update this addon.
2. Check the value of max_key_length. It may not be configured for your installation if it has been copied from another site.
3. Reduce the value of max_key_length until you cease experiencing the WindowsPathMaxLengthException.

In general, keep the value of max_key_length as high as practical for your installation. The ConcreteCMS core applies a hashing algorithm to all cache keys and a longer value for max_key_length reduces the (already extremely small) probability of cache key collisions.

Manually Installing or Updating an Addon or Theme

Perhaps you are unable to connect your site directly to the ConcreteCMS marketplace to install an addon or theme. This manual process works for all addons and themes - not just mine.

1. In your concrete account, find the addon/theme in your Purchase History > Licenses and download. Make sure to scroll down to the most recent version before downloading.
2. Then copy the zip to your site /packages/ directory and unzip (not /concrete/packages !!!). This is usually easiest using your host cPanel file manager.
3. In your site, visit the Dashboard > Extend page and the theme package will be ready to install or update

The process is exactly the same for addons and themes, except themes have an extra step of activating the theme after installing.

Sometimes step 3 above can run out of PHP execution time. This is most likely when installing an addon or theme that installs a large amount of sample content. You should not run into such an issue with any of my addons or themes.

If you do run into such issues, you can run the install manually from the shell command line.

$ concrete/bin/concrete5 c5:package-install my_package_handle

or

$ concrete/bin/concrete5 c5:package-update my_package_handle

When updating, be sure to replace the previously installed package directory rather than adding to it. If not, you could end up accumulating obsolete debris from a previous version of the package.

If you find yourself needing to install or update many addons or themes manually, consider my Package Magic addon.

• Package Magic Starter
• Package Magic (requires starter to be installed first)

Once Package Magic Starter is installed, through the marketplace or manually, all further installs can be handled from the site dashboard using Package Magic.

Easy Setup

1. On this site find an example similar to what you require, click the View settings button, copy the settings JSON.
2. On your page, add a block. On the Support tab click the button to Import settings, paste the settings JSON and import,
3. Edit the settings to what you require.