Problem Solving & FAQ

Search++ should be ready configured on installation for most sites. Nevertheless, if you encounter any difficulties, here are some tips to help you along and find tune the search experience.

Some content never shows in results

If some content or pages never show in the search results

1. Check the pages do not have the Exclude from search index attribute set.
2. Check the dashboard page at Dashboard > System & Settings > SEO & Statistics > Search Index. This contains a list of areas indexed and not indexed. By default, this is a blacklist. Areas selected are not indexed!
3. Run the Index Search Engine - All job at Dashboard > System & Settings > Optimization > Automated Jobs. In v9+, run the equivalent Index Search task.
4. Check the search is not excluded by security settings.
5. Use Dashboard > System & Settings > Search++ > Index Inspector to check the errant search terms actually appear in the index.
6. Maybe the results are there, but just cropped off the bottom of the list by low search scores and the minimum relevance threshold. Score weighting and relevance can be adjusted in the dashboard.

Only getting results for exact matches

This is the behaviour you would expect from the core concrete5 search. It can happen for two reasons:

1. Check you have edited your site's search results page and replaced the core search block showing search results with a Search++ Results block. The settings are similar to the core search block results settings.
2. In the Search++ dashboard page Full Text Options, make sure Full Text Search is enabled.

Too many irrelevant results

Search++ has a scoring mechanism that can be tuned to bias results in favour of particular aspects of the matching algorithm. The default settings in the Search Score dashboard page are a good starting point for most sites, so before adjusting them, have a look at the Search Results dashboard page.

Searches are scored and the top scoring result has a relevance of 100%. All subsequent results are assigned a relevance percentage with respect to the top scoring result and the Minimum Relevance percentage is the value below which results are not considered relevant.

If you are getting too many irrelevant results, try raising this value a little. If you are getting too few relevant results, try lowering it. Generally, somewhere between 20% and 50% appears to work reasonably well, but you may need to adjust further if you have changed the  Search Score weightings.

To help you fine tune this, the score and percentage score can be shown in search results by configuring the respective settings on the Diagnostics dashboard page. You can limit the users who see the scores to just Administrators, so you can see the scores but guests do not.

Not enough relevant results

This is the opposite problem to too many results. The simplest solution is to lower the Minimum Relevance. But you may need to do a little more than that. To help diagnose, configure the Diagnostics in the dashboard to log failed searches and/or to show search scores with the results.

• Check your database collation. A database that is not collated correctly for your concrete5 installation may affect search results.
• Make sure indexing is enabled for the page areas you have. For example, a theme with non-standard template area names will not be correctly indexed by default. Run the Rebuild Search Index concrete5 job.
• Consider mis-spellings and synonyms. You may need to configure common cases in the Synonyms and Abbreviations dashboard page. 
• Check that any custom blocks with content you are expecting to be found are actually designed to be indexed.
• You may need to adjust the weightings in the Search Scores dashboard page to bias results towards exact phrases or the page name or description.

To help you fine tune this, the score and percentage score can be shown in search results by configuring the respective settings on the Diagnostics dashboard page. You can limit the users who see the scores to just Administrators, so you can see the scores but guests do not.

Results break permissions

Search++ implements the concrete5 permissions system, so when configured properly will only list pages a user is permitted to see.

If you find users are seeing search results listing pages they are not permitted to see, the cause will be cache configuration.

Search++ facilitates caching at 3 scopes:

1. Guests only - guests share cached search data, registered user searches are not cached.
2. All users share cache - one cache contains all search results for all users, so a search made by an administrator could then have its results shared through the cache with a search made by a guest. Use this if you don't have any permission protected front-end pages.
3. Cache by user groups - cached results are only shared between users with exactly the same group membership.

So if you find users are seeing search results listing pages they are not permitted to see, check to see if All users share cache is enabled, then change it to one of the less permissive cache settings.

Synopsis text within a result is out of sequence

What we mean here is that text extracted from blocks appears in a different sequence to the blocks on the page. This is unfortunately a consequence of a concrete5 core bug, where the actual PageSearchIndex content is populated with blocks in the wrong sequence. Most of the time the index is correct, but sometimes the core just gets the blocks out of sequence when it extracts the searchable content.

With the PageSearchIndex content out of sequence, Search++ then extracts a synopsis and the rest follows. The causality where you may encounter this is:

• When the searched term appears very close to the start or end of a block
• and when the size of the synopsis is big enough to span the boundary between blocks
• and the PageSearchIndex has been populated out of sequence 

The same happens with the concrete5 core search because it is the data saved in the PageSearchIndex table that can be faulty. If you suspect this core bug, the faulty PageSearchIndex table can easily be confirmed using the Search++ Index Inspector.

There are some aspects of Search++ that can make it more likely to be noticed. Synopsis 'From top of content', 'At first match' and 'Bias to top' could be fed with incorrect data and can consequently pick a different synopsis to what would be shown with correct data. 'At densest match' is unaffected unless the synopsis length spans a boundary. If a synopsis is shorter, the issue with out-of-sequence PageSearchIndex content is less likely to be noticed.

From v9 the below is in the Concrete CMS core. Before v9 a partial resolution can be made by modifying the query the core uses to index searches. In file concrete/src/Page/Search/IndexedSearch.php, modify the query on line 130 to:

SELECT `bID`, `arHandle` FROM `CollectionVersionBlocks` 
WHERE `cID` = ? AND `cvID` = ? 
ORDER BY `arHandle` ASC, `cbDisplayOrder` ASC,

Rather than modify the code directly, the above is also available as an approved pull request on concrete5 Github.

This modification ensures that from v9 onwards or where the pull is applied blocks within areas are indexed in their on-page order, but does not resolve the issue for area sequence, or for layouts or other accumulations of blocks within areas. Hence it is an improvement, but not a complete solution to a correctly sequenced search index.

What are Stop Words?

Stopwords are all those words used to link together sentences, but in most cases don't actually have any relevance to the relevance of the subject. 

Words like and, is, of , at, about, from, that, where and when are all in the standard list of MySQL Stopwords. However, stopwords vary between installations and between languages. To view a list of the stopwords configured for your site, visit the Search++ dashboard Diagnostics page and enable List all stopwords in-page. This will dump the stopwords onto the search results page when you run a search.

Stopwords in MySQL are configured in the MySQL server settings and cannot be adjusted from within your concrete5 dashboard.

What is Stemming?

Stemming is a way of trimming off tailing variations of a word to find the root or stem of the word. For example:

• The stem of stemming is stem.
• The stem of developers is developer.

However, there is a lot more involved to stemming than just truncating training 's' or 'ing'.  Search++ uses an algorithm called Snowball. Stemming will try to configure Snowball for the currently active language.

If the stemming library does not have an option for the currently active language, a default language set in the Full Text Options dashboard page will be applied when words are stemmed. On install, the default is English.

My language has characters outside A-Z

Characters outside the configured character set are used to detect word boundaries and calculate word lengths when detecting three character words. Hence a word containing, for example, a German Ä may not be searched as you would like.

If your language includes characters outside the regular ascii A-Z, you may want to add them in the Character set extension section of the Full Text Search dashboard page.

Handling short words

The underlying MySQL free text search used by Search++ ignores all words shorter than 4 characters. Anything 3 characters or less is considered by MySQL to be a stop word.

However, especially in technical applications, we often have 3 character words or abbreviations. For example, an unmodified MySQL search would completely ignore "URL" and "CSS".

To get round this, Search++ provides a pair of mechanisms:

• Search++ makes a special case for 3 character words and builds them into a more complex search query. This is enabled and weighted on the Search Score dashboard page.
• Many 3-character words are acronyms or abbreviations, so when you know of specific 3-character words relevant to your site, you can add them to the Synonyms and Abbreviations list for special treatment.

What are visitors looking for?

Search++ can be configured on the Logging dashboard page to log all successful searches to the concrete5 log. The addon has its own log channel at Search Plus Plus, so you can easily filter the log to just show search information. Successful searches are logged as information with the query terms and score.

For autocomplete they are logged as notice.

What are visitors not finding?

Search++ can be configured on the Logging dashboard page to log all failed searches to the concrete5 log. The addon has its own log channel at Search Plus Plus, so you can easily filter the log to just show search information. Failed searches are logged as warnings.

If you find visitors are often misspelling words or using alternative words, you can add them as synonyms in the Synonyms and Abbreviations dashboard page.

Is search++ fast enough for my large site?

Search++ development was originally funded by customers with big documentation sites having thousands of pages and performed shockingly well, despite the complexity of the queries that Search++ can assemble.

The trick is the same as with any web site. You need a server sized to match your site size and load.

Since v0.2.6, Search++ also provides caching of searches. If the same search is executed multiple times, results can be pulled from the cache.

This support site is hosted on a typical shared hosting platform, so has considerably less server resources than a large business site would usually have.

If you are concerned over search speed, Search++ has options to show millisecond search timing with query results. See the Diagnostics dashboard page.

You can also install the concrete5 Speed Analyzer addon to examine how Search++ timings contribute to overall page rendering speed.

How do I style search forms and results?

Styling of results is relatively agnostic within a Bootstrap theme. Within most Bootstrap themes Search++ query and result blocks will simply fit in straight away or be easily configurable through minor adjustments to a block template's view.css.

For other theme frameworks, Search++ blocks use the same classes as the core page list and search blocks, so again should adapt relatively easily to styles your theme already provides.

Filtering for tags or other attribute values

Search++ doesn't currently support filtering by tags or other attribute values (It is on the roadmap if you would like to sponsor development).

An easy alternative that helps in most situations is to make sure the tag terms are in the searchable content of the pages you need to be found. This can be in titles, description or content. You can also add attribute display or tag list blocks to the page. These block types will need a getSearchableContent() method as described in "A block type is not being indexed" below.

You can then create links to the 'tagged' searches, for example:

• https://c5magic.co.uk/addons/search?query=Universal+Content+Puller
• https://c5magic.co.uk/addons/search?query=Magic+Tabs

Here these are styled as links so you can see what is going on, but they could just as easily be styled as buttons or even as a tag cloud.

The 'tagged' pages wont always come out top of the search results, but will usually be high on the list.

Whoops on install

What we mean here is an internal code exception in an addon or the core when you click the install button from the Add Functionality dashboard page or when you visit a page after installing an addon package. 

If you experience such an issue, here are a few things you can check that may resolve the problem.

1. Php Version. Check your php version is compatible with the addon, For example, check if you are running php5.6 when an addon requires php7+. You can find the php version in the report at /dashboard/system/environment/info. As a general guide, if your site is ConcreteCMS version 8+ then aim to run on at least php7.2. (It will run on lower versions, but less efficiently)
2. Database Entities. ConcreteCMS uses a database abstraction layer called Doctrine. Doctrine works by creating proxy classes that map between php data and the database and sometimes these get out of step or muddled into a chicken and egg dilemma. The dashboard page at /dashboard/system/environment/entities has a switch to control when the proxy classes are regenerated. Try placing this into Development mode when installing an addon package (but remember to disable development mode to speed up a live site)
3. Cache. ConcreteCMS makes extensive use of a number of different caching mechanisms to speed up web pages. When installing a package, sometimes outdated cached data causes issues with the install. Try disabling all the caches at /dashboard/system/optimization/cache when installing an addon package (but remember to re-enable the caches for a live site).
4. Theme. Most marketplace themes are well behaved. Nevertheless, sometimes a theme can interfere with an addon package, so try switching the site theme back to Atomik using /dashboard/pages/themes. You won't loose any content on the front of your site, but some page areas may not be shown until you swap back to your theme.
5. Middleware. Some addons implement middleware layers that manipulate pages before and/or after the main rendering of each page. If such addons are installed, there is a possibly they could interfere with the dashboard and other addon installation. Most such middleware implementations include condition checks to ensure they don't mess with the dashboard, but perhaps you can switch them temporarily off just to be sure.

If you experience a code error on or immediately after install and need assistance, please use the Get Help link from the addon marketplace page to report the problem. On the Whoops report, click the [copy] button immediately below the error message. That will provide a stack trace you can paste into the help request and save time having to request that report later.

Search Indexing times out

Search indexing has nothing to do with Search++, its all handled by the core search index jobs. Nevertheless, should you find the Index Search Engine - All job is running into php execution limits and failing, you can reduce the batch size in the file concrete/jobs/index_search_all.php.

Unfortunately the batch size is currently hard-coded into the source. In a better world it would be a configuration parameter. Hence you need to override the job file or edit the original source. This is a case where editing the original source is not that big a deal. Just change

public $jQueueBatchSize = 50; // whatever smaller value works reliably.

A block type is not being indexed

If you suspect a block type is not being indexed with the page, first off, double-check the simple things:

• The page attribute 'Exclude From Search Index' has not been set.
• The areas you have added the block to are being indexed. Dashboard > System & Settings > SEO & Statistics > Search Index
• Use Dashboard > System & Settings > Search++ > Index Inspector to check text from the block type actually appears in the index.

With the obvious double-checked, all block types are indexed by calling a getSearchableContent() method in the controller class. Find the block controller class and check for

public function getSearchableContent(){
    // method returns $text derived from the block content
}

Without that method, they have no searchable content and hence will not contribute any text to the PageSearchIndex table.

The solution is conceptually easy, you need to implement that method in the offending block controller or in an override of the block controller.

From version 0.3.3, Search++ provides a generic helper class for block search indexing. In the block controller you can implement:

public function getSearchableContent(){
    $sc = new \JtF\SearchPlusPlus\SearchableContentHelper($this);
    return $sc->getSearchableContent();
}

The SearchableContentHelper class is a bit of an overkill approach because it actually renders the block and strips out the text from that. For many block types, there will be a more efficient way of capturing searchable content.​​​​​​​ Contact me if you need further assistance with Searchable Content.

As a final check, from version 0.3.5 use Dashboard > System & Settings >​​​​​​​ Search++ > Index Inspector to investigate the search index table for specific pages.

Alternatively you can instead look at the PageSearchIndex table using phpMyAdmin or the MySQL command line with the query.

SELECT `cID`, `cPath`, `content` FROM `PageSearchIndex` 
  WHERE `cPath` NOT LIKE '%dashboard%' 
  AND `cPath` NOT LIKE '%!%'

Manually Installing or Updating an Addon or Theme

Perhaps you are unable to connect your site directly to the ConcreteCMS marketplace to install an addon or theme. This manual process works for all addons and themes - not just mine.

1. In your concrete account, find the addon/theme in your Purchase History > Licenses and download. Make sure to scroll down to the most recent version before downloading.
2. Then copy the zip to your site /packages/ directory and unzip (not /concrete/packages !!!). This is usually easiest using your host cPanel file manager.
3. In your site, visit the Dashboard > Extend page and the theme package will be ready to install or update

The process is exactly the same for addons and themes, except themes have an extra step of activating the theme after installing.

Sometimes step 3 above can run out of PHP execution time. This is most likely when installing an addon or theme that installs a large amount of sample content. You should not run into such an issue with any of my addons or themes.

If you do run into such issues, you can run the install manually from the shell command line.

$ concrete/bin/concrete5 c5:package-install my_package_handle

or

$ concrete/bin/concrete5 c5:package-update my_package_handle

When updating, be sure to replace the previously installed package directory rather than adding to it. If not, you could end up accumulating obsolete debris from a previous version of the package.

If you find yourself needing to install or update many addons or themes manually, consider my Package Magic addon.

• Package Magic Starter
• Package Magic (requires starter to be installed first)

Once Package Magic Starter is installed, through the marketplace or manually, all further installs can be handled from the site dashboard using Package Magic.