Configure LiteSpeed Crawler

The LiteSpeed Crawler is a part of LiteSpeed’s WordPress cache plugin. Usually, a cache page is created once the first user visits an un-cached page, any other user that will visit the same page will see the cached page.

Enabling the crawler functionality will simulate “visitors” crawling around your website pages, making sure your website cache is up to date and generated constantly.

Here are the steps needed to enable the cache –

1. Enable crawler in LiteSpeed server settings
2. Making sure relevant php functions are enabled
3. Install any missing php module
4. Extra step for cPanel servers
5. Setup crawler in LiteSpeed WordPress plugin

Enable crawler in LiteSpeed server settings

Log into your LiteSpeed’s server admin console, the default port for that is 7080 (https://SERVER_NAME:7080), then navigate to –

Configuration -> Server -> Cache

On that page, edit the “Cache Storage Settings“: Select “On“, “Crawler“, “ESI“, and then Save and restart LiteSpeed

Making sure relevant php functions are enabled

Edit the relevant php.ini file or all of them, they are usually located in “/usr/local/lsws/lsphpXX/etc/php.ini“, that really depends on your setup.

Look for the “disable_functions” section, and make sure that the following functions are *not* listed there –

curl_multi_exec, exec, pcntl_exec, popen, proc_close, proc_open

After editing and saving the file, restart LiteSpeed.

LiteSpeed crawler is using the “PHP XML” extension, make sure it’s installed and activated –

(XX stands for the version, for example, lsphp70)

yum install lsphpXX-xml

Extra step for cPanel servers

Log into your cPanel server, edit the “pre_main_global.conf” file, and add the following line –

CacheEngine on crawler

To apply change changes, execute the following –

/scripts/ensure_vhost_includes --all-users

Restart LiteSpeed once again.

Setup crawler in LiteSpeed WordPress plugin

• Navigate to LiteSpeed cache plugin settings
• Go to crawler
• Select General settings and enable crawler, then set server load and threads as appropriate for the server.
• Save settings

Tags: litespeed, Wordpress