Patrick Mylund Nielsen

Hi Sven,

Optimus Cache Prime does indeed, as Naifamoodi said, support sitemap index files. It will prime all of the URLs contained within each of the sitemap files that the sitemap index refers to. It can be run from any Windows or Linux box–it doesn’t have to be used on the server itself.

An example from an external machine:

./ocp https://example.com/sitemap_index.xml

Example with W3 Total Cache when running locally on your server:

./ocp -l /var/www/example.com/wp-content/w3tc/pgcache -ls _index.html /var/www/example.com/sitemap_index.xml

The advantage of running it locally with something similar to the second example is that OCP checks whether the pages are already cached, and only primes the ones that aren’t, resulting in fewer requests to e.g. Apache.

Hi Avin,

I can’t help you with the code itself, but if you change crawl_delay to e.g. 5 inside ocp.py, you can reduce the rate at which it opens new pages.

Hi Avin,

Well, it looks like local mode is working, at least. The 500 internal server error makes it sound like there was a temporary problem with the web server. When I run OCP on your website now, everything looks fine.

OCP crawls the pages with the highest priority first. I like your idea, though — I will make it so you can choose a crawl order in the next version.

Hi Avin,

Then it seems if you change the paths to the sitemap and pgcache in ocp.py from /var/www/noa.al/noa-re/… to /home/yourusername/public_html/noa-re/…, it will work. If you get the same error message as before — that it can’t find the folder — it’s probably not the right path.

If you go to WordPress options -> W3 Total Cache -> Install, does it tell you the path under “Rewrite rules”?

Hi Avin,

Sorry if this is a stupid question, but IS the website actually stored in /var/www?

What result do you get if you run the command find / -name "sitemap.xml" as root?

Hi Avin,

It’s hard to say without knowing what kind of access you have to the server, and what kind of server it is, but the problem is that the user running the cron job cannot access the files in /var/www, which means ocp.py can’t access them. How are you setting up the cron job? Can you specify that it should be run as an administrator? Do you have shell access?

HG should be able to fix it easily ?? It doesn’t sound like a problem with OCP, just basic file permissions. You don’t need to change the files or folders, ocp.py just needs to be run as the correct user. (Usually the user that owns the files in /var/www is www-data, apache, nginx, or similar.)

Sorry I can’t help more.

Hi Avin,

It looks correct.

It sounds like ocp.py doesn’t have access to /var/www/noa.al/noa-re/wordpress/wp-content/w3tc/pgcache or /var/www/noa.al/noa-re/wordpress/sitemap.xml, which would explain why it can’t find the sitemap or the cache files on the disk. You should run ocp.py as a user that can access those folders and files — root or, preferrably, the file/folder owner, e.g. “www”, “www-data” (check ‘ls -la’ in /var/www) — THEN it should work ??

Hi Avin,

You must use Local mode — what makes it fast is that it’s checking the files on the disk instead of the pages via web requests. What error do you get when using local mode?

Make sure that ocp.py can read the files in the ‘pgcache’ folder, by the way — for example by running ‘sudo -u www-data ocp.py’.

Hi Avin,

I just tested with Python 2.4, and got the same error. I fixed the line mentioned, but the module for parsing the XML (ElementTree) didn’t exist in Python 2.4, so it won’t work without major alterations.

I am almost completely sure it would work if you tried with Python 2.5-2.7. If you are able to compile applications, you can compile the Python sources without root access.

Please let me know how it goes ??

Hi Avin,

I will test it, but if you have any possibility of upgrading to Python 2.5, 2.6 or 2.7 (https://python.org/download/), that would be an easy solution. Python 2.4 is very old. (It’s from 2004.)

Hi Avin!

Glad to hear it will be useful for you ??

Could you tell me the URL of your sitemap and your Python version (‘python –version’) — I will try to recreate the problem.

Feel free to email it to contact at pmylund dot com

I too pondered this a bit and it seemed like the ideal would be to do the priming outside PHP, so I ended up writing a Python script, Optimus Cache Prime, which does essentially the same as the W3 priming feature, except for a few things: It supports throttling (to my knowledge, PHP can’t really do this without blocking other requests), but most importantly it checks the state of your static file cache before making any requests, so only the pages that aren’t already in the cache get crawled.

After doing some testing, it seems that OCP can check around 10,000 pages per second with WordPress and W3 Total Cache, assuming that the cache is already mostly primed (if not, there would be a lot of requests).

You need a server with Python and the ability to run a script to use it. Hopefully someone will find a use for it ??

I’m trying to install Disqus (2.49) for the first time, and whenever I go to the Disqus page it’s blank. So, me too! Needs more QA.