Feature Request, Working on Big Sites

Hey @lechon,

while the enterprise segment isn’t really the major segment which Simply Static will be built for (maybe wp2Static or a Saas solution would be better here), there will be some improvements for that:

Crons:

This eliminates the ajax progress you see when running a generation which often results in a slower task processing.

Batch size:

Currently Simply Static has a fixed batch of 10 when fetching URLs. With stronger hardware, this can be upped to e.x. 100. This value will be configurable.

PHP 7/8:

Also, the large availability of newer PHP versions allows Simply Static to use more modern tools for processing and task handling than PHP 5.6. One of the reasons why I increased the minimum PHP version now.

Redis / Memcached:

This is a bit of future talk, but currently Simply Static relies on $wpdb to create page entries in a custom table, this could be drastically improved with a more efficient db engine/cache storage like Redis and Memcached. Support for that isn’t planned yet but could be in the future.

Cloudflare:

This will be available. I’m using Cloudflare Workers for the transfer so that you can deploy it to Cloudflare Pages and provide the static page from there.

Hope that helps,
Patrick

I was also just experimenting with a large site and realized – after only 10% had been processed after an hour – that it’s not really viable for large sites currently.

Some thoughts/ideas:

1. Wouldn’t it be faster to copy images from wp-content/uploads/ instead of making HTTP requests?
2. Would it be possible to run something similar to rsync, checking whether files already exist in target directory, instead of starting every build from a blank slate?

#2 would probably speed up the process by a lot, for subsequent builds after the initial “cache” has been built, as most of the data will be from wp-content/uploads/.

Hi @patrickposner

Many thanks for taking the time to explain this, I totally understand. You really have a fantastic plugin with a bright future.

Cheers!

Hey @erickhavel,

thanks for your suggestions.

@lechon this could be also interesting for your use case.

I’ve done some exploration on that topic.
After some modifications to Simply Static I got the following results:

– Completed a static export with 12.000 pages/files in 3 hours
– the process indicator on the generate tab does not stop (which happens in the current version on larger sites).

I haven’t increased the batch size here, but I do another test within the next few days to see the difference.

This test was done with a cron instead of the ajax/ping solution implemented currently.

@erickhavel

ideas:

1) Normally, you would not include the entire /uploads/ folder in your static site, instead only that files which are actually used within the page.

This is something Simply Static is doing already (by discovering -tags for example) without the need to add them to the additional files setting.

This is something I will make more clear in the next update (e.x. removing the suggestions in additional files).

2) While this is an awesome idea, I assume it would result in a longer build time and a large storage problem for tracking modifications. I will test it, but I think partly builds (exporting just a subset of files/directories) are the way to go (which I’m already working on).

Cheers,
Patrick

• This reply was modified 4 years, 1 month ago by patrickposner.

@patrickposner

That is fantastic! thanks for the very detailed feedback.

1) Normally, you would not include the entire /uploads/ folder in your static site, instead only that files which are actually used within the page.

If your site doesn’t have any orphan uploads (not used in any page) then all of the images would be linked to from somewhere, leading to all of them having to be included in the static build. Or am I missing something?

In any case this is an awesome plugin and I will be using it for the many small sites I’m currently managing, simply to cut down on potential security issues. Thank you for your hard work, it will save me hours and hours of mine!

And if you were able to solve the issues with larger sites (way shorter build time essentially, at least on subsequent builds) I believe people could be interested in paying money for it as a premium plugin.