Failed during “Crawling initial file list” no front page

Past 10 lines of so, please do not post that in these forums. It doesn’t work and triggers the spam queue.

If you need to share that data then please use https://pastebin.com/ instead and just share the link to that paste here.

Jan: sorry about that.

I put my export log on dropbox (too big for pastebin), please replace the dashes below with “dropbox”:
__________.com/s/ql2ectiop3um0ao/WP-STATIC-EXPORT-LOG.txt?dl=0

WP-STATIC-PROGRESS.txt: 79
WP-STATIC-INITIAL-CRAWL-TOTAL.txt: 1145
404 log: 10 pages
Export log (debug mode): 3.5 megs
Theme: X-Code

I did some of the suggestions to debug.
– I bypassed cloudflare
– I uninstalled any unnecessary plug in and theme.
– I set max execution time to 600
– I set memory limit to 1024M
– I added www to the site url (I still don’t understand why the plugin requires the old url to be different then the new url. My plan is to create a static site once, deploy it at netlify, shut down the wordpress server and move the domain name.
– pretty sure that zip is enabled.

What I haven’t yet tried
– browser stuff – how would that affect a server process?
– php7.2. I am using 7.1. I got an error about mysql when I tried to switch to 7.2. I probably would need to figure out a bunch of dependencies to make it happen. How important is this?
– modsec – I haven’t yet turned it off as this seems to take some doing on WHM.
– modpagespeed – need to figure out if I have this
– not sure if curl is enabled
– didn’t disable all plugins as certain ones are needed to render the site
– didn’t try switching the theme.

I can do more of the steps, but maybe some are a lot of work, maybe there is an obvious solution before I do?

Any help would be great. Thanks!

Ok, I upgraded to php 7.2 and now it works. Except it still doesn’t create the homepage.

Ok, I found the home page at /page/1/index.html. I am going to see if I can move it to the root and make it work.

Hi @davebos,

Thanks for persisting with this and making some progress.

re the domains, you may get away with both the same, but I would suggest for Netlify, that you target https://somenewsub.netlify.com ensure that works, then change your DNS to point to Netlify, which will result in your primary domain looking fine, as Netlify will rewrite again behind the scenes. But if it works with the same WP URL and Destination URL, great!

Not sure what’s going on with the homepage, but if you can email me your Initial Crawl List and a Debug Log-enabled Export Log (Advanced tab to enable), I have a feeling I’ll find something in there: [email protected]

That URL you’ve found and /posts/page/1/ may soon change to /page/1/ for an unrelated issue, but we should be getting it exported at the site root /.

What I’ve seen happen before is that a URL like /?foo=bar&fee=beer or such gets inadvertently crawled, which overwrites the already correctly crawled / in the static export.

I am hoping we find such a URL in the initial crawl list preview or in the Debug Mode-enabled Export Log.

If we can track down this issue for your site, it may help with other people’s, too. I’m here to help you with this, but may be unavailable a bit tomorrow, whilst traveling.

Cheers,

Leon

PS. I will bump the minimum requirements to PHP 7.1 or 7.2 soon. I have a simplified version of the plugin that will work with older sites I’m going to make available, that can likely support PHP 4 or earlier versions and will still do an OK job at exporting. Once exported, that can be deployed with the latest plugin version…

There were a few things I needed to deal with manually, but no big deal. Site is now fully static sitting on bitbucket and served by Netlify.

I re-ran the export using just the base url without the www and that didn’t cause any additional issues. The nice part is that once the site is static I can do a control shift F in VS Code and make changes as necessary.

I was playing with stuff so I overwrote the logs that I used to create the export that I actually used for the production site. I had since moved the original wordpress site to old.baseurl.com to keep while I am letting the static live site burn in. I just re-ran the export. I am emailing you a link to download the logs and export zip.

Note that on this run, it did properly create index.html. On my prior run, moving the page/1 thing to index.html at the root worked perfectly, so no big deal once I found where the page was hiding.

It does seem that some of the Yoast produced site maps aren’t gettin download. /sitemap-index gets exported fine, but seems that wpstatic doesn’t crawl everything referenced in the index. For example, /x-portfolio-sitemap.xml which has a lot of important stuff.

Of course, there was no attachment sitemap as the attachment pages didn’t download, but I didn’t find much value in those pages anyway (I am talking about the WP produced attachment html pages not the attachments themselves that worked fine).

I have since moved the original site to a new temporary url so not sure if I rerun the export it will give you useful info.

Anyway for my purposes of a one time move, it worked great. I would need to play with it a lot more if I needed to sync a wordpress staging area to a static live site.

Hi Dave,

Please try with the latest 6.6.3 release and let me know if everything’s sorted now.

Cheers,

Leon