Not scanning with local or cloud

Hi @radgh

I hope you are doing well.

“The problem might be related to the front page not being part of the WordPress installation. This site does not have a front page. Also, some post types I would like to scan require you to be logged in.”

In this case may be a problem, the Cloud scan uses the front end to find out the links and the local would use the database. I assume they have the Published status right? https://monosnap.com/file/DXXPqFriCc0mEu91Ih8SB9cLO7SNwe

Could you switch back to the local, navigate to “Advaced” tab, enable the logs https://monosnap.com/file/ODFZnHxrYgSbiYL42eMza5PaGcWppP and force a new scan, if the plugin still not scanning the rest of pages / post share the log via email [email protected] using this specific subject: ATTN: WPMU DEV support – wp.org also include the thread URL on body.

Let us know the result you got.
Best Regards
Patrick Freitas

Thanks Patrick,

I disabled the front page redirect temporarily and started the scan with the cloud tool. That worked, but it completed too early. I don’t think it scanned any of the private pages. It found 320 total links, but one account page has 169 URLs itself so I think this number should be much larger.

I then set it to the local scanner and forced a re-check from the advanced tab. I don’t know how to start a scan with local so I just left it on the dashboard as it seemed to slowly make progress. After a few minutes it ended and only found 37 urls this way

I sent the log and additional information to the email address you provided. The main issue right now, is that the scanner doesn’t seem to scan the account pages. Maybe it’s because those pages require you to be logged in. Not sure if that is supported. I’ll await a reply to the ticket, but I’ll post this here to keep up to date.

Thanks,

Radley

Hello @radgh

Hope you’re doing well today!

Thank you for sharing detailed information and logs.

I have shared the logs with the BLC team for their further insights about the issue. We will update you here as soon as we have further information about the issue from them.

Thank you for your patience.

Kind Regards,
Saurabh

Hello @radgh

Hope you’re doing well today!

I have a quick update from the BLC team about the issue. Since the account page is behind the login the BLC local engine would only check the database for broken links it would not be able to crawl each page and hence the content rendering via shortcodes or any custom ways will be missed.

With that said, would you please let us know if the “Accounts Pages” are a custom post type and how they are created on the site? Also is there a way you will be able to allow a crawler by IP or a User Agent on the Account Pages post types?

Awaiting your response.

Kind Regards,
Saurabh

Hi Saurabh (@wpmudev-support7),

Yes it’s a traditional custom post type. The template checks if you are logged in and have access to the page based on your role and other information. I can absolutely make it bypass those checks so the pages appear “public” for a certain IP, user agent, or for requests with a secret query arg. Any of those options work.

Could you let me know (or email me) that information, and I can set it up and try again?

Thank you!

Radley

Hi @radgh

Thanks for response!

I believe it would be best in this case to stick to “Cloud” engine of Broken Link Checker and then, since you can allowe bypassed access to those pages, you would want to allow free access for following IPs

165.227.127.103
64.176.196.23
144.202.86.106

(source: https://wpmudev.com/docs/getting-started/wpmu-dev-ip-addresses/)

and/or following user agent

WPMU DEV Broken Link Checker Spider
(source: https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#broken-link-checker-user-agent)

If you are able to allow those so the site would allow access to those pages without the need to login and other security restrictions, it should help.

Note: if you are also using any kind of firewall on site/server, it would be recommended to allow them there as well.

Kind regards,
Adam

Hi again and thanks for the update!

Here’s what I did:

• I added the three IP addresses, as well as my own IP, to a whitelist.
• I made it so account pages can be accessed by the IP without being logged in and fixed the navigation so it always uses the same menu with the most links.
• I added links to the footer to the account page so the spider can find it from the front page
• I confirmed this is all working by testing it on my own IP address while incognito, and it seemed to work.

Once I scanned using the cloud tool, it found the same 320 links as it did last time. It seems like the IP address workaround didn’t work.

EDIT: Nevermind! Of course I forgot the site was cached, so I cleared the cache and the cloud tool worked. It found a healthy number of 4143 links, 18 of which are broken. Success!

Thank you for the help with all of this. I’ll close this out. I appreciate the support!

• This reply was modified 6 months, 1 week ago by radgh.
• This reply was modified 6 months, 1 week ago by radgh. Reason: fixed it!

Hi @radgh,

EDIT: Nevermind! Of course I forgot the site was cached, so I cleared the cache and the cloud tool worked. It found a healthy number of 4143 links, 18 of which are broken. Success!

Thank you for the help with all of this. I’ll close this out. I appreciate the support!

Glad to hear its sorted. Have a great day ahead.