error crawling https://

hi . I saw this message trying to crawl my website.

Hello @kostasmic21

I trust all is well!

I’ve tried to scan your site from our end but got the following error:
Security: The target is responding with a 403, this might be due to a WAF or a plugin.

The Smarcrawl IP address or the crawl process is being blocked by a security plugin. Please first try whitelisting the IPs address in your security plugin:

165.227.66.214
45.55.78.242
35.171.56.101

If the issue still there, please try temporarily deactivating your security plugin and try to scan the page. If the scan will work, please let us know what security plugin you’re using.

Cheers,
Nastia

i disabled the plugin akismet. Could you try again please?

Hello @kostasmic21

I trust you are doing well!

I’m afraid the issue still persists. Are there any redirects in the .htaccess file added from your end? Please try temporarily renaming an .htaccess file to .htaccess-old and run the scan. Please rename the file back if the issue still persist.

It could be a conflict with a plugin on your site. Please run a quick test:

– Deactivate all plugins besides the SmartCrawl and try to crawl your site
– If all is well, please start activating one plugin at a time, and checking when the issue will be resolved.

Please have a site backup before running this test so you could restore it. If you have a staging site, please run this test there.

Have a good day!

Kind regards,
Nastia

following your instructions i deactivated the plugins besides the smartcrawl. The problem persists. I could give you access in my website in pm to check yourself.

Hello @kostasmic21

I hope you are doing well

Would you please try increasing max_execution_time? To do so, please add into the php.ini file this line
max_execution_time = 300

If the php.ini file doesn’t exist, please feel free to create it in the main site directory.

If the issue still persists, please email us directly to [email protected]. In a subject please include

Subject: “Attn: Nastia”

so it will get assigned to me and n the main content a link back to this thread.

Kind regards,
Nastia

Hello @kostasmic21

I hope you are doing well!

We haven’t heard back from you for a while now so we’ve marked this ticket as resolved. If you do have any followup questions or require further assistance feel free to reopen it and let us know here.

Kind regards,
Nastia