why wp_safe_remote_get?

Hi, WordPress can be configured in many different ways, including hosting plugin files on a remote server (e.g. on a CDN, etc.). As such, SLB must be able to retrieve these files regardless of whether they are configured to be stored locally or remotely.

Of course, further optimization is always planned, and optimizing this specific area is on the roadmap.

Also can you please provide additional details on how this relates to bot crawling?

Thanks.

Thank you for your answer.

My website have tens of thousands of pages. On a regular day i have only few hundreds users viewing few hundreds pages, but if a crawler decides to crawl the full website, there is a lot of pages for it to view.
A few days ago I had such crawler which resulted in retrieving “layout.html” over 190,000 in January. My homepage was accessed less than 7000 times this month.

Logs from the last 3 days (removed the server ip)
[~/logs]# find . -mtime -3 | xargs zcat 2>/dev/null | grep [ip-removed] | awk -F ‘”‘ {‘print $2’} | sort | uniq -c | sort -rn | head -10
124864 GET /wp-content/plugins/simple-lightbox/themes/baseline/layout.html HTTP/1.0

My hosting (SiteGround) say that this call is counted as “execution” which is a limited resource in my account.

Thank your for the additional details. You’re saying other pages are being crawled, which are loading the lightbox’s layout. This is true, but the lightbox layout would be loaded either way, though it might just show up in the logs differently (or not at all depending on your site’s configuration).

Also for clarification, cURL (used by WordPress’ wp_safe_remote_get()) is faster than a PHP function like file_get_contents(), so it’s not “wasting CPU”.

Thanks.