Broken Spotify embed links are not recognized [old checker]

Hi @asafm7,

Hope this message finds you well.

Not sure what’s the issue, Spotify is returning a 200 which is the expected result since the link is not broken or is not being restricted (status 403). I am getting a status 200:

https://prnt.sc/1oZ9fL_tFE1G

When using Postman the method or UA might be blocked by Spotify, but, since is a link that will be embedded inside a Web the UA is not being blocked.

Can it be fixed? Maybe a different user-agent or an API?

Our Cloud version does use our API, but, our UA or IP due the its nature (crawler) might be blocked by Spotify and it will return a status 403.

Let us know your comments.

Best regards,
Laura

Thanks for your reply, @wpmudevsupport3.

Not sure what’s the issue, Spotify is returning a 200 which is the expected result since the link is not broken or is not being restricted (status 403). 

Actually, the link is broken and does return 403 with any other method. Please see here:

Screenshot 2024-02-18 185858_2.png

When using Postman the method or UA might be blocked by Spotify

It isn’t. Valid Spotify links return 200. Only broken ones, like the one I shared, return 403 (as they should).

I hope the issue is clearer now.

Please let me know if any information is missing.

Hi @asafm7,

Thanks for the clarification. Now is clear.

I notified our Broken Link Checker team, however, the Local feature is legacy and is not receiving updates or improvements anymore, so we can’t guarantee that this will be fixed in that version.

Once we get a reply from our team we will let you know on this topic.

Best regards,
Laura

@wpmudevsupport3

I would have switched to the new tool, but there are essential things that it can’t do, for example checking YouTube videos.

Maybe a combination of the old and new tools can be a good solution, but I don’t think your plugin allows it.

The issue might be the use of HEAD.

Maybe Spotify always returns 200 for HEAD, even if broken.

Unfortunately, the plugin doesn’t allow conditionally overriding it.

The blc_use_get_checker filter doesn’t apply in this scenario.

Maybe introducing a filter to replace HEAD with GET can be an easy fix that will probably help in more scenarios.

BTW, it seems like you have a bug in the http.php file.

I think that you should check for $use_get and not ! $use_get:

$use_get = apply_filters( 'blc_use_get_checker', false, $result );

		if ( $nobody && ! $result['timeout'] && ! $use_get && ( $result['broken'] || $result['redirect_count'] == 1 ) ) {
			// The site in question might be expecting GET instead of HEAD, so lets retry the request
			// using the GET verb...but not in cases of timeout, or where we've already done it.
			return $this->check( $url, true );

			// Note : normally a server that doesn't allow HEAD requests on a specific resource *should*
			// return "405 Method Not Allowed". Unfortunately, there are sites that return 404 or
			// another, even more general, error code instead. So just checking for 405 wouldn't be enough.
		}

Thanks.

Hi @asafm7

Thank you for response!

I need to consult that so I’ve asked our Broken Link Checker team about both the HEAD request itself and the filter and I’m awaiting their response.

We’ll update you here soon again.

Best regards,
Adam

Thank you, @wpmudev-support8.

Just adding that the blc_use_get_checker?filter?isn’t relevant here.

It is only being used if the original request was marked as broken. In our case, the request is marked as not-broken.

Hi @asafm7

Our BLC Team ask if you could share with us more example of those URLs so we can make a proper test.

Kind Regards,
Kris

Thanks, @wpmudevsupport13.

Yes, of course.

Broken examples:

• https://open.spotify.com/episode/24ASpZiURySlfoUaUxa7B4?si=eX426o8-SLisW130VixyyQ&utm_source=oembed
• https://open.spotify.com/episode/6HM4MReRyd85xa2ZfFoQkn?si=5-5ycUq-QZaeGPoAJcQ4EQ&h4l_mark=true&utm_source=oembed
• https://open.spotify.com/episode/67xBvenSiygMbCTGLqnKpl?si=DqJmbWzeTBuTWeHqfTDNDw&utm_source=oembed
• https://open.spotify.com/episode/2XqoF8Cjzl5JTRvkjxZyly?si=jUeNHlonSqy8W8r-O1Eumw&utm_source=oembed
• https://open.spotify.com/episode/5rlcJEMBkvMyx8zKyQiQCx?si=EeEDlhMJRbixsnuXM9ujGg&utm_source=oembed
• https://open.spotify.com/episode/1AqGcRf1mlnT1B3tq5Q0Hr?si=FEuYw6mSRpG7pCxnBHQqCg&utm_source=oembed

One valid link for comparison:

• https://open.spotify.com/episode/32vtKiHqNDQSIORqogfaQG?si=AFkll1VeT7eEf7Webb8ExQ&utm_source=oembed

Looking forward to updates.

Thanks.

Hi again @asafm7

Thank you for additional data. I passed them to our developers. Thank you for your patience while we look into this further.

Kind Regards,
Kris

Hello @asafm7

We can confirm those links are 403 on cloud version but return 200 on local version. We notice this is the User Agent we’re using in local version core.

Our BLC developers will work a on fix and it will be implemented in next versions.

As a temporary fix please navigate to:
/broken-link-checker/legacy/modules/checkers/http.php
live 179 and comment those lines

$ua = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.34 Safari/537.36';
		curl_setopt( $ch, CURLOPT_USERAGENT, $ua );

Kind Regards,
Kris

That’s great. Thanks, @wpmudevsupport13.

Maybe a filter for the request’s arguments can help with similar future cases.

@wpmudevsupport3, @wpmudev-support8, @wpmudevsupport13, is a fix planned to be released soon?

Thanks.

Hi @asafm7

We do not have any ETA at the moment when it comes to a fix. As for the suggested fix and Spotify links showing as broken now I pinged our BLC Team once more. Thank you for your patience while we look into this further.

Kind Regards,
Kris