Post missing, search results’ order

Hi @rfischmann

I have updated WPFTS to the 1.67.231 but do not update your local copy this moment.

Please could you make this short test in order to see difference:

1. Run this MySQL query and record results somewhere

select
	(select count(*) from wpftsi_index) n_index,
	(select count(*) from wpftsi_docs) n_docs,
	(select count(*) from wpftsi_words) n_words,
	(select count(*) from wpftsi_vectors) n_vectors,
	(select count(*) from wpftsi_tw) n_tw
	

2. Update your local copy of the WPFTS plugin to the 1.67.231 and Rebuild the Index

3. Please run the MySQL query from #1 again and again record the numbers (should be different, n_vectors should be way large)

4. Run the latest big MySQL query from my previous post and export CSV for me, please.

I have a big hope that we found the problem and I fixed it. Let’s see.

Thanks!!!

Of course, @epsiloncool, will follow your instructions right now.

It just finished now, @epsiloncool, about 2h15 of processing time.

I don’t know what would you consider “way” larger for n_vectors — it is larger, but I don’t know if “way larger”.

Here are all the CSV files you’ve requested: https://d.pr/f/677tTj

BTW, at least the Spirit Awards and AppleCare posts are correctly showing up in search results. However, the order of results still bothers me a lot I must say. I wish I could go back to how it was before v1.65. ??

Hi @rfischmann

Thanks for the reports, I am investigating them now.

According to search results order – could you please check which options you selected on the pages:

Sure thing:

Filter: https://d.pr/i/u9BLLC
Relevance: https://d.pr/i/wome94
Sort Results: https://d.pr/i/H1vm1x

I didn’t change any of that since before the v1.65 update.

Well, order settings looks good. However, they can be overriden by some other plugin.

Could you please explain in words, why the order is bad now? How it should look you think?

I don’t remember if I have changed ordering logic in the latest WPFTS update, so reasons may be different.

What we can check also is Analytics tab. There is a list of all search queries and we can see details. Please click to the latest query and open the Explained tab. There is a item called “Query Parameters (actually used)”, please check “order” and “orderby” values. In case they are not “relevance” and “DESC”, they was overriden by some 3rd party plugin.

Just checked that and it looks good, @epsiloncool: https://d.pr/i/0rNVsN

What’s bothering more, and that changed when I updated to v1.65, are some searches for posts that we post every week. For example, posts with the 5 most read articles of the week.

Here’s a search for “mais lidos” (that’s “most read” in Portuguese): https://d.pr/i/Ko7UZO
And the same query from the Sandbox Area: https://d.pr/i/LxfGZb

As you can see, the first result doesn’t even have “mais lidos” in its title, which is very weird in my opinion.

All the other results do have that, but I feel the order is very strange. I’d expect, and that’s how it was before v1.65, that the most recent post with “mais lidos” in its title would be the top result (these two: https://d.pr/i/1gV4SE). You could say there might be some differences in the posts’ contents that would affect this, but they’re all mostly the same. I look at the results and they don’t make sense to me, the way they’re ordered.

And this “mais lidos” is just one example of a few others I’ve came across.

Hi, @rfischmann

Okay, for the “mais lidos” query and results I can say next. First article does not have a phrase “mail lidos” in the title, but anyway it got the biggest relevance score. I think because there are some “mais lidos” phrases in the post content.

What we can do? We can lower the relevance coefficient of the post content, so existence of the phrase in the title will be MORE significant than existence of the phrase in the content.

This is what “cluster weights” for.

Currently you can see the “post_title” weight is 0.8 while “post_content” is 0.5. These are default values.

I would propose you to set “post_title” to 0.9 and “post_content” to 0.4 and see what happen. I think it should work better.

You can even set “post_content” to 0, in this case WPFTS will not try to search phrase in the post content at all. But for my humble opinion it’s not friendly for users (I bet they never use strict phrases for search that you planned/imagined).

Please try and let me know how it works in your website.

Thank you!

By the way I see no except text under posts in search results which can be a bit unclear for users. Most people use Google/Bing/etc and are used to seeing the context in which the requested phrase appears. You could improve the user experience by adding Smart Excerpts. Just a recommendation.

Yeah, I’d never set “post_content” to 0, sounds like a real bad idea.

Making the change you’ve suggested got rid of that weird post on the top, but there are still lots of posts showing up before the latest two.

Search from WordPress: https://d.pr/i/tqFPhK
Search from Sandbox Area: https://d.pr/i/3UI6QR

However, I do understand that, from this point on, it’s up to the plugin’s algorithm to determine each post relevance score based on its settings. I’m just saying that, before v1.65, the two latest posts on that same query would be the top two results.

Oh, I’ll look into your Smart Excerpts suggestion! Thanks! ??

Hum, just ran a quick test with it and there is some HTML code showing up, don’t know why: https://d.pr/i/5EjLah

It looks like get_the_excerpt() call is screened by htmlspecialchars() or something that converts plaintext to HTML. You need to find it and remove, so the_excerpt() will be shown in the page directly.

In your theme (caards) there is a file template-parts/archive/content.php

https://e-wm.org/i/E20240526-040307-001.png

In selected code you can find the code

<?php echo esc_html( csco_get_post_excerpt() ); ?>

I would propose to replace it by simple

<?php echo the_excerpt(); ?>

That should work.

Normally you can copy this file to the child theme and modify it there, so it will not be replaced on next theme update.