WordPress limits on search? I have over 280k posts…

  • I’m using a dedicated box…
    Intel Core 2 Duo E7500 2.93GHz
    2048 MB Memory
    CentOS 5 (64bit)

    This is an advanced question for WP pros. I have a test installation that I was kicking around. I’m using an RSS feed to populate custom post types(jobs) and when using the search, and when including in that search a custom field or 2, the search becomes all but dead.

    Am I pushing the limits of WP too hard?
    Am I pushing the limits of my Dedi box too hard?
    A little of both?

    Do I need to use the search filters to fully customize my search, bypassing the core, using indexing and or creating separate tables for a more custom search?

    It seems that with the structure of the postmeta, I will always need an inner join (if I need 2 or more results(custom fields) in a single post from this table) and this is a resource buster for sure.

    At 50k-80k posts I would get results back in 1-3.5 seconds with no other server load.

    At 250k-280k posts my the script times-out. I make time-out adjustments and the search has given results after 20 seconds or more with no other server load.

    While using the search for posts in the back-end(admin section) and no other filters to the search query(no extra postmeta query inclusions on my part), I get results in about 5-10 seconds with 280k posts in the system.

    If I actually had traffic, this would all be bad. right?

    Has anybody come across search query fixes for large amounts of data in WP?

    Thanks in advance for any input at all.

Viewing 10 replies - 1 through 10 (of 10 total)
  • Thread Starter Brayne

    (@brayne)

    So I see that the forums here are showing well over 1 million posts. I’m assuming that the forums are using wordpress code but I see that the search function being used is from google. Is this the recommended search utility for wordpress?

    I use Google Search on my website, I think it’s better than the default search and, if you want, you can put AdSense in it.

    Thread Starter Brayne

    (@brayne)

    Thanks for the reply @TheTux. Did you do an xml site map and have google index your site? I’m thinking that’s really the only way they can know what’s going on with the content… curious.

    brightonmike

    (@brightonmike)

    What about server-side caching of results? For example if you had the results for the term “hello post” cached, it would serve up the cached page instead of performing an actual search.

    If you’re worried about the user not finding fresh content, you can have the cached pages refresh – just not every single time somebody searches.

    prionkor

    (@prionkor)

    I also believe you should use google search for that. if you have that much of a post 280k. The new pages/posts of your site should be indexed quickly and should not have any problem while people search your site via google. This will also help you reduce your server burden.

    Thanks!

    You could also look at something like:

    https://www.ads-software.com/extend/plugins/solr-for-wordpress/

    If you don’t want to try setting up and hosting your own Solr server, there is apparently a hosted service, and an adjunct plugin to use it:

    https://www.ads-software.com/extend/plugins/advanced-search-by-my-solr-server/

    No. you need to change to another box right now, quad xeon and raid hard discs, and choose a good bandwich for this server. Then optimize you site again and again. Remember u have 280k post in this moment.

    Is better u have the control of your searches.

    Your biggest bottleneck is going to be MySQL. I highly recommend focusing your energy on tuning MySQL.

    Please also remember that having a search tool to muck around in ~280k posts seems like a potential DDoS nightmare. Definitely want to find out how to cache your results and search efficiently, and limit how many searches per minute a single IP can make.

    This is an interesting thread. I don’t know why 280k posts should cause a problem. After all, that is nothing for forum software.

    1) Is WordPress sufficiently efficient just to run a site with 100,000 posts and comments? How is the response time with regard to basic process of pulling up a post and clicking on a category? What happens if you enter a category that may have 8,000 posts – is WP quick to respond and provide the latest 10 or 20 articles?

    2) How efficient is search – or rather how inefficient might it be? Is there a better alternative so that you can search specific content that Google wouldn’t (e.g. you have a custom field as per the below example.)

    Let’s say you want to run a site like stack overflow or stack exchange. In some instances you might be porting over content from another database. I can see easily having 100,000 Posts with comments. I’ve got a server that runs forum software super quickly but the question is whether forum software would handle the same thing with much greater efficiency. I wonder if there is a number of posts/comments which would cause a problem.

    For me, I’m mostly concerned about performance of simple page views and navigation, e.g. a user clicks on a category that may have 8,000 posts in it and I need to pull up the first 10 or perform a search that will allow the user to search generally and limit by a custom field (or taxonomy such as “continent” if I had a custom field that would have an entry for each continent each post is coming from.)

Viewing 10 replies - 1 through 10 (of 10 total)
  • The topic ‘WordPress limits on search? I have over 280k posts…’ is closed to new replies.