Import a large amount of objects into an index

Hi,

WPSOLR should retrieve your 20.000 documents from Sql, then send them to Solr by chunks of 100.

Each chunk is sent to Solr with a hard commit, so all documents indexed should remain in the index.

Do you confirm that your index contains data during the process, and then goes back to 0 ?

Hi, actually I also have same issue. to make possible adding great amount of posts into solr index I set
max_execution_time = 3600 in php.ini,
added define(‘WP_MEMORY_LIMIT’, ‘1024MB’); in wp-config.php.
Also I had to increase CURL_TIMEOUT in
wpsolr-search-engine/vendor/solarium/solarium/library/Solarium/Core/Client/Adapter/Curl.php function createHandle line 149.

I am not a wordpress professional, is there way to overcome php memory limit while adding multiple posts in solr index? Moreover there is no posts progress displayed.

Hi,

How many documents do you try indexing ?

Hi, I need to index over 80,000 of posts. And I can put only 8,000 of posts inside solr index.
Thank you, for quick reply.

The way the indexing is done right now is pretty (too) simple: fetch all posts, and send them to Solr by 100.

This is a problem for 80.000 posts.
And moreover, the indexing is not incremental.

I need to rewrite it:
– to mark indexed and unaltered posts as already sent, for incremental sake
– use sql cursors, to load posts from database by chunks, to prevent memory and CPU freeze

The new version of WPSOLR (1.6) should be able to index all your posts.

Let me know if you have any troubles.

Hi, thank you for such quick fix. I will test this functionality soon and will put feedback.

Good luck!

Hi Thanks for your quick fix

in my case the performance did improve now it’s indexing more posts but it’s like it always tries to index the same ones over and over it never goes over 7,900 posts of the 20,000 I need to index.

Ok I found how to fix this issue

I built a custom version of your plugin with the necessary mods essentially one of the problems is when you click the load data button you are executing the delete documents function so the index is getting deleted every single time you attempt to resume the load.

And also I had to add a way to store in the database the Last fetched post and the current document count so that it is accurate when you restart the load again otherwise it will always try to start over

You might want to consider including this changes on a new version.

Happy you could find a workaround.

The solution is to use a real cron job, which is not limited by timeouts when a large # of documents is to be indexed, and to implement incremental indexing to prevent endless re-indexing.

The lack of real cron in WP is the true issue, and asking users to set a cron is challenging (just have a look to other plugins’ support).

Perhaps an external task to trigger the indexing jobs would fit better.

Hi,

The indexing has been greatly improved in new release WPSOLR 1.7

– Better errors/information displayed
– Indexing restart from last indexed docs (incremental)
– Real-time indexing on post status change
– Attachments can now be indexed

Please let me know.

@bogdan1988, @ledavid71:
Did the new version solve your indexing issues ?