Large Import Stalls

Hi,

Sorry for missing your question.

The problem may be caused by your site capacity. Please check the log, https://prnt.sc/10kde0t to see if there are any errors then decrease Products per ajax request.

Best regards,

I am getting the same issue as the OP.

I am using the 1.0.9.6 version of the plugin.

I only have 25 orders at a time running, and its taking way longer than it should. 2CPUs and 4GB of ram should be enough to at least tackle these in around the 2min mark with no visitors on the site yet.

The backend site comes to a crawl while running this thing too. Frontend is fine because we are using a caching solution to handle that with NGINX.

Here is a short snippet of the s2w log file:

[2021-04-22 17:04:37] Order exists, Shopify Order ID: 3758440710339, WP Post ID: 84409
[2021-04-22 17:10:36] Order exists, Shopify Order ID: 3758446280899, WP Post ID: 84410

Take a look at the timestamps. 6mins just to determine that an order already exists? That seems like maybe there is some inefficiencies in reading the database records in a growing database.

It wasn’t always this slow but it definitely progressively got slower and slower. New orders are happening far faster on Shopify than anything is importing (especially now that it is pretty much not importing at this rate). We can’t even catch up.

Has this plugin been tested on a database with over 50k orders? I feel like 2CPU cores and 4GB of ram should be sufficient no?

One more thing, the Database is AWS RDS so its on a different server. So those resources listed are strictly for PHP, NGINX and Docker

• This reply was modified 3 years, 7 months ago by joemoe1984. Reason: Mentioned separate Database instance

Hello @joemoe1984,

Orders can only be imported with the pro version and for pro users, you have to create your tickets at https://villatheme.com/supports/forum/plugins/import-shopify-to-woocommerce/ to receive support

Take a look at the timestamps. 6mins just to determine that an order already exists? That seems like maybe there is some inefficiencies in reading the database records in a growing database.

No, it’s not like that. Unlike products, for orders importing, log file only logs failed orders while successful orders are not being logged. This means that the “6mins just to determine that an order already exists” is totally wrong. A lot of successful orders were imported among that 6mins.

Best regards

No, it’s not like that. Unlike products, for orders importing, log file only logs failed orders while successful orders are not being logged. This means that the “6mins just to determine that an order already exists” is totally wrong. A lot of successful orders were imported among that 6mins.

I think you are talking about the error logs but I am talking about your log.txt file in the cache folder. I did a tail -f log.txt within the cache folder as I was importing orders. I was watching in real time those messages logging to that file.

The requests, for 25 orders at a time, are taking close to an hour or longer. So no there are no orders importing during that time. otherwise they would have finished by then and started the next round of 25 imports. Also, the Post IDs are right next to eachother meaning that they were created right after eachother. So its next in line with whats being imported.

The progress bar stays gray until one round of imports is completed, then it turns red showing completed/total. Left it running last night but it stopped at 75/3000 (or whatever the total was) before I was logged out.

I had the Network Tab open the whole time tracking the requests. 59 mins was the last successful request from last night.

I also managed to go from a really long time to import to a more manageable time to import 25 orders yesterday at one point. Not sure if it was the settings I was updating or what, but a request for 25 orders at a time took somewhere between 2-3mins. So there is definitely something going on where its struggling. The server is capable of 2-3 minutes per request which I can live with but not a hour.

(I don’t want to hog this support item so I will look into creating a support ticket for the PRO version. Its possible that whatever issues the OP is having for products might be similar to orders)

Well I think the issue is with some orders, especially refunded orders.
I suggest you update the latest version(1.1.1.1 – 2021.02.24) and try again.
A lost of our customers have imported more than 50k orders without issues.
If the issue still occurs with the latest version, please create your tickets on the forum for pro users because we are not allowed to support you here.
Best regards

I want to post here what the issue was.

It was the query for the shopify id in the postmeta table. When that table grows, the query gets slower and slower. This is why the existing orders were taking a while.

Searching the meta_key and meta_value fields is slow because those aren’t KEY fields in that table.

I would recommend creating a new table that stores the post_id (order id) and the shopify_id and make the shopify id a KEY field so that you can index it quicker to check if something is existing or not.

This is what I did and I dropped the query down from 133 seconds to 0.006 seconds.

People with small stores won’t notice these issues but larger stores will.

@amaule07 If you have a developer that can implement that or check to see if thats whats happening for your issue, I would recommend it.

Cheers.

Thank you for your suggestion, we will try your solution and improve our plugin in future updates.
Best regards