Could this be because I have many attribute variation in each product?
I doubt it, but you could test by temporarily hiding some products, and just displaying a few.
Site sluggishness can also come from plugins, so test by deactivating some that you might not need, and see if it helps.
You can also use the Network Monitor from Firefox https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor to analyze what happens when you visit your site.
