Hi, as the sitemaps are generated on the fly, a huge site will indeed make it too slow. This is a problem for search engines so you need to address this.
There is an option in the Sitemap settings called “Split by”. Set this to “Month” if you have not done that already.
This should cut your sitemaps up in smaller ones, at least if your posts are not posted all in the same month…
If that does not help, you can try adding an object caching plugin to your site. If your host allows that, it should speed up things for your complete site as well as the sitemap plugin.