Crawler not following all site maps

My site is multilingual using WPML and therefore has multiple sitemaps. Since I can only specify one sitemap for the crawler I created a custom sitemap index:

<sitemapindex xmlns="https://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://ways.se/sv/wp-sitemap.xml</loc>
</sitemap>
<sitemap>
<loc>https://ways.se/en/wp-sitemap.xml</loc>
</sitemap>
</sitemapindex>

However when running the crawler it or refreshing the map it only seem to follow the first sitemap in the index.

Is there another way to crawl a multilingual site / site with multiple sitemaps?

Side note: WPML thinks this splitting of sitemaps is a non issue since search engines will follow the langhref attributes on pages in the default language’s sitemap anyway. LiteSpeed Crawler does not seem to do this however.