Lazy load makes masonry grid have large gaps

I’ve enabled the lazy load and really like it on the posts. But on the category pages there are huge spaces in the masonry grid.

Is there a way to have lazy load active but also not have huge gaps?

thanks