I managed to find the issue, it seems related to the way the images are being called into the template. Creating a custom template without images te 175 records load normal. When I add the image into the template using $record->print_value() it goes back to being very very slow.
However, if I pull the image path out of the object and just embed an image like normal, the page once again loads fine.
<img src=”<?= ($record->values[‘photo’]); ?>” />
Not sure it’s ideal to go about it this way, but… It works!