How to make your blog faster on gallery pages?

I have a pretty decent speed blog. It loads all pages at about 10 seconds. I am using total cache and widget cache plugins.

I have a problem with my photo category page.
This is how I have things set up is:

I have a “category page”(photos category) that have all the photo gallery posts listed in boxes. I keep only 25 photos per post <- thats clear enough right?

My category page loads waaay too slow. What can I do to fix things up?
I am not using any gallery plugins.
Thank you.