Small site on a dedicated server but slow as hell!

I’ve got several wordpress based sites on a quite serous dedicated server (hardware wise, quad core, tons of ram and so on).
Each of them is just few posts at the moment and still they are extremely slow at times!

The homepage is always fast.
The post pages are sometimes slow, sometimes ok.
The attachment pages are slow always, and not because they have some super big pictures. Even with images 50-60kb it takes 5-10 seconds, before it starts to show any of the page.

I had my admins check everything on the server, and there doesn’t seem to be bottle neck anywhere, everything is smooth on the back end and my server load isn’t significant.

Right now they are offering me to install a cache plug-in, but my impression is that only makes sense on larger WP sites?

Any clues what it might be and what to do will be highly appreciated!!!