I added some logging to WP and that showed that the delays were definitely occurring inside of the WP loop, but at random places. The length of the delays also varied greatly, from a couple of seconds to over a minute. What was interesting was that the delays were much shorter during GMT morning hours and got worse in the afternoon.
In the end I had WiredTree move my VPS to another machine and that solved the issues. Apparently some other VPS user on my old machine was doing some heavy lifting or WT is overselling their machines
