Ok, digging into this, I found out why my server was getting random TIMEOUT when talking to the WordPress API, or any API for that matter.
My DNS server, for reasons I do not know, decided to stop answering DNS lookups, and so every time a request from php was occurring, it went through a timeout trying to lookup the target host, before falling back to the secondary. (Default is 3 seconds in WordPress)
This aded enough for to make MOST requests time out.
Be warned. Check your DNS lookups if you have random timeouts hitting APIs. It may also indicate a longer timeout, over 3 seconds should be in use.