[Plugin: Woocommerce / wp super cache] 3,5sec slow loading time?

I have noticed that my webshop running on woocommerce is quit slow.

when i first installed supercache it was 7 seconds to create a dynamic page. I removed some feeds with random products and some other widget shit.

It reduced the loading time to 3.1-3.9 seconds, much better but good enough?

I heard that google realy dont like sites with slow access time and can bann them, is that something that i have to be afraid of?

Now:
<!– Dynamic page generated in 3.446 seconds. –>
<!– Cached page generated by WP-Super-Cache on 2012-06-17 09:03:20 –>

<!– Compression = gzip –>

Before:
<!– Dynamic page generated in 7.071 seconds. –>
<!– Cached page generated by WP-Super-Cache on 2012-06-17 08:33:20 –>

<!– Compression = gzip –>