Background color on link but not img links

To better separate links from text I have a background color on hover, but I don’t like it showing up on images. By setting images to display block I’ll solve the issue for the most part by having the image displayed over the whole background area, but in galleries that doesn’t work.

Is there a way to give a:hover a background color, while at the same time making sure it’s transparent for a:hover img that I’m not thinking about? Pure CSS would be best.