Link to move up one level or back to its category page?

I have an image gallery where each full size image is its own single post. I need the image to be clickable, which would return the user back to the category listing from which it came.

It has to return to the exact page the entry is linked from too, since the category archive is multiple pages deep. The image can’t just link to the first category page.

I was hoping to just do a javascript goBack or whatever that’s called, but there are also next/previous links in the single post so they may have gotten to that post from the previous entry and not from the category page.