display category years in category?

I have a category called letters. Ideally, I would like a list of years to come up in the sidebar when you browse the Letters archive for years – not of all posts, but just years of posts in the category ‘Letters’. Would this be possible? How would i go about it?