WordPress as podcast feeder

Hi,
Im wondering if its possible to use WordPress just as RSS feed provider and host for podcast files for iTunes. I just want to use the backend for the management.

Im using Seriously Simple Podcasting plugin. It works like charm, but I dont want to use the actual wordpress site.

So is there a way to deny the access for all users except for the mp3-files and the RSS feed?