Uploaded files are not protected

After creating a form that accepts file uploads, I found out that the files uploaded are not protected by the permissions mechanism of WordPress, and anyone with a link can access those files.
Is my setup wrong?
Is there a way to prevent unauthorized access to uploaded files?