When protecting uploaded files, the php script reads the file on request and serves it if the user is either logged in or exempt.
The problem could be that memory_limit needs to be increased for PHP. Or that the wp_check_filetype function doesn’t find the mime type for the file.
It could also be that ob_clean() or flush() is outputting some error, that corrupts the file. Try adding error_reporting(0); before that.