No, because if you’re not in the same group as the webserver, then you wouldn’t have access to them either. And if you are in the same group as the webserver, then likely so is everybody else on your shared server, meaning they still have access to your directories.
The short of it is that with a shared webserver and only basic unix type permissions handling, there’s absolutely no way to both allow uploads/modifications and still keep the system 100% secure. You need something more than the simple unix permissions system.
Some hosts allow the webserver instance to run the PHP programs as the user who owns them, thus getting the user’s own permissions. This is a reasonable compromise, as even if somebody gets hacked, only their site is compromised, not the whole shared server.
Regardless of all of the above, this is not a flaw in WordPress. It’s a standard webserver issue. Your host has to deal with it and tell you how they treat security in this sort of a case. Don’t mention WordPress, say that you want to allow the webserver to modify certain files, but without making all the other shared users able to modify those files as well, and without revoking your own access to those files. If they don’t have a solution for you, well, then you can either live with it or talk to another host instead.