Security: why do file permissions matter?

I understand that setting your file permissions on files like config.php open you up to security vulnerabilities but I don’t understand why.

Let’s say for example if I set config.php to 644. Supposedly that allows anyone to read it (not change or execute it).

But how does that help a hacker if all PHP is handled from the server side? It’s not like he can open up config.php in the browser and read the contents … correct?

How could a hacker gain access, in theory, from improper file permissions?