What encode WordPress uses to read plugin and themes metadata

All files should be UTF-8 without the BOM. Is that the exact message you get every time, or you only get something similar that varies in specifics? In any case, it’s rather unusual for there to be any multi-byte characters in PHP files. In general they look exactly like ASCII encoding, one byte per character. There aren’t any start bytes, much less invalid ones. The only time you should see multi-byte encoding is if some HTML portion or static string includes foreign characters.

If you opened a .php file with a hexadecimal editor you’ll see for yourself what I mean. Of course, this only applies to original source files. The zipped files are another story entirely. They’re simply binary data. I suspect the unzip process might be corrupting the file in some minor way, or the UTF-8 decoder itself is flawed. Try not decoding UTF-8, pretend it’s an ASCII file. For the header portion you’re after it probably is equivalent.