UTF-8 with ANSI issue

Thanks so much for the detailed explanation. Dealing with character encoding was definitely an issue both before and after the rewrite. Previously Attachments 1.x base64 encoded the storage which got around issues like these, but also prevented things that should not be prevented (like searching metadata) so I’m actively trying to beef up how those strings are handled.

If possible, would you mind posting a sample of the problematic string? That way I can do some more testing and better handle future users who run into the same issue. Thanks!

Hello again,

the filenames had UTF-8 encoding but were interpreted as ANSI. So now, they have to be read as ANSI and interpreted as UTF-8 character mapping to reverse the effect. For German language there 7 characters where the effect will be visible:

UTF-8 interpreted as ANSI	UTF-8
??				?
?¤				?
??				ü
??				ü
?–				?
??				?
??				?

I’ve never done a conversion between this charcter maps by myself, but it seems that php already provides possibilities to detect the encoding:
https://www.php.net/manual/en/function.mb-detect-encoding.php

I guess the special thing is, that when attachments 1.6 runs in an php environment with ISO-8859-1 encoding, the effect probably does not occur and nothing needs to be converted (just guessing it, not tested it). So it will be mandatory to detect which encoding has been used (mistakenly) for filenames.

And remember, it is all about the filename of the attachment, which carries the error. The json strings are migrated correctly. After the migration process, attachments v.3.3.2 tries to address a file with the exact filename (e.g. ./übersicht.pdf), but the file is only available with the malformed name (./??bersicht.pdf). As result server will return 404.

Thank you for the detailed follow-up, it’s really helpful. To shed some more light: the only thing Attachments does is store the ID of the attachment, so it’s not actually storing the filename, so it must be with the data handling on some other layer. I’ll test with some characters as you’ve provided and hopefully get a fix up shortly. In the meantime if you come up with a patch I’d be more than happy to review and integrate a pull request!

Hey Jonathan,

I kept on testing, you’re totally right. I decoded your base64 strings and got what you mean. The problem is not related to attachments plugin, you are done!

It’s is equal whether a windows or a linux machine runs apache/wordpress, the problem comes down to wordpress inability to handle special (non-ascii) letters at all. So the error occurs even if I upload the file via the media library, and nothing has changed with from wp3.4 to wp3.5 – I think I had just not realized it before.

Thanks for support, solution for me is prevent these characters on files in general.

Thanks a ton for following up, much appreciated!