Ok. My understanding is as follows. There are 2 (principally) ways of upgrading WordPress or a plugin.
1. Uses Curl to get the zip file by http. Streams it directly to the file system. This requires that the web server user has write permissions pretty much everywhere. The second of the two links I posted above applies. This method is used by the “one-click” updates feature.
2. Uses FTP. I believe this is the older way. This obtains the zip file by anonymous FTP or HTTP (not sure / doesn’t matter) and then uses the PHP FTP client class to put the files into place. The first of the two links I posted above applies.
The benefit of 2. is that it means, for example wp-content/ can be owned by a system FTP user with 755 permissions. (Only writeable by the system user). Thus it is not writable by the web server. This is more secure if we accept the principal that files/directories should have the minimum permissions necessary for the system to achieve its purpose. In case 1 as per the Codex notes all files have to be writeable by the web server. This is not just (imho) undesirable on shared hosting setups but on any setup. If only the /uploads directory is writeable by the web server it means less places to look for all that malware which some plugins sometimes let in by mistake.
In my case I can’t get the FTP method to work. The plugin upgrade process tries to use it because I have set my permissions for case 2 (everything is owned by an FTP user with 755 except the uploads folder which is owned by by web user). (It also tries to use it when I direct it to using FS_METHOD in the wp-config.php file). But in /wp-includes/class-http.php it calls into /includes/functions.php wp_is_writable() to see if it can create a directory (or file) /wp-content/mypluginname.tmp . This fails because the web server does not have write permissions on /wp-content.
Either a) my expectations for the FTP method are incorrect. It is not intended to work as I expect or b) this method is broken in in WordPress 3.9.1 or c) my plugin should not be trying to create a file/directory /wp-content/mypluginname.tmp – it should perhaps be trying to do this in /wp-content/upgrade/mypluginname.tmp
It would be fantastic if someone who knows how the FTP plugin upgrade method is supposed to work could tell me which is the case.