I did notice wpa.css also includes #altEmail_container, which will be visible in forms without the css file, and while adding it to my own stylesheet does hide it for the main frontend, it doesn’t hide it in the wp login form.
So my initial idea of disabling wp_enqueue_style isn’t perfect, but the problem nonetheless needs a proper solution.