I ran into similar symptoms today. I updated Jetpack yesterday. I turned on the new Protect. I checked it this morning and it reported it had blocked 60+ malicious login attempts.
Today I updated the “CAPTCHA” plug-in, Version 4.1.0 By BestWebSoft. After I updated CAPTCHA by BestWebSoft I immediately started to see the “Prove Your Humanity” math problem from Protect on the login screen in addition to the math problem from CAPTCHA by BestWebSoft. When I hit the login button my browser would throw out the 500 error. I could not access wp-admin.php.
I tried signing in from a different I.P. and was able to login. I disabled Protect and the issue disappeared. I can now login and hit wp-admin.php just fine.
