Can you be specific about what lock out conditions you want? Your last two objections seem to contradict each other. Custom coding something isn’t a huge endeavor if you have moderate PHP skills. There’s only a few data points that you can reliably use. The client’s IP address, which may be a proxy or load balancer. That’s OK, any IP behind that will not be reliable. The login name or email, and the current time and number of tries. Anything else can be spoofed. Your lock out scheme needs to work with only that data in whatever combination works for you.
The IP is important for distinguishing between brute force attempts against an administrator account and the true administrator. You ideally want to totally block one but never the other. You would of course block the attacker’s IP, but you might consider blocking the entire IP/24 range. Ideally, you would whitelist all of the administrator’s possible IP ranges, but determining what those may be is beyond the skills of average site owners. Attackers will be using multiple IPs, giving them many many tries if it’s important to not block legitimate administrator logins without any whitelisting. You could at least whitelist the IP/24 range the administrator is currently using when the plugin is activated, giving them at least one way in if their username is otherwise blocked by brute force attacks.
Whitelisting an entire range is usually necessary since most people’s Internet connection IPs are dynamically allocated. This introduces the possibility an attacker could be in the same range as the legitimate administrator. This is pretty unlikely, and they will still be limited by the number of tries before their specific IP is blocked. Sure, they can reconnect to get another IP, but that takes time, limiting the overall attempts allowed per time unit.
When you roll your own code, there are all sorts of specific strategies you can employ even within the limited data points. The lockout data can be stored as transients. When the transient expires, the lockout expires, so the normal login sequence would simply check for any matching transients. If none, the login may proceed.
