Authors list > author

I’m having some difficulties to make a correct breadcrumb.
I have a custom page template Authors which just list all authors (users) on my WordPress installation.

Then when someone clicks on a single author he is taken to the profile of one user (which is using the default WordPress template author.php)

But when I am on a single user (author.php) the breadcrumb only shows: Home > User and not, as expected Home > Users > User

How would I go about to make this?