Clearing out useless users from wp_users and wp_usermeta

I have over 8,500 users in my wp_users table and 139,000 records in my wp_usermeta table. My hosting service has closed down access to my site because apparently these tables are causing high resource usage on the server.

After a quick look over the wp_users table, it appears that most/all of the records in that table are junk (apart from my own user which is first in the table). As I am the only person to ever author posts, I was thinking it might be safe to delete all the user records other than my own. My one concern is that these other users might be linked to the people who have commented, but from what I can see, there is no link between wp_users and wp_comments (wp_comments does have a ‘user_id’ field, but this is 0 for every comments, so it clearly isn’t linking back to the wp_users table).

Is there anything I need to be wary of before doing this? Could there be legitimate records in the wp_users table that I haven’t thought of?

Thanks!