Hey there,
Is this showing up for yourself or for customers who aren’t logged into your site?
The Visibility setting for products works the same was as it does for posts and pages, so a private product will not be visible to anyone but those with the appropriate permission levels (Editor or Administrator).
The Catalog visibility option might work better for you. With that, you can choose to completely hide the product, or just hide from search results or the shop (Screenshot).
If those are set properly and customers are seeing products they shouldn’t be able to see, you may want to check if there is a caching issue, either in the browser, on your server/host, or via a caching plugin.
Hopefully this helps you get set up!