Deny Access to Media

I see we can grant specific user roles to see a specific page/post/cpt. But this functionality is not on Media. How can we show uploaded files to only certain user groups. In the past we have used this plugin in combination with Groups and Groups File Access but this causes complications because a user needs both a Role and a Group.