Media Library for Group of Users

Hello, I’m building a site that various departments want to be able to update their pages and upload files to. The departments have several staff that have accounts.

Using user role editor, I have set up permissions that restrict access to pages based on their department. This works perfectly.

However, I’m trying to do something similar with the media library and can’t figure out. Ideally when someone from department A logs in they see the pages they have access to and media library items they have uploaded. But also, media library items that other members of department A have uploaded. They shouldn’t be able to see files that Department B have uploaded.

I’m trying to avoid using multisite as it creates many headaches. But I’m struggling with the media library in this instance.