Disable Asset Manager for Security

Is there a way to fully disable the Asset Manager for security purposed? File managers such as this are prone to pose numerous security risks and we would like to eliminate these risks by disabling altogether.

Right now the only way I see to disable it is to change the setting of which roles can view it within the admin, but that is not very secure as one compromised admin account could result in malicious files being uploaded or modified.

Is there a way to disable this feature completely programmatically such as with a filter or a constant that could override the admin setting?