Prevent user fields from showing in the network admin user section

Hi,

I added a few custom fields to users add/edit forms in wordpress sites that are part of a network (WPMU). The problem I have is that the user custom fields of the first site are also showing in the network admin user section. I tried not showing for SuperAdmin but it then also hides it for SuperAdmin when in the site user section. Is there a way in functions.php I could prevent any fieldgroups from showing in the network admin section or anywhere in /wp-admin/network/?

Thank you