Exclude fields from frontend editor?

Is it possible to exclude the fields from the frontend if I have a custom template that uses do_action('edit_user_profile',$current_user);?

Right now if a user makes an edit like changing their custom avatar the group they belong to gets unchecked. It shows up on the frontend as unchecked event if it’s checked on the backend.

I’d like it to remain checked and for them not to have access to it.