Remove submenu from custom post type

I created a custom post type named portfolio with tag taxonomy support.

Since WP does not make a difference between post tags and custom post type tags, I created a menu item Taxonomy under which I want to put categories and post tags. I managed to create the the menu and submenus, and also remove category and post tags from the Post menu, but i didn’t manage to remove Post Tags from the custom post type menu.

I tried:

remove_submenu_page( 'edit.php?post_type=portfolio', 'edit-tags.php?taxonomy=post_tag&post_type=portfolio' );