Hide Block from inserter sidebar, show in toolbar

I have a custom Card block, We also have another block that is meant to contain many Card blocks. The Card Group block has an <InnerBlock> component that only allows Cards. It’s great! The user clicks a little plus sign and a card shows up. Great.

Now, I would like for the user to only ever be able to add Cards to this Card Group block. Specifically, I don’t want them to be able to even SEE that there is a Card block in the sidebar. I only want them to click that plus sign to add a Card to that Card Group and never ever anywhere else should they see a Card block.

Is this possible? It feels like the ‘inserter’ support is all or nothing.

Would it be easier to set the inserter support to false and create some kind of Inspector Control for the Card Group to increase/decrease the number of cards? That seems like, perhaps, overkill.