Hide empty categories

My variations are shown in a dropdown on the product page.

If that helps

Increasing the threshold to the number of variations may give you the behaviour you want. Use the code in the 3rd post from the bottom:
https://www.ads-software.com/support/topic/how-to-have-variable-products-only-show-valid-combinations/#post-8804489

Where would I need to put this piece of code?
And will the code remove a certain variable from 1 dropdown? or can it only remove the whole dropdown?

The code goes in functions.php. To be update safe, your functions.php should be in a child theme:
https://codex.www.ads-software.com/Child_Themes

The first line of functions.php should be <?php

Increasing the threshold to above the number of variations will prevent users from being able to choose unavailable variations. Best to just try it.