Losing href on switching

I have a custom link added to a button using the Text editor, when I switch back to Visual editor I lose the href, not the class or anything else, just the href.