Site still accessible via "Ugly" permalinks

I was wondering why, after changing permalinks to “pretty”, I can still navigate to https://www.mysite.com/?p=123 and get to a post. I would think that once you edit the permalinks, this would no longer be valid. Is there a way I can block someone from accessing the site this way?