The issue is a block I created in my custom theme. So I have to keep my theme active to test. I could still mess with the plugins I guess. Although, it seems like Gutenberg in the latest WordPress is the only player here. I downgraded my WordPress version to 5.3.2 and it works fine! So it has something to do with the 5.4 upgrade.