It was weird. I added the plugin to 2 other wordpress sites and their time is correct. I checked the calendar settings, the wordpress time setting, the calendar settings, etc.
I had to just stop using it because I couldn’t figure out how to fix it. I’ve embedded the gmail calendar directly into a wordpress page and it works and looks just fine. you might try that as well. You can add it as an agenda format (showing only the scheduled events), a full calendar page, etc.