Yoast Slowing down my site

I just recently installed Yoast premium for my wordpress site biglean.com. After a few days of working with it and making some major upgrades to my site i started noticing drastic slow down to my site. sometimes taking upwards of 45 seconds to load according to webpagetest.org.

I deactivated yoast and then my site started running extremely quickly again. Is there a fix for this? Something i can change in my Yoast settings or my wordpress that will allow this plugin to run and not slow down my site?