Yoast slow down my website

I tested page speed with google pagespeed insight and got a result of 69 on mobile and 75 on desktop.
After I installed yoast and got a score, for the same page, of 55 for mobile and 63 on desktop.

Is there a way to fix this problem or to reduce the gap due to yoast plugin?

Thanks