Thanks for the feedback!
Depending on the modules you’ve activated, and the new elements you added to your site thanks to Jetpack and, your theme, and other plugins, that could indeed be expected. For example, if you’ve added Like buttons, Related Posts, a few widgets, and sharing buttons, that will indeed increase your site’s loading time.
Like with every other plugin, when enabling features, you decide to decide if that feature is worth the performance hit it could bring to your site. In general, any features that use images, or call resources from third-party services (like the sharing buttons or the Facebook and Twitter widgets) will have an effect on your site performance.
If you’re willing to run some tests to find out more about this, it would be interesting to go to Jetpack > Settings in your dashboard, and deactivate each module, one at a time, until you can figure out what module seems to cause the issue on your site.
Let me know what you find!