Hi byoussin,
I am a newbie who started using Jetpack and some of its fine collection of features. I too felt that there was some depreciation in speed associated with certain features within the pack.
I suppose it has to be generally accepted that if a feature requires access to additional resource, there will be some level of impact (putting a wing mirror on a car for example).
This for me is balanced on cost against benefit.
Anyway, I now use “Simple Custom CSS V3.2” by John Regan & Danny Van Kooten. It does the job quite adequately, and dsiplays no discernable speed deficit.