Looks like in most cases you have only one admin-ajax.php request per page.
So I believe the influence on loading time is related to the fact that admin-ajax.php is called at all, not that PVC is doing that.
If there was a different plugint doing Ajax on page load, you would have the same result probably.
The time required to run admin-ajax.php depends on number of things like your server enviroment, number and quality of plugins used, current theme, etc.
I may tell you we have a plan to provide “slim” ajax mode for PVC, not relying on admin-ajax, that will load only minimal resources (and make the request much faster). However this would be an experimental ferature – according to any WP guidlines Ajax calls should run just as it is now in PVC.