Sadly, there is no solid way for the plugin to ‘detect’ if the script is going to be needed before content is rendered by WordPress. If only all images where embedded with a shortcode like galleries then there would be a trigger to hook on to… The alternative is filtering each post content on every request with a complicated regex search to find any possible media links that might need script action. This method is more ‘expensive’ (in terms of server load and response times) than simply including the script on all pages.
Just disable all FancyBox options that you do NOT need to make sure you have the leanest script version possible. Then, if you want to go further, I can recommend installing W3TC for caching and activate script minification + concatination. But test your site as an anonymous visitor to make sure no script action is broken after activating these advanced features!
