WP Gallery Custom Links uses regular expressions to swap in the custom links, and regular expressions tend to be a little on the expensive side. Unfortunately, I’m not sure there’s a reliable way to keep all the same functionality without regular expressions – to my knowledge, WordPress doesn’t provide gallery hooks more specific than “large single chunk of HTML for the whole gallery” (ie no hooks for individual images). If you or any other developer you know would like to review the code for suggestions on how to improve performance, I’m definitely open to ideas, as I know large galleries can cause issues. ??
Off the top of my head, the plugin loops over all possible image sizes looking for those urls to replace, and it may be possible to limit that to the setting on the gallery to remove some of those iterations, which I’ve been meaning to look into for a while and haven’t gotten around to yet. I’ll see if maybe I can make that modification in the next couple of days, and hopefully that may help you a bit, but the regexes in general are still probably the biggest issue.