At a very minimum, GCL should provide an option in the IMAGE block to disable the GCL setting PER BLOCK for instances where the image is NOT part of a gallery but used instead for other purposes (such as sampling content).
I understand, of course, but many issues. First would be to override all the blocks using image to add such an option: super hard (almost impossible, support would be close to infinite). Also, people would need to use blocks, and therefore, Gutenberg. Second is that most users aren’t on Gutenberg, they use page builders instead.
So the only way would be to provide another field in the Media entry in which you can enter the IDs of the Post/Pages to ignore for the Custom Link. That would be the cleanest way. What do you think?