Target the main comments query/loop below each post?

My own plugin retrieves a few comments at the beginning of the post through my own WP_Comment_Query. I save these IDs so I can then alter the WP_Comment_Query request and not fetch these IDs.

When I use the pre_get_comments hook to hide these already-fetched IDs, they are also hidden from my first query at the beginning of each post. It defies the point.

add_action( 'pre_get_comments', $plugin_public, 'hide_the_comments' );

function hide_the_comments( $comment_query ) {
    $comment_query->query_vars['comment__not_in'] = $the_ids_to_hide;
}

How can we target the bottom request only, just like there is is_main_query() for the post loop?