Notification Emails taking hours to go through

I have a website with a custom notification email setup, which is sometimes working with a long delay. Here is the setup.

1. This site is a private site with various user roles
2. When a post is created, the administrator can check which user roles who will get the notification (created with advanced custom fields)
3. When the post is published, the selected roles will receive an email.

Problem: It seems that when all the user roles are selected, the email will go out right away, however, when only a few user roles are selected, sometimes it takes hours for users to receive the email.

Here is my simplified code:

function notification_emails($roles, $post_id){

	//** there is a query here to see which category the
	if ( in_category(array('announcements'))) { 

	//** get checkbox selection fields
	 $chosenroles = get_field('user_selection');
		if($chosenroles){
			foreach($chosenroles as $value){

				//pass the values as roles
				$userargs = array(
					'role' => $value
					); 

				// Get user emails and set up user email loop
				$roles = array_merge( get_users($userargs) );
				foreach ( $roles as $user ) {

				//set up email
					$to = $user->user_email;
					$subject = 'Update';
					ob_start();

					$cat = get_the_category();
					$cat = $cat[0];
					?>

	        		<p>email copy here</p>

	        	<?php
	        	//send the email
					$message = ob_get_clean();
					ob_end_clean();	

					$headers[] = 'From: Namne<info@thewebsite.com>';

						if( wp_mail( $to, $subject, $message, $headers) ) {

						} else {
						};

			} //end for each user loop

			}//for each chosen roles
		}//end if chosen roles

	 } else{
	} //end if in cateogry
}//end staging fucntion
add_action('publish_post', 'notification_emails', 10, 2);

I thought this might be a hosting issue, but then my client mentioned the delay only when he selects less amount of users. Any ideas? I’m wondering if there is an issue with my parameters or priority levels?