Check is custom post type category is empty

Hi guys,
What I’m trying to do is quite simple but I just can’t work it out. I have the following code which Displays the categories of my custom post type ‘Food’:

<?php // Output all Taxonomies names with their respective items
$terms = get_terms('category');
foreach( $terms as $term ):
?>
    <h3><?php echo $term->name; // Print the term name ?></h3>
    <ul>
      <?php
          $posts = get_posts(array(
            'post_type' => 'food',
            'taxonomy' => $term->taxonomy,
            'term' => $term->slug,
            'nopaging' => true, // to show all posts in this taxonomy, could also use 'numberposts' => -1 instead
					));

          foreach($posts as $post): // begin cycle through posts of this taxonmy
            setup_postdata($post); //set up post data for use in the loop (enables the_title(), etc without specifying a post ID)
      ?>        		

          <li>
					<?php
					the_post_thumbnail();
					the_title();
					the_content()
					;?>
					</li>

        <?php endforeach; ?>
    </ul>
<?php endforeach; ?>

It successfully outputs each custom food post type under the category it is filed.
However, the “Uncategorized” category also shows. There are no custom post types that are uncategorised so I do not want it to show. How can I check if the category is empty to not show it?
Thank you,
Chris