Hi,
You can change the thumbnail size by changing values of Generated Thumbnail Dimensions and Frontend Thumbnail Dimensions options.
Generated Thumbnail Dimensions (Photo Gallery Options > Global Options > Thumbnail Options) defines the physical size of thumbnail images, while Frontend Thumbnail Dimensions (Photo Gallery Options > Shortcode Defaults > Thumbnail Options) is its display size. So it is important to keep frontend dimensions smaller than generated dimensions.
After making these changes you need to Recreate thumbnails. This is done to have a nice resolution for thumbs.
Then edit your gallery page and make sure Image thumbnail dimensions is set to the desired size here as well.
Thank you!