OK.
The original WP gallery is a shortcode. Without any parameters, it will show all the images attached to the post it is in. When it is inserted using the Media modal dialogs, it puts the attachment IDs into the gallery shortcode so that you can show images that are not attached to the post it is in.
If you remove images from a gallery, they remain in the Media Library, and in whatever other posts they were used in.
To delete an image from your site, you have to go to the Media Library and delete it. But it can still be referenced in a post (so it would be a broken image).
To delete an image from a post, you have to go to that post to delete it. But it is still in the Media Library.