The padding setting was added because people asked for it since PhotoSwipe itself has this option. See https://photoswipe.com/options/#padding.
Padding controls how much space is added around the image as minimum if the image is large enough to fill the viewport horizontally or vertically. However, the padding will not be added if the image is already so small that a larger padding is already visible around it. And “padding” means the distance from the image to the viewport border and not inside an image frame.
For example: let’s assume an image is 1280×720 and the viewport is 1920×1080. This means there is already a padding of 180 top and bottom and 320 left and right because PhotoSwipe will never make images larger as they are by default but only make them smaller, so they will fit the viewport. So if you set the padding to 100 on all four sides, that will not change anything at all for a large viewport. But if you have an image with an original size of 1920×1080 or larger, adding a padding of 100 on each side will cause the image to display at 1564×880 to make sure, that there is at least a padding of 100 (1080 minus 100 top and minus 100 bottom = 880).
Plugins for PhotoSwipe can also add a padding if needed to make sure there is enough space for a caption or other elements. The “dynamic caption” uses this to add empty space right or below the image for the caption box.
Edit: also just a 1 pixel border will cause problems as this changes the image size. Do not add any padding, margin or borders at all. Maybe with future versions this may change – but right now it is designed this way to display an image in front of a background without any borders.