Do deactivated plugins affect the speed of a site?

I understand, for security reasons, it is wise to remove unused plugins. However, there are plugins that I like to ‘activated only as required’.

Do deactivated plugins affect the speed of a site?

I have read on various forums that the answer is ‘NO’ however it was mentioned on a podcast that they do slow the site because they are loaded into the header.

Please can anyone clarify this.