Very slow server response time

Hi everyone,

I’m having some trouble with the server response-time (https://retailplatform.eu). Sometimes it responds very, very slow. The strange thing is that it seems to be completely random. I already optimized the speed with tools like W3TC and this improves my score in the Google Pagespeed Tool to 84/100 (when the server responds normally). See here: https://imgur.com/a/ECC4v

The second image shows what my score is like when the server does’t respond to fast (12,8 seconds…). I just can’t get my head around why this is happening so randomly. I already contacted my server provider, but they told me everything on their side is fine and I need to improve my site.

Does anyone see what is going wrong here? A response time of 10-20 seconds is just crazy. BTW: I also already installed P3 profiler, but can’t find any strange things.