empty all caches issue

Hi
Recently I have problem with clearing caches and with my site. Sometime in may dashboard I have internal server error and it’s not related with memory limit. After click on empty all caches It’s removing a lot of time cache. In first try usually there is internal server error 500 second or third try clearing normally with notice at the end All “caches successfully emptied.” and my site has cleared cache and another clearings goes well, but strange thing is that in folder cache there is still a LOT of cached files.

My file manager panel in my host crashed after calculationg numbers of files so my hosting provider disabled this calculating files option for me and now I have possibility to open manager of files.

I have question. Can I remove this cache folder manually ? or some folders from it (config , db, minify, object, page_enhanced, tmp)? What will be the ressult ? W3 total cache automaticly rebuild this folder after deleting it ? I’d like to clear all these files maybe they have bad influance on my site…

https://www.ads-software.com/plugins/w3-total-cache/