I have this issue as well. I used the same plugin as ukasru to measure performance and found that WP Security Audit Log takes up 50% of the loading time for my site. This occurred after my admin user had done a bunch of work, but the profiling test involved only viewing pages.
I do really like this plugin – it keeps track of everything I need. Was there anything discussed in the email that would be relevant to other users?