@stodorovic @donncha thanks for the replies, I will try updating to the newest version of the plugin and see if that solves the issue.
Due to the nature of the site traffic we implemented a static 404 page. I will review the code and how that is being implemented as well if the newest update does not solve the issue.