Thanks for you reply.
Is there anywhere else that I can look into on my own to further troubleshoot this issue?
To iterate the problem, this problem only happened when I choose to offload images from cloudfront (s3 origin) , if I disable Cloudfront, webp images are served locally without any problem. Webp images are uploaded to S3 as well.
I suspect it has something to do with redirection. When disabling CDN, the nginx is able to handle the redirection to webp images by using try_files method. However, with CDN enabled, the images are actually loaded from Cloudfront and hence no such try_files functionality applied.
Please let me know if you have anything that can help me with solving this issue, thanks!