The same problem raised with my site. I renamed the folder and plugin was deactivated. It seems fine now.
Now if i want to go back to version 2.1, Should i only needed to replace thank-me-later folder from my backup or upgrade has made some changes in Database too and I would be required to restore my database as well.
Need help for this. Thanks