Terminating plugin installation by a condition

I’m try to code a new cache plugin for WP. WP executes my installation function using the register_activation_hook for installation process. Q:There is any method to cancel registering/installing process of plugin in the installation function. I tried to return false in the function but not worked. Help please.