Dynamic CSS AJAX call slows site

The call for this small CSS file take 5 to 7 seconds:

/wp-admin/admin-ajax.php?action=evolve_dynamic_css&ver=4.3.1

I have read about other themes calling their dynamic CSS with AJAX and having the same problem. One solution would be to make the CSS NOT dynamic so that this file can be cached. It would become static after a person saves their settings in the theme customizer. Any further changes will overwrite the static file.
Is that possible or is there some other solution?

Thanks