Alright, so you mentioned that you changed the field type in the database to decimal, that’s good.
The next thing to do is to find all instances of the quantity variable in the php files of the WP ECOMMERCE plugin directory.
Specifically look in: /wp-content/plugins/wp-e-commerce/wpsc-includes
and at: ajax.functions.php
What you want do is change instances of (int)$_POST['quantity'];
to (float)$_POST['quantity'];
Also change instances of (int)$_POST['wpsc_quantity_update'];
to (float)$_POST['wpsc_quantity_update'];
as well as (int)$_POST['wpsc_quantity_update'][$_POST['product_id']];
to (float)$_POST['wpsc_quantity_update'][$_POST['product_id']];
You will also want to modify your checkout and shopping cart to display the quantity as a floating point decimal rather than as an integer. You may need to expand some of the input widths to allow for the display of the decimal.