I was able to make it work and there is really no API that I could find, I had to go trough the whole wp-admin/includes/media.php
file to get it to work as apljdi suggested.
The link that craig552uk provided is great if you using the media upload for menu pages, but I wanted to use it on a widget. so is a bit more complicated because I need to modify the content passed by javascript and pass an extra parameter so that I did affect the default functionality of the media upload.
here are some additional hook that may be helpful for some people
add_filter( 'media_upload_tabs', 'media_upload_tabs'); //hide media tabs
add_filter( 'media_send_to_editor', 'media_send_to_editor' ); //to modified the string send by javascript
add_filter( 'media_upload_form_url', 'media_upload_form_url' ); //used to send new parameter
Thank you for all your help.