Custom Featured Image Uploader

I’m trying to add a custom featured image uploader meta box, but I’m having issues getting to actually set the uploaded image as a featured image (I only get the insert into post link). The end goal is to allow the user to set a featured image, but I will restrict the process so if they upload a new image, it will delete the old one, and they won’t have access to the media library (this is on a public custom post). Here is how far I got on this feature, it won’t actually set the uploaded image as a featured image (no featured image link after upload).

// Hook to add a custom meta box.
add_action( 'add_meta_boxes', 'ca_create_meta' );

// Register meta boxes.
function ca_create_meta() {

	// Register an image upload meta box.
	add_meta_box( 'ca-outfit-image', 'Outfit Banner', 'ca_outfit_banner', 'outfits-toplist', 'normal', 'high' );
}

// Create the banner upload meta box.
function ca_outfit_banner( $post ) {

	// Load scripts and styles for image upload.
	wp_enqueue_script( 'media-upload' );
	wp_enqueue_script( 'set-post-thumbnail' );
	wp_enqueue_script( 'thickbox' );
	wp_register_script( 'ca-toplist-script', WP_PLUGIN_URL.'/ca-toplist/ca-script.js', array( 'jquery', 'media-upload', 'thickbox' ) );
	wp_enqueue_script( 'ca-toplist-script' );
	wp_enqueue_style( 'thickbox' );

	$post_id = $post->ID;
	$upload_iframe_src = esc_url( get_upload_iframe_src('image', $post_id) );
	$set_thumbnail_link = '<p class="hide-if-no-js"><a title="' . esc_attr__( 'Set Outfit Image' ) . '" href="%s" id="set-post-thumbnail" class="thickbox">%s</a></p>';
	echo sprintf( $set_thumbnail_link, $upload_iframe_src, esc_html__( 'Set Outfit Image' ) );

	// Echo the attached image.
	echo wp_get_attachment_image( get_post_meta( $post->ID, '_thumbnail_id', true ) );
}