I was able to do it by editing wp-includes/link-template.php. I changed:
$link = '<a rel="shortlink" href="' . esc_url( $shortlink ) . '" title="' . $title . '">' . $text . '</a>';
to
$link = '<a rel="shortlink" href="' . esc_url( $shortlink ) . '" title="' . $title . '">' . $shortlink . '</a>';
I don’t like having to edit core files to do this kind of stuff, since you always have to remember to make the changes again whenever you upgrade.
Is there an easier way to do this in a cleaner fashion using the_shortlink()?