Filter not working in internal API calls

When making an internal API request, the filter does not work:

$request = new WP_REST_Request( 'GET', '/wp/v2/media' );
$request->set_query_params( [ 'filter[media_category]' => 'test' ] );
return rest_do_request( $request );

With an external request the filter works as intended:
.../wp-json/wp/v2/media/?filter[media_category]=test

How can I fix this?
Thank you!