Hi exbm: Right now it’s not possible to extend the API, but we’re looking at the best way to allow plugin authors to do that.
The reason you can’t find anything about the rewrites in the plugin is that the API is actually designed to operate through a central server/host (which is hosted by WordPress.com), so all of that information is hosted by us. This makes it easier for people implementing against the API (they only need to point to one endpoint), but it means that that server needs to be aware of all possible URLs. In your case, if you wanted to write a custom endpoint then the WordPress.com server would need to know about it (which it doesn’t) so that it could relay those requests through to your Jetpack site.
So API requests are made against public-api.wordpress.com, which then either executes them against a blog hosted on WordPress.com, or relays those requests (as XML-RPC) to a Jetpack site. All requests that the Jetpack site sees come in as XML-RPC, even if they were actually originally made as REST/JSON.
Once we figure out a good way to allow plugins to extend the API, we’ll have documentation up at https://developer.wordpress.com/