Sitemap with nginx: Getting 404 AND the content

Ok.. I followed the sitemap 404 threads.. And I fixed most. But now I have something weird.

I used the nginx rewrite rules:

# For WordPress SEO plugin
        rewrite ^/sitemap_index\.xml$ /index.php?sitemap=1 last;
        rewrite ^/([^/]+?)-sitemap([0-9]+)?\.xml$ /index.php?sitemap=$1&sitemap_n=$2 last;
        rewrite ^/news_sitemap\.xml$ /index.php?sitemap=wpseo_news last;

And when I browse to my site /sitemap_index.xml my browser shows the content. I see in my logs that Googlebot (and I get a 404!)

Did a manual GET and indeed. It returns the content, but the return code is a 404! How can I fix this?

HTTP/1.1 404 Not Found
Server: nginx/1.1.19
Date: Fri, 13 Jul 2012 21:54:04 GMT
Content-Type: text/xml
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/5.3.10-1ubuntu3.2
X-Pingback: https://XXXXXXXXX.com/xmlrpc.php
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Last-Modified: Fri, 13 Jul 2012 21:54:04 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Pragma: no-cache
X-Robots-Tag: noindex, follow

1a1
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="https://XXXXXXXXX.com/wp-content/plugins/wordpress-seo/css/xml-sitemap-xsl.php"?>
<sitemapindex xmlns="https://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://XXXXXXXXX.com/page-sitemap.xml</loc>
<lastmod>2012-07-13T21:47:33+00:00</lastmod>
</sitemap>
</sitemapindex>
<!-- XML Sitemap generated by Yoast WordPress SEO -->