My server:
nginx
boost 1.18
xmlsitemap module 6.x-2.0-beta1

My site have many links which xml sitemap have create sitemap.xml and sitemap.xml?page=1 2 ... of it.

The problem is Boost only cache sitemap.xml not cache page=1, page=2 ...
Everytime googlebot access the sitemaps, the inner sitemap.xml?page=1 ... alway returen the same content of sitemap.xml.
That cause googlebot can't get the sitemap.

regards

http://articlesforge.com

Comments

mikeytown2’s picture

does page query strings work with html on your site?

wqmeng’s picture

Hi,

Yes, they should work, I have views created pages, they all ok with page=1, ...

I have seen the html file and the gz file created by boost with the page=1 in the file name.

While boost will only create sitemap.xml_.xml and sitemap.xml_.xml.gz, no page=1 file will be created even I access sitemap.xml?page=1.

Now I haved added sitemap.xml* to the excpetion list. And after a few hours, google may download the sitemap right again.

I am not sure which cause this problem, boost or something else like rewrite of nginx, seems that I have no problem with apache server before.

Thanks

http://articlesforge.com

wqmeng’s picture

Here is some result from google bot. You may see that the sitemap.xml?page=1 indeed return the content of sitemap.xml, and in the cache folder, there are only sitemap.xml_.xml and sitemap.xml_.xml.gz, no inner page files will be created.

This is how Googlebot fetched the page.

URL: http://articlesforge.com/sitemap.xml?page=1

Date: Wed Jun 02 20:30:46 PDT 2010

Googlebot Type: Web

HTTP/1.1 200 OK
Server: nginx/0.8.37
Date: Thu, 03 Jun 2010 03:31:16 GMT
Content-Type: application/rss+xml; charset=utf-8
Content-Length: 260
Last-Modified: Wed, 02 Jun 2010 03:32:44 GMT
Connection: keep-alive
Keep-Alive: timeout=20
Vary: Accept-Encoding
Content-Encoding: gzip
Expires: Tue, 24 Jan 1984 08:00:00 GMT
Cache-Control: must-revalidate, post-check=0, pre-check=0
X-Header: Boost Citrus 2.4

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="http://articlesforge.com/sitemap.xsl"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>http://articlesforge.com/sitemap.xml?page=1</loc><lastmod>2010-06-02T01:00Z</lastmod></sitemap>
<sitemap><loc>http://articlesforge.com/sitemap.xml?page=2</loc><lastmod>2010-06-02T01:00Z</lastmod></sitemap>
</sitemapindex>
<!-- Page cached by Boost @ 2010-06-02 07:32:40, expires @ 2010-06-03 07:32:40 -->

Thanks

http://articlesforge.com

wqmeng’s picture

If I don't cache sitemap with boost, It will get

This is how Googlebot fetched the page.

URL: http://articlesforge.com/sitemap.xml?page=1

Date: Wed Jun 02 21:08:27 PDT 2010

Googlebot Type: Web

HTTP/1.1 200 OK
Server: nginx/0.8.37
Date: Thu, 03 Jun 2010 04:08:57 GMT
Content-Type: text/xml; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=20
Vary: Accept-Encoding
X-Powered-By: PHP/5.2.13
Set-Cookie: SESSfbd8eddb2cff5834e6f8a0935439485c=1179edc2af3f7f093810c3ebb452ab36; expires=Sat, 26-Jun-2010 07:42:17 GMT; path=/; domain=.articlesforge.com
Cache-Control: post-check=0, pre-check=0,must-revalidate
Last-modified: Wed, 02 Jun 2010 10:36:30 +0000
Etag: "a05731ce7ff43135c998c7e808c5cd70"
Expires: Thu, 03 Jun 2010 10:36:30 +0000
X-Robots-Tag: noindex, follow
Content-Encoding: gzip

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="http://articlesforge.com/sitemap.xsl"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>http://articlesforge.com/</loc><changefreq>daily</changefreq><priority>1.0</priority></url>
<url><loc>http://articlesforge.com/blog</loc><priority>0.9</priority></url>
<url><loc>http://articlesforge.com/faq</loc><priority>0.9</priority></url>
<url><loc>http://articlesforge.com/why-submit-articles</loc><priority>0.9</priority></url>
...
too long

Thanks

http://articlesforge.com

mikeytown2’s picture

do you have xml caching turned off? I just tried your sitemap.xml file and I don't see the boost tag or any tag in the headers.

wqmeng’s picture

Hi, I have add sitemap.xml* to the avoid cache list. So meantime the cache disabled for sitemap.

thanks