After some stable time...
First I had this error

PDOException: SQLSTATE[HY000]: General error: 3 Error writing file '/tmp/MYj2Xtbp' (Errcode: 28): SELECT x.loc AS loc, x.lastmod AS lastmod, x.changefreq AS changefreq, x.changecount AS changecount, x.priority AS priority, x.language AS language, x.access AS access, x.status AS status FROM {xmlsitemap} x WHERE (x.access = :db_condition_placeholder_0) AND (x.status = :db_condition_placeholder_1) ORDER BY x.language DESC, x.loc ASC LIMIT 50000 OFFSET 0; Array ( [:db_condition_placeholder_0] => 1 [:db_condition_placeholder_1] => 1 ) en xmlsitemap_generate_chunk() (línea 165 de /home/empresa/public_html/sites/all/modules/xmlsitemap/xmlsitemap.generate.inc).

sitemap.xml gives page not found.
Then after no success running cron sitemaps i tried to regenerate updating cache files, then i had this error:

Hubo un error HTTP AJAX. Código de Resultado HTTP: 500 A continuación se detalla la información de depuración. Ruta: /batch?render=overlay&id=1248&op=do StatusText: error ResponseText: PDOException: SQLSTATE[HY000]: General error: 3 Error writing file '/tmp/MYPO9q7u' (Errcode: 28): SELECT x.loc AS loc, x.lastmod AS lastmod, x.changefreq AS changefreq, x.changecount AS changecount, x.priority AS priority, x.language AS language, x.access AS access, x.status AS status FROM {xmlsitemap} x WHERE (x.access = :db_condition_placeholder_0) AND (x.status = :db_condition_placeholder_1) ORDER BY x.language DESC, x.loc ASC LIMIT 50000 OFFSET 0; Array ( [:db_condition_placeholder_0] => 1 [:db_condition_placeholder_1] => 1 ) en xmlsitemap_generate_chunk() (línea 165 de /home/empresa/public_html/sites/all/modules/xmlsitemap/xmlsitemap.generate.inc).

Do you have any solution but rebuilding index, rebuild index is time consuming i have >300.000 and i will lost my google SEO if i rebuild and Do will have to rebuild each time a single error occurs? i do expect to have > 2.000.000
why is sitemap.xml disapearing if it is a small file, why is so sensible that an error destroys all the indexing work?
Please help

Comments

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented
Status: Active » Postponed (maintainer needs more info)

First of all, you are *not* going to "lose" your Google SEO if your sitemap is not working. That's just false.

Your webhost is having problems writing files to your server's /tmp directory. You should forward this error to them to see if they can help debug this.

egarias’s picture

egarias commented

Many many thanks for answering so fast.
I have enought space and the /tmp is 777 after this incident, I asked my hosting to place the /tmp on my account to have more control if needed, I have space please tell me what I can check?
Regarding SEO, This Sitemap Module is very important for me. When i do not submit sitemap to google (or I start to resubmit) i see a lower results on "seach queries" on google webmaster tools. It take time to increase this number in google and this number is directly related with traffic to my website. Really, Sitemap is a great module! I do need it to work clean

egarias’s picture

egarias commented
Status: Postponed (maintainer needs more info) » Active

I do change the status to active, please any help before having to regenerate from scratch again
Thanks

egarias’s picture

egarias commented
Status: Active » Closed (works as designed)

You were right, the problem is in the /tmp and its size, i needed to increase the size.
Could i make a suggestion? could you design in order not to need such amount of size for temporal files, or could you maybe write a complete sitemap in steps so if one step goes to error it doesn't affect the previous correct sitemap (this was my case, my sitemap was ok until it needed more memory to perform the process for more nodes) why it doesn't simply say i cannot process more nodes but you still have the sitemap?
and, it is possible to have a limited size sitemap, so new nodes replace the older ones? even if you still have them in the tables
Thanks for this extremely usefull module and sorry if my questions are silly.

Enrique

egarias’s picture

egarias commented
Category: bug » feature
Status: Closed (works as designed) » Active

While rebuilding my sitemap I found at the end of normal completion that some of the initial files were lost, I think it is because cron started sitemap processing in between, after this, i decided to rebuild again and i noticed when starting to rebuild, that all the files were deleted. Could it possible be to keep the good sitemap until each sitemap process is correctly finished, if not a single error causes to lost the full sitemap. My sitemap has now 41 files so i would prefer a maybe more safe update, is this possible?

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented
Version: 7.x-2.0-beta2 » 7.x-2.x-dev

There are plenty of options to try and help reduce the file size of each sitemap page. We only write one file at a time via the Drupal File API, which at the end of writing, gets moved to the files directory. I could possibly investigate not using file_save_data() and writing to the files directory itself rather than the temp dir + move.

I can't change the file deletion behavior on regeneration because an error would leave you with some files from the rebuild, and some files from the older build, which could conflict or mis-match. That's why we clear the sitemap's files as the first step of regeneration. Feel free to create a separate issue if you can provide a patch, but it's not something I have time to work on.

egarias’s picture

egarias commented

I do understand that it should be not a common situation having such amount of sitemaps, could i suggest to build the sitemap but deleting the old only when the new is all done?
This may avoid lost of sitemap if error while building and also inconsistencies. The problem is with such amount of files, rebuilding take time, and you have to be aware to notice the problem because google starts marking sitemaps as errror and then you have to go fast resubmiting to google its very time consuming.
What i did is to copy the correct sitemap on a local directory so if i loose it at least i can have the last situation my question is if I do this it will be no inconsistencies? your process is writting something in the database? or only in the xml files?

many thanks for your time

Anonymous’s picture

Anonymous (not verified) commented
Priority: Major » Normal
Status: Active » Postponed (maintainer needs more info)

I think the point that we need to keep a valid sitemap.xml for viewing while the new is rebuilding is a valid one. However, the user of Drupal can create a tmp directory in the sites directory containing the settings file. You then configure Drupal to use that directory as the temporary directory. Does that help?

egarias’s picture

egarias commented

Thanks for taking into account my suggestion, if you plan to modify the module with this functionality it is great!!
I didn't understand the temporary and the tmp for settings where i can see more info regarding this?

Anonymous’s picture

Anonymous (not verified) commented

In D6 you can find the form at admin/settings/file-system, so whatever that moved to in D7 which I don't have up yet. You can change the Temporary files setting from /tmp to one you create elsewhere. If you're running multiple sites then it is suggested that you modify this setting per site so that the sites do not have a potential for conflict. I create directory sites/mysite/tmp, chmod 777 sites/mysite/tmp and change the admin setting to sites/mysite/tmp for each site I operate.

egarias’s picture

egarias commented

In D7 I cannot see where to modify this /tmp path.
There is the option at admin/config/media/file-system and a item to fill for the temporary directory, In my case it points to sites/default/files/updates and the time i see it is used is in the updates, for xmlsitemap for example, the /tmp is allways used.
Maybe you are right and my sitemap cron is conflicting with my other sites that for the same reason use the same /tmp wich is then shared on my VPS.
Having personal /tmp for each site seems to be good, but in any case I still think that while constructing we should keep a valid sitemap avoiding loss of all data if there's any problem and having 41 xml files takes time to process the cron so we are sensible to any error and in the meantime even if the cron ends OK, if google searches for any sitemap it will not find it.
Another issue for me is that at each cron, the 41 files (and growing) are rebuild, I am only adding nodes so at cron it would be great to have a process that only adds to sitemap, having the option to manually rebuilding with the GUI if some of the nodes are deleted. This maybe is specific for me but if you can give me any help i will much appreciate.

egarias’s picture

egarias commented
Status: Postponed (maintainer needs more info) » Active

I would prefer to have this active.
I managed to have a private /tmp for the Mysql tmp files so it will be better but ,I am not updating the sitemap because each time I have an error it takes me close to 1 day to have the system ok including rebuilding and google webmaster, i mean it.
I will update the sitemaps more manually, doing backup of the previous sitemap, and then updating.
Any help will be good.

egarias’s picture

egarias commented

Is there any chances to have a way to "only add new nodes to sitemap" avoiding the full rebuild?
Thanks

Anonymous’s picture

Anonymous (not verified) commented

In D7 I cannot see where to modify this /tmp path.
There is the option at admin/config/media/file-system and a item to fill for the temporary directory

I finally got D7 installed in development and this is field is where you give it a temporary directory, in *NIX it defaults to /tmp and in Windows it defaults to c:\temp or some other appropriate directory. I tend to create a directory in sites/mysite named tmp and then put the relative directory sites/mysite/tmp in the field.

In my case it points to sites/default/files/updates

Uhm, it perhaps shouldn't be set to a child directory of files/. I can see this might have undesirable results.

the time i see it is used is in the updates, for xmlsitemap for example, the /tmp is allways used.

If the Temporary Directory is being filled with something besides /tmp and /tmp is being used, then IMO there is something wrong somewhere. This value really should be used for all temporary file actions.