Hi,

With the recent versions of the biblio module, we have importation issues, which did were not happening with early versions of biblio v6. When importing a large number of references, the time taken is very very long. I join the problematic bibtex file for convenience. I tried to split it in two files to ease importation a little bit, but the final number of imported references was surprisingly far less than the real number. Any hint? Thanks

CommentFileSizeAuthor
#4 tout.bib_.zip265.23 KBsebos69
colloques.bib_.zip74.98 KBsebos69

Comments

ntripcevich’s picture

ntripcevich commented

Using the v5.7 version I've been having to limit my import (Endnote X1 XML) to less than 100 records per file.

rjerome’s picture

rjerome commented

There is a new routine in 6.x which converts "latex" encoded special characters to UTF8. This seems to be where the problem lies, but I'm on vacation for the next few weeks so don't expect a fix before then.

Ron.

sebos69’s picture

sebos69 commented

OK, I will try to cope with that until you return. Have nice holidays in the meantime!

sebos69’s picture

sebos69 commented
StatusFileSize
new tout.bib_.zip265.23 KB

A followup: u just tried beta 9, and it imports the entries fine, but accessing the /biblio page takes a very long time (1 to 2 minutes).

I join an updated version of our .bib database if someone wants to test it...

sebos69’s picture

sebos69 commented
Title: Import woes : limit on number of references? » limit on number of references?
Version: 6.x-1.0-beta8 » 6.x-1.0-beta9
rjerome’s picture

rjerome commented

I imported this file onto my test site and although it took quite a while to import (and it did revel some errors in my import code) once imported the page load (/biblio) times are normal.

Ron.

P.S. The errors I eluded to above have corrupted some of your entries so I'll let you know when it's fixed and you can re-import them.

sebos69’s picture

sebos69 commented

Thanks for your support!

Finally, I have found that the problem with the display comes from the fact that my site is a multilingual (French/English, default is French). The /fr/biblio page displays as fast as expected while the /en/biblio page takes ages...

On the server side, it is the mysqld process which is talking a long time. (shown by the 'top' command using linux)

Should I open a new bug? Do you need more info?

sebos69’s picture

sebos69 commented

More investigations : it seems that there is a problem with i18n :

the page /fr/biblio takes a few seconds to display, and the corresponding mysql request is :

SELECT DISTINCT n.*, b.*, bt.name as biblio_type_name FROM node n left join biblio b on n.vid=b.vid left join biblio_types bt on b.biblio_type=bt.tid WHERE (n.language ='fr' OR n.language ='' OR n.language IS NULL) AND ( n.type='biblio' )ORDER BY b.biblio_year DESC, SUBSTRING(n.title,1,3) ASC LIMIT 0, 25

whereas the page /en/biblio takes a long time. The corresponding mysql request is :
SELECT DISTINCT n.*, b.*, bt.name as biblio_type_name FROM node n left join biblio b on n.vid=b.vid left join biblio_types bt on b.biblio_type=bt.tid LEFT JOIN node i18n ON n.tnid = i18n.tnid AND i18n.language = 'en' WHERE (n.language ='en' OR n.language ='' OR n.language IS NULL OR (n.language = 'fr' AND i18n.nid IS NULL)) AND ( n.type='biblio' )ORDER BY b.biblio_year DESC, SUBSTRING(n.title,1,3) ASC LIMIT 0, 25;

it seems that removing the parts like:
"LEFT JOIN node i18n ON n.tnid = i18n.tnid AND i18n.language = 'en'"
"AND i18n.nid IS NULL"
makes the request faster (tested directly with the command line, sorry, no patch available)

should I bug the i18n guys?

Frank Steiner’s picture

Frank Steiner commented

No, this is the correct behaviour and it's caused by your i18n settings. You have set "mixed", that means on the language selection page you have "Current language and default language". Therefore the query must consider all english and all french nodes which causes this more complicated query.

Consider if you really need the mixed mode. This will show you all pages which are either in french or in english. I'm not this makes sense.

You could set "current language only" to get rid of this, but then you will see always only french or only english pages.

If you have pages that are available in one language only but want them to be shown as long as there is no translation for the current language (an english page for a french speaking viewer might be better than no page), you can also try the "current language and no language" and define those page that are available in one language only as "language neutral" (selection box in the node edit page). Only you really translate a page into both languages, set it "french" or "englisch" language. Then you will always see "french and language neutral pages" or "english and language neutral pages" with the "current language and no language" setting.

sebos69’s picture

sebos69 commented
Status: Active » Closed (won't fix)

No, this is the correct behaviour and it's caused by your i18n settings. You have set "mixed", that means on the language selection page you have "Current language and default language". Therefore the query must consider all english and all french nodes which causes this more complicated query.

Thanks! I switched to "Current language and default language" and the biblio page is fast in both cases!
So my case is closed :)

I am thinking about other users who could face the same problems. Would there be a way to warn the user about the potential slowdown caused by choosing this option?

Anyway, thanks a lot, again!