By Phronk on
I'm working on a script to take the dmoz RDF dump data ( http://rdf.dmoz.org/ ) and import the categories (~600K of them) into Drupal as taxonomy terms in a vocabulary.
The script can produce SQL like the following:
INSERT INTO term_data ( tid, vid, name, description ) VALUES (1, 1, "Top", "Top");
INSERT INTO term_hierarchy ( tid, parent ) VALUES (1, 0);
INSERT INTO term_data ( tid, vid, name, description ) VALUES (2, 1, "Arts", "Top/Arts");
INSERT INTO term_hierarchy ( tid, parent ) VALUES (2, 1);
INSERT INTO term_data ( tid, vid, name, description ) VALUES (38, 1, "Movies", "Top/Arts/Movies");
INSERT INTO term_hierarchy ( tid, parent ) VALUES (38, 2);
INSERT INTO term_data ( tid, vid, name, description ) VALUES (465672, 1, "Titles", "Top/Arts/Movies/Titles");
...
It's very much a work in progress, and at this point, I don't have any specific questions to ask anyone. The purpose of this post is simply to let people know what I'm doing, in case anyone is interested in the same thing, or has any suggestions or knows of any gotchas.
-Frank.
Comments
wow
It would be interesting to hear how this goes. Should be a large taxonomy.
Yes, very large
In fact, it seems to break the normal taxonomy logic. If I try viewing the terms of the taxonomy, drupal seems to just give up (haven't dug into it enough to pinpoint the cause yet though).
I think I'll have to invent other ways of selecting terms than a single list.. perhaps some kind of taxonomy search filter, e.g type in one or more terms, press a button, and a smaller list of dmoz categories will be displayed for your selection pleasure.
There was a similar
There was a similar discussion a month or so ago, and it turns out that there is a bug in some taxonomy function that did not scale well with tens of thousands of terms.
As I recall Gerhard Killesreiter (killes) came up with a fix for it.
Can't remember what the title of the post was, nor was it in the forums or the drupal-devel mailing list.
--
Consulting: 2bits.com
Personal: Baheyeldin.com
--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba
morbus did
Morbus has one. I do not know whether it hit already the core or not. Also, there is a pager for this .
--
Drupal development: making the world better, one patch at a time. | A bedroom without a teddy is like a face without a smile.
taxonomy_get_tree
The first problem I encountered when trying to administer my huge taxonomy was the PHP memory_limit was being exceeded. So I bumped it up from 8M to 128M and I was able to start viewing the terms (not pretty, but works for debugging purposes).
I'm using 4.6, which has a pager for free tagging vocabularies, and I've turned free tagging on. I can now import a 'large' amount of DMOZ data and view the terms using the standard admin form.
The problem I'm now seeing is that when paging through the taxonomy, the operation of viewing a page is linear with the size of the vocabulary, since the entire taxonomy tree is slurped in for each page view. Once the vocabulary gets 'big enough', things go to s***.
It looks like I'm going to have to write my own UI for administering and assigning terms to nodes. Something that doesn't rely on slurping in the entire vocabulary each time -- I'm currently thinking something involving keyword searches of the taxonomy. For instance, I type in 'linux', and then I'm presented with a list of DMOZ topics that match... then I pick the ones I want from that more managable list. It's sort of like using free tagging to help select a list of potential formal terms.
I'll keep posting here as I progress.
ajax?
Good idea. Another approach may be to use xmlhttprequest to load each parent vocabulary at a time. Discussed a little here and sounds like factoryjoe (Chris) is working on something.
Instead of loading the entire vocab perhaps xmlhttprequest is an option to load the first level ... then select ... load second level based on selection ... then select ... load third level based on selection ... etc.
Dmoz+Weblink.module
I'm not sure does it fit here, but I found nice directory script which can work standalone or with DMOZ data. It has a long feature list, which might be taken as some kind of TODO for the weblink.module implementation.
Description page is here.
weblink is not browsing interface.
Eventhough people often need a directory interface to shoz weblinks, zeblinks has very little to do with a directory view. In fqct it has nothing to do with it, other than that weblinks are displayed in such an interface more often then other types.
So, no, we will not include any dmoz features and interfaces in weblinks. Please see Matthias' directory.module for that.
---
If this solved you problem, please report back. This will help others whom are looking for the same solution.
Next time, please consider to filing a support request.
[Bèr Kessels | Drupal services www.webschuur.co