I've not been able to find a module or suggestion from my searches so far.
Case:
We are importing (have successfully imported) an old site with 16,000+ items. (import_html)
tens of thousands of menu items in one Drupal menu

The structure is mostly actually relevant. Some sections (news archives, corporate reports) I've been able to exclude from menus and deal with as views and things, but that only excludes a few thousand nodes.
The rest is actually one big sitemap, of deep, not-very-wide, well-named, structured content. So it all went into one menu.

But now normal admin is really hard. There is no way a user can find the right parent item in the menu select drop-down, and each page load takes a serious load time. The menu admin page can't load.
Page serving and the menu router are holding up fine - which is great.
This is a dev site, so it's not about recovering or anything, I can apply any solution going forward, but ... what solution?

I can't find any UI add-one that will make menu management more sane. suggestions?

I can (and probably just will) divide our 9 top-level sections into their own menus, but even that doesn't solve the problem.
I've not used books yet (apart from for the true book-like annual reports) and the site doesn't seem to suit it. I can't see how to embed views pages inside books either, though I can do that with normal Drupal menus.
Solutions that involve further subdividing of content (domain or subsite-like sections) are not very practical in this Information Architecture (though if it were remodelled majorly, I'd consider that seriously)

Yes, this needs some sort of architecture re-think, but what should I look towards?

Any ideas?

Comments

Kirk’s picture

I know this doesn't help you any, but it seems to me that having 16,000 menu items is not only ridiculous, but probably totally unusable.

I would look at the content and figure out a better way to deal with it all.

dman’s picture

Well, yeah.
I'm looking for a way to break it up somewhat. I guess.
There seem to be a few subsections that still need breaking up and put into indexed bags.
But it's an import of a long-run, government site containing technical data, with many (surprisingly logically structured) subject areas, originally built in static HTML. Not time-sensitive, not (strongly) split by subject area. Just lots of it.

Some sections may be able to be pulled out into books. I'm looking for some extractions.

But every time I dig into it ... well the structure actually just happens to be a topic inside a subject area with soft relationship to its neighboring content. All sensible, at each layer, but just scaled up too far.

It's a mission to manage but actually reasonable to use when you go looking for things. Hard to imagine, I know.

dman’s picture

Useful potential module :
Menu Browser [menuux]
seems to resolve the worst of the node-edit-form issue - replacing thousands of select options with a (odd, but more context-sensitive) UI

d_l’s picture

Even with the introduction of menuux module (or other modules) the user still has to manually put node content into different "bins".

A different approach would be as follows ..

  1. batch process / parse your data silo from html and text format into validated XML content
  2. import your XML data silo (collections) into an XML data server (recommended eXist on www.exist-db.org)
  3. eXist requires to run on a tomcat server
  4. produce a reduced menu set in your drupal site (running on apache server) and each node is mapped to a series of queries running on eXist.
  5. display the results of the queries via iframe in drupal page(s).
dman’s picture

that's an interesting abstraction.
I've already got the import to Drupal nodes, paths, cck and URLs via import_html. Including menus into Drupal too.
The Drupal manu rendering is (surprisingly) handling it just fine, and navigation works.

It's only the admin UI that's letting me down.

I take it that the suggestion to let an external data store manage the 'menus' would abandon Drupal menus altogether, and you think I should build a new system instead?
The drupal menu links table is actually OK - and I'm able to query it as usual. It's just when it tries to list all items at once in select optins that things slow down, and listing all links at once just doesn't happen (even with heaps of extra memory allocated to build it, a browser page can't do it)

With the data I need already in a native SQL database, and a number of other Drupal modules integrating with it (menu_breadcrumb) I don't see how brining in an external XML datastore would simplify anything :-}

If I'm able to 'reduce' the dataset ... that looks like the most practical way forward. Right now I'm lacking an overview of which bits to reduce ... because I can't even see the sitemap !

d_l’s picture

The suggestion was to use XQuery instead of hierarchical menus to search for the content in a very large and unstructured database.

The menus would not be abandoned but would be at a top level in drupal (but certainly not running into thousands which is unmanagable).

The (few) drupal menus at top abstraction would be simply topics or subjects and would query different collections in the back-end XML server.

If you are well along the way and feel comfortable with managing the drupal menu set then stay with drupal alone.

But I assumed that you were at the beginning of an architectural rethink.

"Yes, this needs some sort of architecture re-think, but what should I look towards?"

MySQL is different from a native XML database engine.
For unstructured content the latter is often more useful.

....

Try this free utility ..

http://www.niederlandistik.fu-berlin.de/textstat/

build a corpus from your data silo and navigate the site

dman’s picture

I see what you are saying, if it WAS unstructured, and more relational then bagging it into areas, holding it together with cross-references would be the way I'd go.
What's got me stumped at the moment is that is is currently entirely structured, hierarchical, and fits a tree menu very well! It's hard to think of so much data that this would be true for, but it is. This stuff just doesn't fit a tag cloud. Not topics or collections - it really is parents and children.

Most topics (folders) are not too broad - 5 to 9 pages or subsection per folder, each logically children of the parent. but many go deeper and deeper (quite logically) and pretty soon 7^7 branches = a heck of a big tree. But it's a tree. None of the patterns are self-similar, (apart from the collections I've already excluded and bagged) but each reasonable in its position and route from the top.

Also, I'm sorta constrained by 'making it work the way it used to' so totally discarding the current structure can't happen.

The thousands is unmanageable at the moment - but that's just a logistic UI challenge. If I were able to drill down sensibly - the way I currently cannot in Drupal admin - then each level of the menu, say viewing any three levels of the menu at one time - would be feasible.
We can manage a traditional file system with appropriate file-explorer interfaces. (there are radical alternatives, but the traditional way has a lot of merit).
I don't think a big disk is unmanageable just because it has thousands of files in hundreds of (well-named) folders. But it is if your UI shows them all to you all the time.
I'm looking for (and maybe eventually will have to build) a UI that just shows us a (limited) tree-structure reflecting the Drupal menu.

The architectural re-think currently is just the potential to divide the 9 top-level sections into independent sub-branches. But going fully relational (and with an external DB) is ... too free-form ... not possible to pitch to the team :-} .

... so I'm seeing if I can AJAXify the Drupal menu widgets for now

dman’s picture

So ....
I went and re-wrote the Drupal menu admin screens :-)

http://drupalcode.org/viewvc/drupal/contributions/sandbox/dman/bigmenu/

I have made the huge menu management tree collapsed to just two levels by default, and sub-trees can selectively be opened, recursively if it gets deep.
You can still drag and drop between them once opened.

My life is sane again.
I'll review it after I get some sleep and see if it should become a project.

.dan.

d_l’s picture

I downloaded your bigmenu module .. but I had to try hacking it to "7.x" since I'm building on Drupal 7.
I've installed but I'm not sure how to test it.

...

I also noted your earlier "import_html" module you referred to above.
This module looks interesting so I tried a crude hack of six *.info files from "6.x" to "7.x" hoping it might work in drupal-7.0.

import_html.info
sample_content.info
import_html_indexes.info
import_html_skip_forms.info
spider_daemon.info
static.info

But I get this error message in the add content page ..

Creating new filter option - Unfiltered HTML             ticks O.K.

Recoverable fatal error: Argument 2 passed to db_query() must be an array, string given, called in C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\drupal-7.0\sites\all\modules\import_html\import_html.module on line 417 and defined in db_query() (line 2213 of C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\drupal-7.0\includes\database\database.inc).

Here is the line 417 in import_html.module

                // can't use filter_admin_add(); as it rewrites the whole page
                // sorry, direct SQL
417           db_query("INSERT INTO {filter_formats} (name) VALUES ('%s')", 'Unfiltered HTML');
                $fid = db_result(db_query("SELECT format from {filter_formats} where name='%s'",     'Unfiltered HTML'));

What is needed to get import_html module working in drupal-7.0?

And to test bigmenu in drupal-7.0?

dman’s picture

HAh, yeah, it was 4:30 AM. I'd not done the UI override.

replace
/admin/build/menu-customize/primary-links
with
/admin/build/bigmenu-customize/primary-links

And the new UI will be seen. I'd been testing with my own URLs, not sent the change through if you didn't know where to go!..
I'll patch that today I think.

RE import_html - it's a bit big for just an info file hack to fix!! It touches every part of Drupals internal node_save, plus ... lots. D7 progress - http://drupal.org/node/965194

dman’s picture

UI override now fixed in code.
Thanks to hook_menu_alter, it's a drop-in replacement for the menu form.

You need a site that is deeper than 3 layers in the menu tree before it starts to be worth it, it shows the top to levels already expanded, so no change there

nw’s picture

I have just visited http://drupalcode.org/viewvc/drupal/contributions/sandbox/dman/bigmenu/

Ironically the server responds with a huge list of projects. Unfortunately, I can't find bigmenu in there :-(

dman’s picture

hmm, the CVS - git migration turned all that upside-down.
I'll need to republish it under git.
It looks like some follow-up development of it will be sponsored and released in my current project though, so a solution is on the horizon

dman’s picture

http://drupal.org/sandbox/dman/1104872

Follow up there if you are interested

willeaton’s picture

Hi, is there an update to the URL, I get page not found

willeaton’s picture

DaPooch’s picture

If you're just looking to get a better hold of the parent selector dropdown in the menu item edit page take a look at Hierarchical Select and enable the menu submodule.

mpp’s picture

This discussion is quite old but the issue remains for Drupal 8.

Big menu solves part of the problem (the menu ui) but the parent selector on the node form is still an issue.

Are there any known workarounds for a huge parent menu selector (besides writing your own custom ajax handler)?

Pere Orga’s picture

For this issue in Drupal 8 (huge menu in node edit forms, etc.), try the combination of https://www.drupal.org/project/cshs and https://www.drupal.org/project/menu_link_weight

See instructions in https://www.drupal.org/project/menu_link_weight

aludescher’s picture

The Collapsible Drag 'n Drop module is another option. It's a UI change for menu editing, no AJAX or performance changes for huge menus.