Restore parent-child order of pages in Beyond the Basics book

This could have been an issue with the menu upgrade. It would probably be best to export the tree info from the D5 dump and compare it to the tree in the D6 dump. Of course we have a dump, but we do not have a site running the dump, so we just need to export the book info.

Looks like that whole Beyond the Basics handbook is badly scrambled.

Raising the priority to critical. It's a huge percentage of the documentation that's in a bad state.

It was also noted on #376920: Drupal.org menu messed up that breadcrumbs here and on module release pages are messed up.

Changing title to be more inclusive of problems

Oops wrong handbook. It's BtB that has the problems.

Reverting to Gabor's title.

This is happening on a great deal of documentation beyond the BtB handbook.

was this a problem with the data, or a bug in the book update function?

Where do you see it? I've been poking around the other handbooks and I've only seen a problem with BtB (which is pretty much toast).

I see it in many documentation pages for contrib, which is what fago was reporting in the original post. http://drupal.org/node/206787, for instance, used to have child pages for most contrib spam modules, including Mollom (http://drupal.org/node/338553).

Chiming in so I'll be aware if help is needed - I'm available to help.

Keith, all the contrib documents are in BtB (or at least they were until the upgrade hit).

Links from #10 are from Beyond the Basics handbook. I also went to /admin/content/book/* to have an overview of the each book's order. It seems that it is only BtB which is affected.

Ah. I didn't realize that. My bad.

The print module documentation is also gone.. I don't even remember correctly the 7-8 child pages of the main page.. Should I fix the links, or will this be fixed from the backup?

Preferably, this would be done without losing any recent changes to the nodes themselves.. It should be a simple matter of restoring the children/parent relations.

João Ventura

@jcnventura: yes, the idea is that we'd restore the parenting relationships, therefore my title "Not all docs pages got parented properly". Correct parenting should also solve breadcrumbs, since that is generated from the parenting data.

changed the title (again) to reflect the nature of the problem. As said above, it seems that this only affected the Beyond the Basics (BtB) handbook.

Subscribing.

subscribe - can help update pages if needed.

Subscribe.

All of my documentation got "unhooked". None of the child pages correspond to the top parent page (http://drupal.org/node/163565). And, no matter how many times I edit the "book outline" of top parent node, I'm unable to see it anywhere that I actually place it (when it does appear, it appears under some other node that I did not select).

What is the status on this? Is some fix in the works? Is there a workaround available now?

Here is all the parenting data from the Drupal 5 table (dump of the book module) as well as the book table as it is on the live site. The schema is wildly different (part of which is the reason that the D6 dump is significantly smaller), so it probably requires some data mining to find out what is exactly missing. Theoretically, some items are not even in the D6 book table, so that might prove a quick avenue to find the problematic nodes.

I hope someone from the documentation team can pick this up, since you have more knowledge on what to expect in the books.

Thanks for the dumps, but unfortunately I'm not sure that we have someone available on the docs team who is savvy with SQL . My time is overbooked as it is until after Drupalcon. If we can't get a SQL guru in here to look at it in the next day or two, we will probably just start rebuilding the book manually, since a much larger number of people can work on that task at least.

subscribing

thank god for faceted search, so I can still find stuff

I also just want to clarify that the job that Gabor and the infra team has done with the upgrade was amazing and I am sure you all are quite exhausted. I do not mean to imply that you guys should, or have time to, fix this. I just want the docs team and such to be on notice that we will most likely fall back to manually fixing this in the next few days and that we will need to coordinate that effort.

If someone does have time to work out a programmatic solution, please do post!

I have at least a small amount of time I can spend working on a programatic solution to this problem.

It looks like in Drupal 6, the book structure (parent/child relationships) is actually pulled from the menu_links table. @Gábor Hojtsy: Is it possible to post dumps of this table from both D5 and D6?

Thanks,

Jeff

I've just did a comparison of nids appearing in the two dumps. I've ordered the node ids in ascending order in both dumps and exported the list and made a diff. This unviels a number of missing nodes in the new book and a number of new nodes. The new nodes are all at the end, which means they are all new nodes, but the missing nodes are all around the map. I'll look these nodes up in the old dump and see where should they have been.

Ok, the above diff unearthed 30 missing nodes in the current book. Here is the parenting data for those:

mysql> select node.nid, node.title, book.parent, parent_node.title, book.weight from node left join book on node.vid = book.vid left join node parent_node on book.parent = parent_node.nid where node.nid in (42592 ,62462 ,62470 ,62532 ,82661 ,92375 ,97573 ,101708 ,101735 ,101736 ,101746 ,105808 ,106716 ,106951 ,112792 ,126530 ,142322 ,149225 ,149232 ,149233 ,149234 ,149236 ,149237 ,180013 ,249907 ,290443 ,297030 ,313110 ,322855 ,325939);
+--------+--------------------------------------------------------------------+--------+-----------------------------------+--------+
| nid    | title                                                              | parent | title                             | weight |
+--------+--------------------------------------------------------------------+--------+-----------------------------------+--------+
|  42592 | Views                                                              |  22361 | Contributed module documentation  |      0 | 
|  62462 | CCK for Themers                                                    | 101723 | Content Construction Kit Handbook |      0 | 
|  62470 | FAQs                                                               | 101723 | Content Construction Kit Handbook |      0 | 
|  62532 | Glossary                                                           |  62459 | Content Construction Kit (CCK)    |      0 | 
|  82661 | CCK Overview and Structure                                         | 101742 | CCK for Developers                |      0 | 
|  92375 | Related Modules                                                    | 101723 | Content Construction Kit Handbook |      0 | 
|  97573 | Drupal 4.7.x-dev                                                   |   NULL | NULL                              |   NULL | 
| 101708 | Field and Widget Settings                                          | 101742 | CCK for Developers                |      0 | 
| 101735 | CCK for Administrators                                             | 101723 | Content Construction Kit Handbook |      0 | 
| 101736 | Installing and Updating CCK                                        | 101723 | Content Construction Kit Handbook |      0 | 
| 101746 | CCK Hook Execution Order                                           | 101742 | CCK for Developers                |      0 | 
| 105808 | Practical example of CCK's flexibility                             | 101742 | CCK for Developers                |      0 | 
| 106716 | A Sample CCK Field Module                                          | 101742 | CCK for Developers                |      0 | 
| 106951 | HOW TO: Teasers in CCK nodes                                       | 101742 | CCK for Developers                |      0 | 
| 112792 | Single, Multiple, and Shared Fields                                | 101742 | CCK for Developers                |      0 | 
| 126530 | Calculating depreciation and valuation                             | 149228 | Computed Field Code Snippets      |      0 | 
| 142322 | Liquid Wiki Engine Project: providing wiki functionality to Drupal |   NULL | NULL                              |   NULL | 
| 149225 | Configuring Computed Field                                         | 126522 | CCK Computed Field                |     -4 | 
| 149232 | Making a node link to itself                                       | 149228 | Computed Field Code Snippets      |      0 | 
| 149233 | Adding two other fields                                            | 149228 | Computed Field Code Snippets      |      0 | 
| 149234 | Calculating a duration given a start and end time                  | 149228 | Computed Field Code Snippets      |      0 | 
| 149236 | Calculating average speed                                          | 149228 | Computed Field Code Snippets      |      0 | 
| 149237 | Calculating age from birthday                                      | 149228 | Computed Field Code Snippets      |      0 | 
| 180013 | Computing a field from a view                                      | 149228 | Computed Field Code Snippets      |      0 | 
| 249907 | Working with multiple values                                       | 149228 | Computed Field Code Snippets      |      0 | 
| 290443 | Reference a CCK field from a different content type                | 149228 | Computed Field Code Snippets      |      0 | 
| 297030 | Adding a field to a vocabulary                                     | 149228 | Computed Field Code Snippets      |      0 | 
| 313110 | Adding an 'auto_increment' field                                   | 149228 | Computed Field Code Snippets      |      0 | 
| 322855 | Exploding a string snippet                                         | 149228 | Computed Field Code Snippets      |      0 | 
| 325939 | getting data from a nodereferrer                                   | 149228 | Computed Field Code Snippets      |      0 | 
+--------+--------------------------------------------------------------------+--------+-----------------------------------+--------+
30 rows in set (0.00 sec)

Since this is a rather limited number of nodes, it would probably take substantially more time to write, test and deploy a programmed solution, compared to doing the fixup by hand. So I am hopeful that the documentation team can pick this up from here.

BTW for the two nodes which lacked parenting info for their current vid (shows all NULL above for parent info), if we join on the nid instead, we get older data, which will be useful to fix their parenting now:

+--------+--------------------------------------------------------------------+--------+-----------------------------------+--------+
| nid    | title                                                              | parent | title                             | weight |
+--------+--------------------------------------------------------------------+--------+-----------------------------------+--------+
|  97573 | Drupal 4.7.x-dev                                                   |  10259 | Contribute code                   |     10 |
| 142322 | Liquid Wiki Engine Project: providing wiki functionality to Drupal |  22361 | Contributed module documentation  |      0 | 
+--------+--------------------------------------------------------------------+--------+-----------------------------------+--------+

This should complete the set of information you need to fix this up!

I'm not sure this is the extent of the problem. For example, I don't see any Rules module docs on this list, and I know there are problems there. Correct me if I'm wrong, but I think there are two issues with running a diff on the two dumps to identify problem nodes:

1) This only seeks to identify nodes without parent data. As I browse through the current Beyond the Basics book, it seems that there are quite a few nodes with INCORRECT parent as well.

2) Parent data in Drupal 6 is not stored in the book table. The only fields there are nid, bid (nid of the "cover page" of the book), and mlid(a link to the menu_links entry for the node). The menu_links table then has a field that stores the parent.

Therefore, it seems to me that we could have:
1) Incorrect book.mlid values
or
2) Incorrect menu_links.plid values (the parent link)
in addition to the nodes missing from the books table.

I don't claim to be an expert, and I don't have much experience with the book module, but I'm happy to give it a shot. However, I would need:
- Drupal 6 menu_links dump
- Drupal 5 menu dump (I think that's what the table was called)

Gabor - As others have said, thank you for all of your hard work on this upgrade.

As I have said, the Drupal 5 menu data is pointless for book data mining. The number of records in the D5 menu table is 228, while the number of book nodes is above 4000. The D5 menu did not care about books and the D5 books did not care about menus.

I need to run know, but will look into getting that menu_links data from Drual 6 as soon as I get back.

Here is the D6 menu_links dump. I've verified that this contains all the mlid-pid parenting data, while the D6 book dump above contains nid-mlid relations, so you can recreate the hierarchy as seen by D5 (book table) and D6 (book and menu_links tables); compare and come up with stuff to fix. I'm sadly not going to have more time to work on this.

Thanks for the additional information. I'll starting comparing and trying to come up with a fix.

I just wanted to provide an update. This is a fairly complex problem. There are several different types of inconsistencies in the two tables. The biggest problem (affecting over 1,400 nodes) is that the entry in the book table reports node as a part of book 257 (Beyond Basics), but the corresponding entry in the menu_links table places the link in a menu called 'book-toc-258' (the menu for the Getting Started book). This is what's causing most of the wacky behavior I think. There are a couple of other issues as well.

I expect to post an SQL fix by the end of the night.

Jeff

Is anyone familiar with the p1 - p9 fields in the menu_links table? I understand that they represent the entire menu tree down to the link. My question is, does rebuilding the menus or some similar process set these to the correct values, or must they be manually set to the correct values? The 1,400+ nodes I mentioned above all have invalid data in there. I can write a script to change them if necessary.

Unfortunately, I'm not going to be able to finish this tonight. I'm in the process of writing an SQL script that should correct all of the errors in the tables relating to this book. I'm about 80% done at this point.

Once I complete the script, I can test it against the table dumps I have, but that's about it. Is there a dev/test copy of drupal.org somewhere that this can be tested on before messing with the live db?

I've been working on this issue also. The SQL dumps posted by Gabor in #22 aren't quite sufficient for generating a complete SQL dump to fix the problem, since we also need the contents of the node table to pull in the titles.

Here's a PHP script I've written which should be able to generate a SQL dump for completely re-inserting an entire book. We need to run it on a Drupal.org 5 database, which will generate a series of inserts we can run on the Drupal.org 6 site. I've also attached a sample of the SQL file that would be generated by this script. Several of the variables at the top of the file would need to be adjusted to match the current database.

(Sorry jlevy, I put this together while on a plane today so I didn't know you had already undertaken the same task).

@quicksketch:

- I am not getting how we should set up those variables exactly, especially given an ever moving D6 database.
- Your query: "SELECT b.*, n.title FROM {book} b INNER JOIN {node} n ON b.nid = n.nid WHERE parent = %d GROUP BY b.nid ORDER BY b.vid DESC" is interesting; why did you not join on vid. As said above, only two nodes fell out if you join by vid; otherwise it would be much cleaner.
- This script seems to drop all existing menu links, while as the above data demonstrated, we have several new book pages (which then will get removed from the book).

BTW, I've made this slight revision of the script, so one can run it without an actual Drupal bootstrap. Otherwise it would require a running or at least bootstrappable Drupal 5 site. This just uses direct MySQL functions. The only reason for the bootstrap was the use of the db_query() function to get data, other parts of the script lack any Drupal API use.

Run this script on the shell.drupal.org server with my creds for the backup db (in place of password is ****) and got this:

$ php book-upgrade.php.txt db1-test-vip.drupal.org gabor ***** drupal

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 77 bytes) in /home/gabor/book-upgrade.php.txt on line 33

Line 33 is where it fetches the next result record object. Eating up 128MB this quickly shows that this recursive data building is not suitable for our data. Especially, since there is no Drupal bootstrap memory added onto it, it is just plain old PHP.

- I am not getting how we should set up those variables exactly, especially given an ever moving D6 database.

I assume you mostly mean the "mlid" variable, where we need to set what the next menu item auto-increment value is from the live Drupal.org site. For this I was thinking we'd just use a high number (say 500 more than whatever the highest mlid is). Since we're running mySQL, it won't have any problem doing inserts of an auto-increment key.

- Your query: "SELECT b.*, n.title FROM {book} b INNER JOIN {node} n ON b.nid = n.nid WHERE parent = %d GROUP BY b.nid ORDER BY b.vid DESC" is interesting; why did you not join on vid. As said above, only two nodes fell out if you join by vid; otherwise it would be much cleaner.

Eh, it's a one-off script. We can clean it up if you want.

- This script seems to drop all existing menu links, while as the above data demonstrated, we have several new book pages (which then will get removed from the book).

Yes, this is true. But there is no way we're going to be able to do an effective "merge" of the new handbook order and and the old one. Especially considering a lot of these new pages might be in a hierarchy under some of the old pages. We've gotten ourselves into a funk by needing to merge in data. We'll be much better off getting this done as quickly as possible and then fixing the small number of new pages added.

Eating up 128MB this quickly shows that this recursive data building is not suitable for our data. Especially, since there is no Drupal bootstrap memory added onto it, it is just plain old PHP.

My intention was that this script be run locally, so we can increase the memory to ridiculous limits (1GB+) if necessary. The end result is a SQL dump, which we can then import directly onto drupal.org through command line or phpMyAdmin.

Good morning all,

Quicksketch, thanks for your help on this. I took a quick look at your script, and I have a few comments. First, I agree with Gabor, removing all of the menu_links isn't the best option. Also, the script assumes that the menu_name is properly set to identify which entries to delete. For about 1,490 nodes, that's not true. The links think they are in the menu for the Getting Started Guide. Some nodes just have 'book-toc-' in the menu_name field (there should be a number there, like book-toc-257).

There are several kinds of problems here:
1) nodes with the corrent book.bid, but wrong menu_links.menu_name
2) nodes with the wrong menu_links.pid (parent)
3) nodes with the wrong book.bid (these nodes have some of the above problems too)
4) the 30 nodes missing from the book table and menu_links table that Gabor printed above
5) probably some I'm forgetting (I'm not looking at my notes)

Because there are so many different types of errors, it would be difficult to identify all of the different kinds bad entires to delete before recreation. I think the best plan is a non-destructive script using UPDATE whenever possible.

My approach was to use java scripts (not to be confused with javascript) to generate sql dumps. I chose java because it's the language I am most familiar with, and I wanted to avoid php memory/timeout issues. I have working scripts for correcting 1) and 3) above. I have a java script for generating the fix for 2), but some of those parents are the 30 missing nodes, so I can't set the parent until those 30 are inserted. I'm sorry I don't have time to post them now. They are on another machine.

There is one final issue we may need to tackle. There are some fields in menu_links which have data that can be reproduced from the rest of the table. For example, has_children, and p1-p9. To set has_children, you can do a query to see if there are any mlinks that reference the link as a parent. The question is, does drupal maintain those fields, or do we need a script to set them, because most of them are wrong for this book. Anyone?

I worked on this almost all day yesterday (I didn't think it would take that long either), but I have class for most of the day today. Tomorrow I am free to work on this again, and nail down a patch. Please let me know if you want me to keep going, or if you decide another approach is better.

Thanks,

Jeff

A) Well, I am fine with gaps in mlids. I did not get that "parent book" == "the root node of the book we need to fix".
B) If it picks up all the right data, then all right.
C) Uhm, well, we need to have easily accessible backups of that then. Unlike we had in this case, where it took a few days, before we had a ready to query backup.
D) Unfortunately I have no creds to up this limit on the machine and dumping the node table is crazy big to dump for local execution. I'll try to track someone down to up this limit for the duration of this script.

Also, the script assumes that the menu_name is properly set to identify which entries to delete. For about 1,490 nodes, that's not true. The links think they are in the menu for the Getting Started Guide. Some nodes just have 'book-toc-' in the menu_name field (there should be a number there, like book-toc-257).

Perhaps we should do the deletes based on the new structure. Since we'll already have a list of the nodes we're migrating, do an individual delete for each new row we insert. We can match on the link_path easily (node/x).

D) Unfortunately I have no creds to up this limit on the machine and dumping the node table is crazy big to dump for local execution. I'll try to track someone down to up this limit for the duration of this script.

We should be able to get around this just by using ini_set(). Here's a revised script with some better docs and removal of the unnecessary print_r().

One more thing: The question is, does drupal maintain those fields, or do we need a script to set them, because most of them are wrong for this book.

Drupal sets these fields for us, but only when updating the book nodes or menu items. So yes, we need to set all of these values manually for our own inserts. See my sample inserts in #37.

I would think that PHP code would be better. How is the Java code supposed to run, on a server running PHP?

I already ran the java code on my machine against the dumps provided to generate the sql files that should be run. I only included the java files to document how the sql was created.

I like the approach in #45, but it definitely won't work the way it is currently. We need to calculate all the p1-9 and depth options as well, since every page that's been edited since the update has screwed up it's parenting and depth levels. I like that it's using UPDATES instead of inserts, though I really thought it might be easier just to start from scratch entirely.

Maybe I'm missing some of the update files? #45 mentions fixes 1-5, but the archive only contains fixes 1 and 2? Once we get these files I think it'd be worthwhile to at least give the imports a shot, since all we have to do is run an import on the SQL files jlevy has provided.

@#48 - My plan was to save setting those fields for last. Once the all of the parent fields (plid) are set correctly, we can run a script that will iterate through each menu_link in the book, and trace the parent references, building a stack, then pop the values out into p1-p9. I listed this as Fix5 above. Does that work?

@#49 - Only Fixes 1 and 2 are generated at the moment. The java for 3 and 4 are also just about done. I think I could have them all finished by today.

I have good news and bad news. The good news is, all 5 fixes listed above are complete, and they work perfectly on my test database. All of the queries I have to detect inconsistencies in the structure return 0 results after the fixes are applied.

The bad news is that the when the rows are inserted into menu_links for the missing items in fix3.sql, new mlid's are generated. The fixes after that point rely on those mlids to work. So basically, the .sql files for fix4a, fix4b, and fix5 that were generated on my test database are no good, because some of the mlids will be different.

Here are two possible options:
1) Run fix1, fix2, and fix3 on the backup database. Then, post an updated dump of the menu_links and book tables. I'll create tables from those dumps here, and run the scripts to generate fix4a, fix4b, and fix5. If we setup a time for this, I can have it done within minutes.
2) If someone with access to the test database has a java runtime setup, he/she can use that to run the scrips to generate fix4a, fix4b, and fix5.

Again, here are the descriptions of the fixes and what they do
Fix1 - This changes all of the book.bid to 257 (the right book) for nodes that were in book 257 in drupal 5.
Fix2 - This changes the menu_links.menu_name to 'book-toc-257' for entries that have 257 in the book.bid (and an incorrect menu_name). I applied #1 before generating the sql for #2, so those nodes are included.
Fix3 - insert the 30 items listed above into the book and menu_links tables
STOP HERE WHEN RUNNING SCRIPTS!
I included the ones below just so you can see what they look like. They should not be run.
Fix4a - change the plid of nodes that have the wrong parent listed.
Fix4b - change the plid of nodes that have NO parent listed (plid currently is 0). There is stupid reason why this had to come from a seperate query.
Fix5 - Iterate through all menu_links items with menu_name book-toc-257 and verify that fields weight, has_children, depth, and p1-p9 are correctly set.

Let me know what you want to do

Jeff

One more thing I forgot to mention. When I ran the the script to generate the depth and p1-9 fields, one menu_link came up with a depth of 10. Given the fact that the p fields are numbered 1-9, and this documentation, this is probably bad.

The node is node/334141 parent->node/333485 grandparent-> node/149228

Two questions arise:

1) Could this be what broke the d5->d6 upgrade script? Did Drupal 5 have a limit on menu depth?

2) What should we do with this now? Initially my plan was to assign it depth=10, and just put the first 9 levels in the p fields (the missing p10 would be a reference to itself). The more I think about it though, the more I think this is a really bad idea. Maybe the best plan is to bump the node up to be a sibling of its current parent?

Yes - bump it to be a sibling of the current parent - any more than depth 9 will just break.

Post DCDC it seems like we should do a post-mortem starting from the D% DB to see whether there are serious bus in the D6 book update function.

@#53 - Ok, I updated fix5.java to check for depth>9, and adjust the depth, plid, and p1-9 fields accordingly. I also updated fix5.sql in the version attached.

Please read #51 before applying these patches.

Gabor, quicksketch - I'll await your advice on what to do next. Just FYI, if you want to coordinate applying the patches as I mentioned above, I will have limited availability and internet access from 3/1-3/6. Also, I'm in US Eastern time.

Thanks!

#45 was deleted upon request #384136: Please remove attachment and here's it's transcript for history:

I'm posting what I worked on yesterday so I don't feel like it was wasted time. If you decide to go with a php delete/insert approach, I understand.

Fix1 - This changes all of the book.bid to 257 (the right book) for nodes that were in book 257 in drupal 5. See below for how i determined book numbers in drupal 5.

Fix2 - This changes the menu_links.menu_name to 'book-toc-257' for entries that have 257 in the book.bid (and an incorrect menu_name). I applied #1 before generating the sql for #2, so those nodes are included.

Items that still need to be done:
Fix3 - insert the 30 items listed above into the book and menu_links tables

Fix4 - change the plid of about 80 nodes that have the wrong parent listed. This needs to be done after Fix3, because a lot of these have parents included in the 30 missing items. That's probably why they got invalid parents in the first place. The java for this one is written.

Fix5 - Iterate through all menu_links items with menu_name book-toc-257 and verify that fields like has_children, depth, and p1-p9 are correctly set. If we want to, we could also compare weight to drupal 5 weight and alter if we need to.

I'm including the sql to run, in addition to the java I used to generate them. A few notes about the queries:
- d5book - the book table from drupal 5
- d5book.bid_calc - Because the d5 table didn't have book id, I added a field to the d5 table (bid_calc), and ran a script to calculate it by tracing the parent references to the root node of the book.
- d5bookv - A view that selects from the d5book table only those records with the latest vid.

Because I already generated the SQL, these won't be running based on real-time queries, but I don't think this matters much. These scripts only operate on inconsistent database entries, and these happened during the upgrade. If people are currently inserting valid entries, they wouldn't come up in these queries anyway.

If fixes 1 and 2 are applied to a testdb, I think they would make a huge difference even by themselves.

If this isn't the way people want to do it, please tell me and I'll stop spending time on it.

subscribe

I'm still awaiting guidance from someone with drupal.org creds. Since it's been almost a week since I generated the fixes above, it might be a good idea to regenerate them from a more recent db dump. Please let me know what we need to do.

Thanks,

Jeff

I'm still awaiting guidance from someone with drupal.org creds. Since it's been almost a week since I generated the fixes above, it might be a good idea to regenerate them from a more recent db dump. Please let me know what we need to do.

Thanks,

Jeff

I've just noticed that there are now problems in the Getting Started book, all seemingly related to the contributed modules documentation which, if I recall correctly, used to be (should be?) in the BtB book.

If you go to the Getting Started guide Contributed Modules section, you'll see some weirdness happening in the left block. From the higher level there's just a Contributed Modules section but as soon as you click on it, the left block displays another section (comparison of contributed modules) as top level ... even though it should be a subsection of the Contributed Modules.

I assume all this would be fixed by jlevy's work (thanks for doing that by the way) but I just wanted to note that it's not all happening in BtB as we thought earlier.

Also I wanted to note that this issue is starting to get really gnarly ... we've got 50 or 60 people in the Drupalcon doc sprint today making all sorts of changes and it's getting really hard to understand or explain what's going on.

jlevy-

Emailed you about this. We are working on fixing this today. If you could email back, we can get this worked out. Thanks :)

-N

@#59 - Yes, I've noticed that the some of the contributed module pages are showing up in the getting started book. I believe this is because the menu links for some of the pages in that section are set to show in the wrong menu. The fixes I'm proposing would restore the pages to the correct book. I don't believe there is anything wrong with the structure of the getting started book (besides the rouge pages from BtB). Does anyone notice pages missing from, or rouge pages from other books (other than BtB) showing up in getting started?

Also, sorry for the duplicate comment up there (57, 58). That was a mistake, not harassment :)

if i try to edit the book doc page i have made, i can see that the parent is well appearing in the book hierarchy select list, but not showing in the book page

If there are "clusters" of pages (i.e a page and children) it seems like we might want to use the API/helper functions

e.g. see: http://api.drupal.org/api/function/book_nodeapi/6

Something is *really* broken though - for example, this node:

http://drupal.org/node/109782

The "Outline" tab indicates that its parent is: a page called "Developer Modules" under the "Beyond the Basics" . I can't even find this page.

but the breadcrumb indicates the parent is: http://drupal.org/handbook/customization/howto

I found the missing page: http://drupal.org/node/206773 - saving the same settings for the book outline it already had fixed some problems.

This is also very weird: http://drupal.org/node/205585

In the menu at left and in the breadcrumb, the parenting seems right. If I go up one level to http://drupal.org/node/206773, it's not listed among the children, however, at the bottom of that node. Does the d.o theme limit the number of children that are displayed?

After further testing, it seems that for many pages in "Beyond the basics" things seem to fix themselves if I load them in the "Outline" tab and just re-save them.

This suggests to me that possibly the p1-p9 columns are generally borked. However, those columns are actually redundant with the plid column, so we could run a script that just recurses through the hierarchy and sets these columns (and maybe depth and has_children) based on the plid values encountered.

After further testing, it seems that for many pages in "Beyond the basics" things seem to fix themselves if I load them in the "Outline" tab and just re-save them.

I tried doing that with my modules and a couple pages ended up in the right spot but the rest ended up attached to other random(?) pages. :(

Michelle

Thanks everyone for your help on this. I have done quite a bit (hours and hours) of analysis on the current state of the menu_links and book tables. pwolanin, you are correct that the p1-p9 fields are wacky for a lot of nodes; however, the fix isn't that simple. There are other problems as well, including nodes with the wrong plid.

If you take a look at comment #51, I outlined my proposed fix. Once everything else is fixed (including correct plids), I would do exactly what you proposed: recurse through the table and use plid to set fields like p1-p9, depth, and has_children.

I'm in contact with Narayan Newton via email, and we are trying to setup a time to get the fixes committed soon. Thanks for your patience everyone!

Well, it looks as though there is brokenness throughout the book hierarchy, so i would run code to fix p1-p9, etc for all items in all books

"Getting Started" was working a couple days ago when I re-did my entire documentation book. Now, it's screwed up again. Should everyone wait until this problem is totally resolved before trying to get out documentation straight?

What I did is gather links to all my child pages and put them in the text of the main page, which is linked to from the project page. Not ideal if you have a deeply nested book but works for those of us with smaller docs.

Michelle

I would recommend waiting until the problem is resolved before trying to fix documentation manually. The fix will restore the BtB book structure that existed in Drupal 5, so changes made to the structure of that book (or to the placement of nodes that should be in that book) will be lost.

Subscribe

Oh dear, 3 weeks with mangled documentation. I'm currently looking for the OG pages but can't find them. Is there anything anyone can do to help? ... On second thoughts, having now read some of the above, this is a bit over my head ..! :-P

This is very frustrating. Please assist jlevy to close this issue asap. Thank you.

Until this issue is fixed, I'm wondering if it's possible to programmatically add some kind of notice to the top of all the pages in the BtB book? I'm seeing from the issue queue and the recent updates page that there are many people who don't realize what's happening and are trying to fix the structure manually or to add and edit pages. I would guess that there are many more people who are just looking for information who are getting a horrible impression of the Drupal docs.

There is a notice on the BtB landing page but it's obviously not enough.

Subscribing. I have the same problem in Getting start book.

Pro tem I am using http://scratch.drupal.org/handbook/customization and http://scratch.drupal.org/getting-started which are missing info added after Aug 2008 but better than nothing.

FOR NOW, I RECOMMEND PEOPLE USE THE ISSUE QUEUES FOR ANY DOCUMENTATION ADDITIONS OR CHANGES.

@drupal infrastructure team: do you have any recommendations or announcements to help guide users/developers for the time being?

Hi all,

Not sure if I add anything useful, but I wanted to create a book page under "Beyond the basics » Developer modules" and I was unable to find "Developer modules" under "Beyond the basics" in the "Book outline" section of the page, so I figured something wrong was going on and decided to delay my attempt to create that book page.

It seems that the book outline selectors don't get the correct structure of the book pages.

I hope that helps.

PS: I cannot send this comment if I do not select a "valid version", so I do. Sorry if that's not correct, but it's the only way to submit my comment.

@markus_petrux: To get your documentation on to d.o and give users/reviewers a chance to start using it, I recommend you add it to the module's issue queue and categorize under documentation.

@mitchell: I'm not sure your suggestion will help place the page in the handbooks under "Beyond the basics » Developer modules", or it will? :-o

What I want to document is the checkall module API, which is pretty simple anyway, and I found 2 places where this could be documented: a) "Beyond the basics » Developer modules" and b) Using the APIs available through contributed modules, but it looks like b) is not so updated, and I don't wish to add more clutter to the docs, so better ask first, I think.

hmm... another option I have is just wait to see how this evolved. I don't belive my little doc is so needed, so I think I can wait.

I think what mitchell is suggesting is that you refrain from trying to add content to the Beyond the Basics book until this issue is resolved. Instead, you can use the issue queue temporarily. I agree that this is the best plan. The structure of the book is not only wrong, it is inconsistent with itself. Trying to add or move pages will probably not work. Please avoid doing this.

I do apologize that this has taken so much time. Narayan Newton and I have been working on generating and applying the fixes to a test sever, and we should be done very soon (tomorrow I hope!).

Thanks for the update jlevy. :)

Thanks for the clarification, and also thanks for the hard work. :-D

Subscribing so I can tell when things are working again. Thanks for the efforts!

Subscribing so I stop putting pages where they don't belong. ;)

The fixes have been applied to d6.drupal.org. Please take a look at the docs there and post anything that still seems wrong or broken in both Beyond the Basics and Getting Started.

Direct Links:
BtB: http://d6.drupal.org/handbook/customization
GS: http://d6.drupal.org/getting-started

Thanks,

Jeff

getting-started looks ok to me

I checked a couple of pages in the BtB book and it looked ok to me, except that the main level seems to have pages that don't belong to that part, but I guess this has been caused by updates when the book hierarchy was not correct.

looks much better. Had a click around and couldn't find anything amiss. Good work!

My APK docs were partially under CCK for some reason. I've got them all back in the right spot but http://drupal.org/node/356711 was under one of my pages and isn't mine. I don't know where it goes.

Michelle

Great, look good for me.

I've spent some time in B-t-B and it looks great... Even if there are little things that aren't caught we're so much better off with this patch and I'll happily sign up to help fix any little quirks if we find them. Great work! Let's get this applied!

Patch...? Oh, I'm an idiot, sorry. I didn't catch that this hadn't been fixed here. Well, something got fixed because I was able to put my pages right where they belong without a bit of trouble. And most of my docs were in the right place and looked fine. Just APK was a bit messed up. Whoops. :)

Michelle

I checked out a number of areas of the handbook I'm familiar enough with to know whether they're the way they're supposed to be or not, and looks good (on d6.drupal.org).

I checked out the CCK and Date and Calendar sections (lots of pages) and I didn't follow every link, but it is a VAST improvement. Thanks!

I just checked the contributed module Rules's documentation: http://drupal.org/node/298476

The hierarchy is not listed under the section that is the module title on the left hand side, and the book navigation is not listed below the content area.

All of the contributed module documentation that I am familiar enough with to check appears fine on the patched staging site (and I didn't notice any problems in the book navigation or related blocks on the pages I examined).

@mitchell did you check http://drupal.org/node/298476 or the proposed corrected page at http://d6.drupal.org/node/298476?

lol. I didn't keep up with this enough to know that there was a staging site for this patch. The structure works, but the content isn't updated. So, my next question is once the fixed version goes live, will we be able to review recent changes and update the old, now working handbook content?

@mitchell - d6.drupal.org was the staging site used when d.o upgraded to drupal 6, so the content hasn't been updated since then (I think...). The patch makes changes to the book and menu structures only; it does not touch the node table. When the patch is applied to production, the actual content text will remain the same as it is now.

Does that answer your question?

@jlevy: Sounds good to me! The only thing that seems worth testing first is with the new content, especially where people have added and moved a few things here, and there.

The print module documentation is correctly restored in d6.d.o.. Thanks for the work.. And, please, pretty please, run this on the main d.o database.. Fixing this (even if there are small bugs still around) is more important than waiting to make sure that every single inconsistency is addressed.

Subscribing... and btw, the handbooks of the modules I maintain look well in the staging site too: i18n, Messaging, Notifications.

(Please, apply the fix, It's been a hard month without handbooks...)

Thanks for the feedback everybody. The plan is to apply the patches tomorrow afternoon. As a result, book editing will probably be unavailable until this is resolved. I'll post an update as soon as we are done.

Thanks again for being patient. I'm also eager to have working handbooks again!

Update: I'm very sorry to report that we weren't able to apply the patches today. Coordinating schedules is a little difficult, but I'm hoping this will be done by the end of the week.

The patches have been applied. As far as I can tell, most things are where they should be. Also, the menu and breadcrumbs seem to be working correctly. Please take a look at the handbook and let me know if there are major issues. Minor issues should be able to fixed manually now.

There are some pages in the root of the book that shouldn't be there. My guess is they were added into the broken book since the upgrade from D5.

Also, a big thank you to Narayan Newton for working with me to test and deploy the patches!

Woohoo!! Thanks jlevy for all your tremendous work. I'm so glad to have handbooks again. I've got some catching up to do!

Looks all good to me. Hard to say whether it's 100% without comparing it to a table of contents from before, but I can't see any problems at all.

I've cleaned up the odd pages at the top level (they all seemed like new pages that people put there because there wasn't an obvious structure) so I'm going to mark this issue as fixed.

Yay!!!

Good work! Thanks!

Thanks for all your hard work! Good job!!

Heartfelt thanks for all the hard work on this! Great to have these handbooks back again.

The docs are back!! Thanks a lot!

Thanks a lot!! I know this was a nasty project :)

@jlevy, dude you totally rock! thank you so much for sticking with all of this. I know that more than just the docs team appreciates your efforts and persistence. I personally owe you a beer (or beverage of choice).

thank you, the doc is back :)

You're welcome! I'm relatively new to the Drupal community, and it was nice to be able to contribute something meaningful back for a change :)

@jlevy: pretty great way to enter the community :). Your help was greatly appreciated.

It looks like the problem that was had here is a problem still in the drupal5-6 upgrade script for books on other sites. Other than putting it back together by hand or by handcrafted script, was there ever a solution written for this that you guys know about?