Tablemanager is hung up

I have now even restarted the account and bounced Apache and httpd with no improvement. I would de-install and re-install, but I have the feeling that it has left things in the database so that I would be back in the same shape afterwards. Is there something I could do to clean up the database? At this point, I don't actually mind throwing away all existing tables if it comes to that.

After reading the code in tablemanager_admin.module ... realizing that I know know nothing about PHP, I noticed that deleting a table seemed to be done by going to a node called .../tablemanager/table_delete/table-N where N was the table number. I tried using this URL with 6, which got me a 404, and then with 5, which got me a page about deleting the table. I deleted that and ... presto ... the tablemanager admin page reappeared. Crisis averted.

But, someone might want to fix whatever caused it.

Apparently, my fix was not a total fix. Today, we started loading additional tables and even doing something fairly simple like a CSV import, we end up with a white screen. After getting the white screen, we can go to view tables and see that the load has completed, but we are getting white screened going to table manager admin. I am going to try deleting some more tables, but clearly there is a problem here.

Can you host the csv file on your site and mail me a link to it? My first thoughts are that the csv file isn't using unix newlines eg. "\n" possibly Windows type ones instead? The effect of this is that Drupal imports as one single line/ header as it can't split the file into separate rows. I'll take a look...

Pobster

Sample CSV is attached to the other thread which you have just responded to.

Would doing a cut and paste instead of an upload solve the problem?

I am a bit surprised if this is the problem since we did get some tables to load.

I used ED4Win to convert the file to Unix format and will attach it here. If we try to load that one, we get the same thing as with the other, i.e., that one does the operations for the upload, but nothing shows up in the box where one is supposed to put the CSV text. Try to save it and it says there is nothing there. If we open the file in Notepad and cut and paste it in ... where it looks like one long line with block characters at line breaks in Notepad, but looks fine when pasted ... then submit that, we get a white screen. Going to table views, we see the table uploaded, but going to admin / tablemanager, we are back to white screen.

Is there some SQL I could run to clean everything out to pre-install conditions? Maybe a fresh start is needed.

I would just delete the install and re-install it, but I expect there are artifacts left in the database.

And/Or, some SQL I could run that would tell you what might be wrong?

Drupal 6.9, if I haven't mentioned it already.

If you haven't guessed, Theresa de Valence and I are working together on this. She is content and I am doing the site setup stuff.

As predicted it's a problem with Windows, just not the one I expected... Some of your rows are corrupted, if you can open them up in an SSH client on a Linux server... If you can't... Just take my word for this...

On my server the file contains unrecognized characters like this; ** NOTE if your browser supports this character set it'll probably look fine - to me it displays as a question mark in a black diamond shape...

"Chef Who Died Saut�ing - Finkelstein & Smily, ",1

These fail import, and result in unclosed/ malformed serialized rows in the db. The above comes out like this;

a:2:{i:0;s:50:"Chef Who Died Saut

Obviously missing the rest of the string (the s:50 refers to length of the string which obviously is an awful lot shorter). On my dev version of Tablemanager the csv actually finishes import and shows just fine (but it reports that 38 rows failed import) but I'm no way releasing that just yet... You'll just have to either ensure your server supports whatever character set you're exporting with or actually specify the export character set (I don't believe you can do this in Excel?) You can do it in Open Office though... I'd recommend UTF-8 as that's probably what your db is set to as well.

Pobster

The character in question there is an e with acute accent é if that makes it though the post. I would have thought that was handled by UTF-8!

But, the big question is how do we find the offending characters ... if there are 38 lines of them.

Is it the case that when it loads it will load up to the bad spot or is it skipping rows? If it stops, we could tediously keep loading and then fixing the next row. But, if we have to compare, that could be really tedious!

Nope afraid you're missing the point... It's not the import which is the problem, it's the export - Windows and Excel use UTF-16 but PHP uses UTF-8... Of course I can change the code in Tablemanager to compensate but meh, this was only ever a rush released branch for someone who I'm quite fond of - it was never intended to be a full release. I'd suggest again, using Open Office to export or ensuring your Excel exports in some kind of compatible Unicode?

Pobster

OK, recognizing that there can be issues with the output, I still think there is something else which is the problem here. In the attached file, we have scrubbed out all the accented characters and such and I have opened the file in vi on Linux and searched through it and there is nothing there that looks in any way like a character out of 0-128 ... not even the high end of one byte. Yet, this gives us the same behavior ... it loads, but white screens after the load. One can then go see the table in table view and one can reference the table in a node and have it display properly, but table administration is then whitescreened so that one can't create any more tables until one deletes the last one.

Is there actually something wrong with this csv?

Oh, the other thing I noticed, btw, is this error repeatedly in /etc/httpd/logs/error_log

PHP Warning: PHP Startup: Unable to load dynamic library '/usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/magickwand.so' - libXt.so.6: cannot open shared object file: No such file or directory in Unknown on line 0

I am pursuing that with WestHost, but I thought I should mention it in case this happens to point to something.

FFS why are neither of you listening to me? I tell you what - I'll use Open Office and re-export your file, here find it attached. I'm not here to babysit you, I've given you the solution ... three times now? I can't understand why neither of you just went ahead and tried it.

Pobster

...And just to point out, your extension not found error is for ImageMagick (image processing), which if you're using GD2 for images won't cause you any problems whatsoever. Check what you have set in /admin/settings/image-toolkit

Pobster

OK, never mind the PHP error. WestHost has fixed it and it is not recurring.

But, I have some new evidence related to the last file. I loaded the first 425 lines by just pasting them into the body, no problem. I loaded another 100 lines adding to the first table, no problem. I tried loading another 100 lines and it whitescreened. Then I deleted that table, did it again and added 425, then 100, then 50, no problem. Then I added in 10s and it took a couple and whitescreened. I deleted that table and tried adding all 575 lines at once. whitescreen. Deleted that and add 525 lines, OK. Then I added 30 lines and white screen.

Is all of this stuff going into one record and might we be having record size issues?

FWIW, I just loaded a whole series of additional tables, various slices and dices from the same set of titles, authors, etc. from the same Excel sheet and by keeping the individual table down to 400 lines or less every one went in without issue. I took the CSV and converted to Unix format using ED4Win, but then cut and paste from the ED4Win window into the body of the Import CSV tool.

So, I would say there was a strong indication that the whitescreen issue is related to volume or row count.

See tables 22-42 here http://www.reviewsbytdev.com/tables

And what happened when you used Open Office like I'm asking you to do for the forth time now?

You're testing my patience... Especially as all the links you keep posting are access denied, so it's pointless you posting any more.

You only have to use Google to see all the problems (both ways) with Windows/ Excel -> CSV;

http://www.google.com/search?q=php+csv+utf8+windows

Here are the steps;

1. Install Open Office
2. Open csv file - don't change any settings, leave everything as default (actually as you're in Windows you may have to set the character set in OO to be UTF-8 - I don't know I don't use Windows)
3. File->Save As...(csv)->Overwrite->Keep In Current Format (everything default here, you don't have to change a thing)
4. There is no 4, import it...

Understand here that using UTF-16 isn't invalid for csv, you can resave your files in whatever programs you like as UTF-16 they won't have any problems - it's PHP which has the problem with Windows but luckily Open Office can save the day as it reads csv files AS csv and not just as files like a text editor does, so when it saves - it doesn't just re-spit out the old format, it saves into whatever character set you set it to with nice valid encoding. Yeah sure I can do all sorts of workarounds in my code but why the hell should I? There's so much crap we have to do to change things to work with Windows and its crappy software - Postfix which uses the classic line - 'broken_sasl_auth_clients = yes' to cope with Windows clients, CSS with all the hacks we have to make to get things to look the same... Blah, blah, etc, etc - I'm bored of replying now you're just wasting my time - don't bother posting here again until you've used Open Office.

Pobster

Its not that I'm not listening, it is just that I think having to download and install Open Office is not a reasonable workaround, especially since there seems to be strong evidence at this point that the issue has nothing to do with UTF-16. Quite aside from the steps I have taken to scrub out any character not in the bottom 128 ... at which point it doesn't matter what codepage one is using ... there is the simple fact that I have successfully loaded **ALL** of the data. I just can't do it if the chunk size is too large or the table gets too big. How could I do that if the problem was bad characters in the data?

I think Open Office is a fine thing, but we have our reasons for happening to want to stick with MS Office. I am very happy for everyone to use what they want, but it is just not a reasonable requirement to use this module to make use run things through Open Office. I *will* try a test load on your file, just to see, but I'm not sure what it is going to tell me since, as I say, I loaded all of the data from the files in the form we already had them.

OK, I tried loading your file using the upload feature and it whitescreened like the others. The white screen comes after hitting upload. The table, or at least some portion of it is created and can be viewed with view tables and can be referenced and displaying in a node, but the tablemanager admin is whitescreened and no other tables can be loaded until one deletes that table.

So, to recap, we did a variety of things to insure that the files only included characters under 128, i.e., they should have been fine regardless of code page. Small files load with no problem. Large ones whitescreen. Creating tables by cut and paste, I can get up somewhere in the 400-500 line range with no problem, but some place above that it whitescreens in the same fashion even though the last addition was a mere 10 lines. Conversely, if I set an upper bound of 400 lines per table, I can load tables all day with no problems.

I even believe we have created nodes including accented characters, which might still be within the 128 to 255 range of the code page. I don't dispute that there are some characters that cause problems as I have experienced this previously with the notorious smart quotes, but I think this was a small part of the issue we have been seeing, if an issue at all. In some case, special characters appear to turn into question marks on CSV output. I have seen this specifically with the copyright symbol ©, but what is in the export is just a question mark as I have confirmed using od; it is not a question mark being displayed in place of a two byte character.

Its not that I'm not listening, it is just that I think having to download and install Open Office is not a reasonable workaround ... but we have our reasons for happening to want to stick with MS Office

I'm not asking you to swap MS Office for Open Office, I'm just telling you how you as a Windows user can import tables into my module with no problems. The issue with special characters is well known (hence how comes it was my first thought) but you don't need to remove them, OO exports them correctly - that's all... Meaning you can keep those special characters in your data. The other crap with that is malformed data, when some special characters display incorrectly, they break the serialization of data into the table. The downside of this is that the csv appears then to have been successful but you WILL get a white screen when the unserialize command runs in the display function on that particular row. This is what I need to discount... This is why I needed you to use Open Office. TBH, I'm still not entirely convinced you have used it... And I certainly don't seem to be able to trust you to follow any instructions or take any advice...

there seems to be strong evidence at this point that the issue has nothing to do with UTF-16

See? That's hilarious! The csv attached earlier failed import into my own test site, white screening exactly as yours does. One Open Office open and save, and it imports fine.

...As you've now (needlessly?) removed all the special characters and you're still white screening I can finally actually accept that you're having a genuine problem. Yet... I'm now completely reluctant to help you seeing as you've been such a PIA. You seem to be under the impression that I have to help you? I really don't... You've paid nothing and I get nothing from doing this other than headaches and stress. TBH, I really couldn't care less whether this module works for you or not. From my view if you need 3000+ rows in your table then you shouldn't be using something generic anyway?...

Pobster

OK, first let me apologize ... Theresa has been very anxious to get this data published because there are a lot of people waiting to see it so it has been frustrating to have what seemed like such a good solution and yet have it keep blowing up in our face. It produced such a nice result when it did work that it was all that more disappointing that we couldn't seem to get it to work.

I think we also got off on the wrong foot with this Open Office thing. I have been well aware of the various issues in i18n, code pages, etc. since the software that I use during my regular work is among the world leaders in providing i18n solutions. I knew perfectly well that some forms of special characters could cause issues, but I was quite sure that we had cleaned out all of those. I suppose that if I had really stepped back and thought about it a bit more I might have explained myself better, but you seemed sure this was the source of the errors so I kept pushing on making sure that we had everything cleaned out. I get that, for you, using Open Office is an easy and predictable solution because you already have it installed and you know it works for the purpose. I just felt that installing an entire office suite strictly for the purpose of insuring that we had no illegal characters was a bit like using a sledge hammer on a tack. Between ED4Win, a programming editor which does a number of things like the Unix/DOS conversion and showing control characters and such and moving the files over to Linux and checking them out with vi and od, I was really quite sure that there were no characters left in there above 127 ascii. That's why I decided to try the piece by piece approach as a last effort to see if I could locate where the problem was.

Well, then when I tried the piece by piece approach it became clear after not too many experiments that I could load what I wanted with no problems as long as I kept down the size. But, if I tried loading something too big or I tried adding successively to a table so that it got too big, then it would crash quite predictably. I didn't do enough experiments to find the exact point where the problem occurs and I don't know for sure if it is line count or total characters or what. What I do know is that it has been 100% repeatable. If I limit the number of rows to something like 400, I seem to be perfectly safe. Possibly 500 is OK too, but I haven't tested that enough. Clearly something less than 600 with that row size breaks every time.

I also realize that I didn't help much by giving you links to things you didn't have authorization to see ... frankly, it just didn't occur to me that you couldn't see it. This is my first Drupal install where I am doing anything other than content, so there is a bunch of things that are new. If you would like to see how it has all turned out, this link http://www.reviewsbytdev.com/BestOf2008 will show you most of the results. Some results require one to be an authenticated user, but there is a reasonable sample there and we are quite pleased with it.

We do have additional material to put in for prior years, so we will be continuing to create new tables. For now, we are going to take the same approach of limiting the rows, posting in pieces, and doing cut and paste from an editor into the body, thus avoiding a lot of the line termination and other such issues.

My guess at this point is that we can even leave in the usual accented characters that are found in the upper half of the first byte of many code pages. If they are a problem, we have techniques for stripping them, so we should be OK.

I hope we can go forward from here in a most positive tone since I think this is a very interesting and useful module that will contribute significantly to the site.

I had a problem with whitescreening on Module administration and was pointed to http://drupal.org/node/31819 . I found that my memory was set to 20M and I bumped it to 96M because of the recommendation about images. I don't have another large table I need to test right now, but do you suppose this memory limit could have been the reason for running into a problem around 4-500 rows?

Absolutely, the recommended memory setting for Drupal is 128M so you were way below. Plus Tablemanager isn't optimized for large tables so it eats memory (should have been fine in your case though, your table wasn't that large) but the lack of memory does explain things...

Pobster

#155638