Building to bad hierarchy category and themes (and other phpbb 3 bugs)

so, if I am reading that correctly, the module is setting the wrong parent? (Are you using the latest (August 2) release of HEAD?)

As a short term fix, you can manually correct that via direct access to the database.

Longer term, i need to find out why. can you give me a list of what the data should be and what it is? (testing on my end does not reproduce the problem - but my setup is just a dummy set up with some forums/categories set up but nothing major.)

http://drupal.org/node/96005 - but make suer you read the project page as HEAD requires phpass, which has a bug in it (there is an issue for that where I supply a hacked up solution).

Once you test, can you make a proper report of it working/not - what needs changing? i need some reports before I make a stable release.

Hi there.

I have tried to make an import and I think my issue is related to this one. The import went ok without any error messages. When I checked the forums after there where only two forum containers with no sub forums inside. So it seems like the are some issues with forum hierarchy.

I tried to import the following:

-Forum container 1
-- Forum 1
-- Forum 2
-- Forum 3
-Forum container 2
-- Forum 1
-- Forum 2
-- Forum 3

The result:

-Forum container 1
-Forum container 2

The latter is how the Forums vocab looks like after import. I just did notice that the forum seem to exist. When I go to /forum/6 for example it's all there and everything seems correct. So the problem is that the sub containers are not listed.

Additionally there seems to be a problem when converting from bbcode to html, I noticed an image tag and an a tag was cut off.

Another things is that the forum admin (user 2) became user 1 in Drupal, which is not what I would have preferred.

This module is rather important for me and I will do all I can to help out. I really hope these issues can be solved.

Also, is there some easy way to 'undo'/redo the import?

Finally I want to thank you for a fantastic module!

Technical specs:
XAMPP (Apache 2)
Drupal 6.4
phpBB2Drupal (HEAD (aug. 2))
All required modules installed.

Can you try reverting this change to see if it makes a difference:

http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/phpbb2drupa...

(basically, remove "WHERE forum_type <> 2" from the query in phpbb2drupal.module file around line 460.)

That change was made to not import forums that were only links, but I can see how it may have unintended side effects - cases where the links are parents of either containers or other forums.

If you want to fix this in an already imported forum, you can manually modify the hierarchy in the taxonomy table(s) - all you need to do is set the right parent.

Another things is that the forum admin (user 2) became user 1 in Drupal, which is not what I would have preferred.

This is selected on the settings page - if you want it to map to a different user, create a new user in the normal way, and then on the settings page choose to map to the different user.

Additionally there seems to be a problem when converting from bbcode to html, I noticed an image tag and an a tag was cut off.

I will be honest and admit most of my test and real imports have mostly involved keeping the bbcode - I have tested otherwise but not too much as things seemed to be ok. Did this happen in all cases or only some cases? do you have those tags in the input format you were converting to?

As for easy reset - not with this module. The settings can be deleted, but that would still leave the imported data there.

A workaround may be try to use the demo module: http://drupal.org/project/demo but I have never tried it.

oh, and what database connection type? (mysql/mysqli/pgsql?)

The database is a regular mysql with the default connection. I had the phpBB tables in the same database, mostly because the option for another database didn't work. If I remember correctly the message said something like "table doesn't exist phpbb_db.drupal_system".

I missed the user mapping somehow, great feature, I'll try that.

You were completely right about the hierarchy. It was an easy fix. In the term_hierarchy I just changed parent value of the tid to the correct one. After that change the forum popped up and got listed correctly.

For the conversion. I think it's in the sig only it was cut off, and it seems to be after 256 characters. The user_sig field is a mediumblob field so it should work, must be an issue with the code. I think I will stick with the BBCode anyways, but to clearify; I didn't change any settings in the bbcode filter as recommended. The input format was set to Filter HTML with BBCode enabled. The a tag was allowed, the img tag was not.

Each empty signature did get an opening and closing p tag as well.

I will make another try and tell you how thing goes.

I have now tried the import on a fresh install with just the required modules but the hierarchy problem is still the present. I did remove the "WHERE forum_type <> 2" part from the sql query on line 460 but it didn't seem to make a difference.

Another thing; The smilies have the following code:

<!-- s:shock: --><img src="{SMILIES_PATH}/icon_eek.gif" alt=":shock:" title="Shocked" /><!-- s:shock: -->

It's the standard phpbb code but it would be nice if it was formatted in some other way. I really don't know what could be the best option for this, but something should be done about this during the import. Perhaps another module could take care of a part of this?

You were completely right about the hierarchy. It was an easy fix. In the term_hierarchy I just changed parent value of the tid to the correct one. After that change the forum popped up and got listed correctly.

Do you remember what the parent was set to before you fixed this?

I have committed a fix that *may* fix the import of the forums. Needs testing.

For the conversion. I think it's in the sig only it was cut off, and it seems to be after 256 characters. The user_sig field is a mediumblob field so it should work, must be an issue with the code.

yeah I need to fix this. current line 396 of the module is 'signature' => substr($signature, 0, 255). Do I need to trim the signature at all?

Is that the smileys code before or after import? or both? I thought phpbb had smileys similar to how they are done by smileys.module for drupal (:s :P :D :) etc etc) [Another question - is this for all smileys or just those in a signature? if it is just the signature, try modifying line 370 of the module.]

Do you remember what the parent was set to before you fixed this?

The parent ids started with 85 with a increment of 3 (85, 88, 91 ...).

Do I need to trim the signature at all?

Not sure it's necessary. Should check how Drupal does it.

Is that the smileys code before or after import? or both?

The code above is from the sql export of phpbb. When I left the bbcode 'as is' the code looked like that after import as well, same for all fields, and all smileys (except for the obvious simley-code.

Yet another thing that seems off. The last access time for imported users seems to be the time of the import, not from phpbb ('user_lastvisit' in the 'user' table I guess it would be).

Where can I get hold of the fix so I can test it?

Try here: http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/phpbb2drupa...

The signature will have to stay trimmed as I just checked and drupal has the field as a "varchar(255)" (I has assumed you meant the drupal signature size was larger than I was setting it, not the phpbb one.)

As for the smilies problem, I have one word: GAAAAAHHHH!!!

I just created a test post in phpbb3 to test them and I can see what they do as for fixing it for the import, we will need to strip out the link and all (probably in _phpbb2drupal_replace_links as that already replaces other links)- only leaving what is in the alt tag. only problem is I am not so hot with regular expressions :/

Yet another thing that seems off. The last access time for imported users seems to be the time of the import, not from phpbb ('user_lastvisit' in the 'user' table I guess it would be).

The module does import that, so I would guess that later imports of either topics or comments changes that.

Thanks for the link, I will give it a try.

My mistake about the signature. I was looking at the phpbb user_sig field and not the signature field in the Drupal table. The only alternative to fix that would be to create a new table or use another module for it. Best option is perhaps to just strip all html and links to keep it as short as possible.

After a quick research perhaps the Signature for forums is the better option?
http://drupal.org/project/signature_forum

I am rather far from being an expert, but the following snippet seems to work for the smileys. Someone should review it thought...

$html = '
<!-- s:shock: --><img src="{SMILIES_PATH}/icon_eek.gif" alt=":shock:" title="Shocked" /><!-- s:shock: -->
<!-- s:idea: --><img src="{SMILIES_PATH}/icon_idea.gif" alt=":idea:" title="Idea" /><!-- s:idea: -->
<!-- s8-) --><img src="{SMILIES_PATH}/icon_cool.gif" alt="8-)" title="Cool" /><!-- s8-) --> 
<!-- s:twisted: --><img src="{SMILIES_PATH}/icon_twisted.gif" alt=":twisted:" title="Twisted Evil" /><!-- s:twisted: -->
<!-- s:oops: --><img src="{SMILIES_PATH}/icon_redface.gif" alt=":oops:" title="Embarrassed" /><!-- s:oops: -->
<!-- s:P --><img src="{SMILIES_PATH}/icon_razz.gif" alt=":P" title="Razz" /><!-- s:P -->
';

$output = preg_replace('#<!\-\- s(.*?) \-\-><img src="\{SMILIES_PATH\}\/.*? \/><!\-\- s\1 \-\->#', "$1", $html);

print $output; 

Concerning the last visit time... exactly what does this line?

373:  if (variable_get('phpbb2drupal_regdate_us_english', 0))

The board I am importing is using British English, would that matter?

That bottom bit is nothing to worry about. Some ancient versions of phpbb probably had the option to save the time in a format other than a UNIX timestamp.

There is/was an option to select this or not on the settings page ("are all your users registered from before 1970?").

As for your other code, baptism by fire - I have committed it to HEAD and the above link should show it now (or you can wait for the next reroll of HEAD in a few hours time...)

One thing about last visit time: Are you checking for moer than one user? I know that for the first user (the one you map in the settings), the last visit time is the same as when you created that account to map to.

For others, I do remember that node_save does set its own time for the last modified... I am not sure but that may also affect the last visited time.

PS if you are also planning to import the private messages, just a warning: I/we are in the process of potentially changing the database schema for privatemsg.module where there will potentially be no update path from current (siocne there is no stable drupal 6 release out yet) - but after this change, I do hope there will be upgrade paths to any future changes.

So I don't worry about the last bit.

I was a bit tired yesterday I think. The regexp I posted didn't work at all. I have updated the previous post with a regexp that actually works! Please commit that to head instead.

A thought... Shouldn't 'access' in the 'users' table be set to $user_lastvisit instead of $user_session_time?

I am not planning on importing private messages at this point.

To go back to the initial issue, the hierarchy still don't work.

I have committed the updated regexp - Thanks!

As for the original issue, I will have to investigate what is going wrong - it should be working!

(and I have just realised that the potential "fix" that I made would not affect you as all your data is in the same database. oops.)

Can anyone see any issues with the following code?

      if ($result->parent_id > 0) {
        $result->parent_id = db_result(db_query('SELECT tid FROM {phpbb2drupal_temp_forum} WHERE forum_id = %d', $result->parent_id));
      }

That is the problem code - I have just added the db_result into the middle line, but even without that I cannot see what the problem is. It should work.

No expert in php (either) but you don't confuse the script by using the $result variable inside the while loop? You seem to overwrite it at line 465 with $result->parent_id = ....

While I am writing I can ask about the password issue as well. Is it correct that if I set phpass to md5 before import all users have to request a new password, and if I set it to phpass all works ok? I admit I haven't really looked into this.

The first one should not be a problem as there are other places where a value is also updated. I see it the same as using $count++;.

As for the passwords issue - if things work correctly, no users have to request a new password, no matter if you set the authentication to phpass or md5 (though there is a bug in the phpass module with regards to the latter... it won't work without changes to the module).

When logging in the phpass module will authenticate the password and if the site method is md5 but the user has a phpass, it will also create a md5 hash, update the usertable and delete the phpass hash, transparently from the end user.

You were right. The code you posted was the faulty one. If you change it to the following the hierarchy will be imported correctly, at least it does in my sandbox. :)

  if ($result->parent_id > 0) {
    $result->parent_id = db_query('SELECT tid FROM {phpbb2drupal_temp_forum} WHERE forum_id = %d', $result->parent_id);
    $result->parent_id = reset(mysql_fetch_assoc($result->parent_id));
  }

For the password issue I have tried both md5 and phpass and I can't login after import. Any thoughts?

eugh! mysql_feth_assoc is too db specific - would probably not work (or is not needed) by mysqli, postgres.

As for the password issue, have you modified the phpass module? and are the password hashes being stored in the correct tables?

EDIT - just put a question abhout this up on irc... no idea if I followed the correct procedure... just blurted it out in the middle of a different convo... hopefully others who know about databases will be able to help.

@ enzipher - mind explaining what your code does?

Maybe if I understood what was wrong I would be better at fixing it.

(and for the password problem, if the correct passwords are not being saved to the db, it could just be the same bug all over again...)

I thought you would get it from the code cause I am sure you are more experienced than me, but I can try to explain... The problem was that $result->parent_id contained a Resource id number eg. "Resource id 80", and when inserted into the table field it was converted to just "80" and stored as that. The code I posted grabs the value the Resource id is pointing to, which would be the actual tid. I can't say why this happen though, but I think it's because more than one value is returned somehow.

The password issue I am sure comes from phpass. I found a bug report that matched my issue. No solution found as of yet, which does not really benefit your module... I have not modified it and it does seem that the data is stored correctly, I would say it's a serious issue with the phpass module. Here it is: #302553: No more Login with phpass 6.x-1.x-dev (2008-Aug-30)

Interesting... After having fixed most issues with the phpass module and imported users still couldn't login I compared the imported passwords with the phpBB passwords. And it turns out that phpBB actually stores the passwords as md5! I changed the script to import the md5 hashes into Drupal users table and Voilà, it works. No phpass module needed. :) I don't know if it's specifik to my install, or perhaps you can choose hash method when installing phpBB? I have never done it so I don't know...

Which tables are the md5 hashes being saved in?

The code that deals with it is this:

    if ($phpbb_user['user_pass_convert']) {
      //The password has NOT been converted from md5 to phpass yet
      $result = db_query("UPDATE {users} SET pass = '%s' WHERE uid = %d", $phpbb_user['pass'], $user->uid);
    }
    elseif ($phpbb_user['pass']) {
      // Update the password:import the phpass hash into Drupal phpass mopdule table after modifying the hash identifier - phpbb3 uses $H$ while phpass uses $P$ at the start of every phpass hash
      // set the normal password as plain 'phpass'. This is needed for the phpass module as it uses the count of passwords as 'phpass' when calculatign wether or not to allow the module to be deactivated.
      $hash = substr_replace($phpbb_user['pass'],'$P', 0, 2);
      $result = db_query("UPDATE {users} SET pass = 'phpass' WHERE uid = %d", $user->uid);
      $result = db_query("INSERT INTO {user_phpass} (hash, uid) VALUES ('%s', %d)", $hash, $user->uid);
    }

where if the hash is md5, it should store it in the users table, but if it is phpass, it should use the user_phpass table.

The situation where the hash is md5 will arise when the board has been upgraded from phpbb2 to phpbb3. I thought the above code covered that - but if you still see an error, I must have missed something else too.

Howeverm my first guess would be that there is a problem with the phpass module where is is trying to use the md5 passwords instead of passing them back to drupal to authenticate.

Ah, I see. user_pass_convert has the value of 0 for all users in my case, and that's when md5 is used, it seems.

k, and are the md5 passwords put in the correct table?

are there any passwords at all in the user_phpass table?

If not, you can safely disable and remove the phpass module

Well, no. The md5 was put in the phpass table with the preceding $P minus the two first chars of the hash. In other words, the 'elseif' was executed for the md5 and not the 'if' as supposed.

Seems like we cannot rely onuser_pass_convert as it = 0 on my test system too - where I have a phpass password.

I have updated the above code in head to now read as:

    if ($phpbb_user['pass'] && strlen($phpbb_user['pass']) <> 32 && substr($phpbb_user['pass'], 0, 2) == '$H') {
      // Update the password:import the phpass hash into Drupal phpass mopdule table after modifying the hash identifier - phpbb3 uses $H$ while phpass uses $P$ at the start of every phpass hash
      // set the normal password as plain 'phpass'. This is needed for the phpass module as it uses the count of passwords as 'phpass' when calculatign wether or not to allow the module to be deactivated.
      $hash = substr_replace($phpbb_user['pass'],'$P', 0, 2);
      $result = db_query("UPDATE {users} SET pass = 'phpass' WHERE uid = %d", $user->uid);
      $result = db_query("INSERT INTO {user_phpass} (hash, uid) VALUES ('%s', %d)", $hash, $user->uid);

    }
    elseif ($phpbb_user['pass']) {
    //The password has NOT been converted from md5 to phpass yet
    $result = db_query("UPDATE {users} SET pass = '%s' WHERE uid = %d", $phpbb_user['pass'], $user->uid);
    }

ok, just tested and everything* works for me - both with database type of mysql AND database of type mysqli. Including the import of private messages (the new schema added to privatemsg.module should now be stable - there will be more added to it, but there should not be anything destructively removed.).

@enzypher - have you tested the module with a fresh checkout since comment #16, where I updated teh code to include the db_result?

*There are some theming errors on pages where the module is chacking for the existence of various tables.

Ok, I need a full run through of current HEAD - either via a CVS checkout, or by the nightly that will be released in about 3 hours.

I need to know if the bugs still exist or not (not talking about the phpass bugs, but I will also try to knock a patch up for that.)

Sorry for the absence. I tried a new import with the HEAD version (rev 1.99) and everything seems to work on my system. Well, except that users can't log in, but disabling phpass takes care of that.

Imported md5 hashes are stored in Drupals users table, and nothing in phpass table. Can't say if it works or not with secure hashes.

Good work!

I do wonder if 'access' shouldn't be $user_lastvisit instead of $user_session_time, it seems to be more correct when I tried it.

398:  'access' => $user_session_time, //<---
399:  'login' => $user_lastvisit,

hey, you have also been a lot of help getting these last few bugs fixed - some with patches, some with suggestions.

I have made the change as you requested.

Just need to wait for a working phpass module along with a beta of privatemsg and this module is ready for release.

Marking fixed.

Just a note to enzypher and others tracking this issue - the phpass module has been updated to allow logins for phpass passwords (and also for MD5 as long as you have the method set to phpass). I have also created a zip version of the module with a fix for the md5 issue on my site (linked to on the project page).

The problem with having data in a different database should also be fixed.

Also, the schema for the privatemsg module is now also stable enough to rely on, and the module ready for basic use.