Did an import from our massive (from my pov) vBulletin db.... so far, I see only two problems, which is a pretty good number all things considered.

First: the BBcode is just displaying all the tags (with a few exceptions here and there in signatures that I haven't figured out why are working).
I've got the BBcode module and intend to keep using it. And our default filter is BBcode enabled... it's set up to filter out all html and allow BBcode and handle linebreaks and links.

So I'm not sure why it looks like it does. Tried editing a comment and reposting, but tags still stick. I think this may have something to do w/the roles that imported and they are for some reason not using the default input filter. More research to do.

Problem two: extremely slow response when I click topic links. The main forum list loads quickly as does the topic/thread list for each forum, but any link to the actual thread contents.... very slow. Any ideas what might be wrong there? This is all dev., but I think if it's dragging like that when we go live, the members will have my hide. (-:

"problem" three: got some errors at the end of the process... Not sure how concerned to be about these. They have not continued to appear, so far.

CommentFileSizeAuthor
vbmigrate_errors.pdf168.79 KBaharown07

Comments

liam mcdermott’s picture

Status: Active » Fixed

First: the BBcode is just displaying all the tags

This is almost certainly a problem with your input filters. Although, IIRC, there is some vBulletin-style BBCode that doesn't work with the Drupal BBCode module, but this is a minority. If none of your BBCode is working, it's more likely the input filters aren't set up correctly. :)

Problem two: extremely slow response when I click topic links.

Probably need to get MySQL to rebuild it's tables and indexes. Generally, threads (depending on the speed of the hosting, and how many/what modules are installed) should load quickly. I run this on the server's command line when MySQL is going slowly:

mysqlcheck -u<insert username here> -cep --auto-repair <insert drupal database name here>
mysqlcheck -u<insert username here> -op --auto-repair <insert drupal database name here>

If this doesn't help, you may need to look at optimising your MySQL installation, getting a faster host, installing an opcode cache, or removing some Drupal modules.

Now let's look at the errors:

user warning: Unknown column 'skype' in 'field list' query: INSERT IGNORE INTO sharpe5_sid512.profile_values (fid, uid, value) SELECT fid,
userid+1000000, skype FROM user, sharpe5_sid512.profile_fields WHERE name="user_skype" AND skype<>""; in /home/sharpe5/public_html
/000/includes/database.mysqli.inc on line 156.

Not important, there was no Skype column in the vBulletin user table.

user warning: Unknown column 'attachment.extension' in 'on clause' query: INSERT IGNORE INTO sharpe5_sid512.files (fid, nid, filename,
filepath, filemime, filesize) SELECT attachmentid+1000000, threadid+1000000, CONCAT((attachmentid+1000000), '_', filename),
CONCAT('files/', (attachmentid+1000000), '_', filename), TRIM(LEFT(MID(mimetype, LOCATE('Content-type:', mimetype)+LENGTH('Content-
type:'), LENGTH(mimetype)), LOCATE('"',MID(mimetype, LOCATE('Content-type:', mimetype)+LENGTH('Content-type:'),
LENGTH(mimetype)))-1)), filesize FROM attachment INNER JOIN thread ON attachment.postid=thread.firstpostid INNER JOIN
attachmenttype ON attachment.extension=attachmenttype.extension; in /home/sharpe5/public_html/000/includes/database.mysqli.inc on
line 156.

user warning: Unknown column 'attachment.extension' in 'on clause' query: INSERT IGNORE INTO sharpe5_sid512.comment_upload_files
(fid, nid, cid, filename, filepath, filemime, filesize, description, list) SELECT attachmentid+1000000, threadid+1000000,
post.postid+1000000, CONCAT((attachmentid+1000000), '_', filename), CONCAT('files/', (attachmentid+1000000), '_', filename),
TRIM(LEFT(MID(mimetype, LOCATE('Content-type:', mimetype)+LENGTH('Content-type:'), LENGTH(mimetype)), LOCATE('"',MID(mimetype,
LOCATE('Content-type:', mimetype)+LENGTH('Content-type:'), LENGTH(mimetype)))-1)), filesize, filename, attachment.visible FROM
attachment INNER JOIN post ON attachment.postid=post.postid INNER JOIN attachmenttype ON
attachment.extension=attachmenttype.extension; in /home/sharpe5/public_html/000/includes/database.mysqli.inc on line 156.

user warning: Unknown column 'filedata' in 'field list' query: SELECT attachmentid+1000000 AS uniqueid, filename, filedata, 0 AS avatar
FROM attachment UNION SELECT IF(userid=0,0,userid+1000000) AS uniqueid, filename, filedata, 1 AS avatar FROM customavatar; in
/home/sharpe5/public_html/000/includes/database.mysqli.inc on line 156.

Do you have post attachments stored in the database or on disk? My suspicion is these errors have been caused by file attachments being stored on disk instead of in the vBulletin database, I might be wrong though.

You may need to copy over file attachments manually.

user warning: Unknown column 'folder_fid' in 'field list' query: SELECT MAX(folder_fid)+1 FROM privatemsg_folder in /home/sharpe5
/public_html/000/includes/database.mysqli.inc on line 156.

This is more worrying, vbtodrupal couldn't find a folder list for private messages. Check your PMs have been copied over correctly.

These errors are very unusual: are you running a very new, or very old copy of vBulletin? What version -- of vBulletin, exactly -- are you running this import on?

aharown07’s picture

Liam, many thanks!
Very old and messy version of vB. 3.0.6. I'm pretty sure the attachments are in the db, but there are some mysterious problems there. Haven't been able to find them, but apparently a few of the attachments are very large. Even doing mysqldump of this db required the max_allowed_packet settings to bumped up several times vs. the hosting co's defaults.

I'm really not worried if we lose the attachments. Probably good riddance in most cases. I'm setting up to much more closely limit attachments on the Drupal side.

As for the PMs, can't test just now. After the vB migrate, I did a Wordpress migrate (this is part of our problem: vB and WP are sharing the old database) the forum in Drupal wouldn't load at all. So I decided to try again w/the WP migrate first.

Question: I'm getting off the thread topic here but, what does vbtodrupal do if it encounters user accounts that already exist? Will it add new accts w/some different name, overwrite old, or attempt some kind of merge?
Vast majority of our accounts are vB, but we've got a few in the wp tables also. Unfortunately, at present, there doesn't seem to be away to skip migrating the wp accounts (but it may work to just drop that table from the db before doing the migrate).

I'll be doing quite a few test runs of both migrates before I've got it all figured out, I'm sure.

aharown07’s picture

Oops. ignore this post.

liam mcdermott’s picture

I'm getting off the thread topic here but, what does vbtodrupal do if it encounters user accounts that already exist?

For the most part, vbtodrupal just sends SQL to MySQL, so it doesn't even know if there are duplicate accounts. However it does add 1,000,000 to all ids in the database, including uids, to avoid conflicts. The problem with users is that the name and mail columns must be unique. For example, if this data is already in the Drupal users table:

+-----+-----------------+-----------------------------------+
| uid | name            | mail                              |
+-----+-----------------+-----------------------------------+
|   1 | Mr Test One     | test-one@example.com              | 
|   2 | Mr Test Two     | test-two@example.com              | 
|   3 | Mr Test Three   | test-three@example.com            | 
|   4 | Mr Test Four    | test-four@example.com             | 
+-----+-----------------+-----------------------------------+

Then we try to import this from vBulletin:

+---------+-----------------+-----------------------------------+
| uid     | name            | mail                              |
+---------+-----------------+-----------------------------------+
| 1000000 | Mr Test One     | test-one@example.com              | 
+---------+-----------------+-----------------------------------+

The import of Mr Test One will fail. Furthermore, any topics/posts written by Mr Test One will be imported, but with the uid set to 1000000; as the import of that user failed, Drupal will not be able to attach the post to a user and the post will 'disappear'.

Note: your mileage on the above information may vary, it's based on my own limited experience, which I don't have time to test!

aharown07’s picture

Very helpful info. I believe what I can do is just rename the existing accts before I import vB. There are not very many and I can just paste a prefix of some sort onto each. Better safe than sorry there I think, so worth the trouble.

liam mcdermott’s picture

I believe what I can do is just rename the existing accts before I import vB.

Sounds like a good idea. You could use a SQL query like this to update all the Drupal users:

UPDATE users SET name = CONCAT('prefix_', name);

And for the e-mail address:

UPDATE users SET name = CONCAT('prefix_', mail);
aharown07’s picture

Thanks! the sql very helpful also

aharown07’s picture

An update for those whom it might interest:
Solved the bbcode problem: it did have to do with the order of input filters I'm pretty sure.
What worked:
A default filter set up with these four filters in this order:
HTML filter (I have it config'd to filter out all html tags, but this is probably not necessary)
Quote filter
BBCode
Line breack converter
URL filter

We still have some problems... some threads break the css for the page for unknown reasons. Also, because the importer runs a bbcode conversion, the quotes css will be bb-quote etc. and not the css from the Quotes module.
(As configured above, all new comments in the forums apply the Quotes module css even when users manually insert quote bbcode).

Result: haven't been able to get the imported quoted stuff to look exactly like I'd like (I find the bbcode css implementation a bit confusing!). But it looks good.

SPEED: ran the mysql suggestions above and loads are still rather slow. seems significantly faster when I disable Advanced Forum... hoping to find a solution that allows us to keep Adv Forum. I like it.
Also looking into cache options.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.

aharown07’s picture

Note on the speed problem: disabling the display of number of new posts (as recommended in the Adv Forum docs somewhere) made a big difference there.