I'm poking around on git7site right now, and I'm seeing issues that have lost track of their files. :( For example:

http://git7site.devdrupal.org/node/1801566

The comments show there were files attached to that issue. The {files_managed} table even knows:

mysql> SELECT * FROM file_usage WHERE type='node' AND id=1801566;
+---------+--------+------+---------+-------+
| fid     | module | type | id      | count |
+---------+--------+------+---------+-------+
| 4474862 | file   | node | 1801566 |     5 | 
| 4475228 | file   | node | 1801566 |     1 | 
+---------+--------+------+---------+-------+
2 rows in set (0.00 sec)

However, {field_data_field_issue_files} does not:

mysql> SELECT * FROM field_data_field_issue_files WHERE field_issue_files_fid IN (4474862, 4475228);
Empty set (0.00 sec)

And the resulting node doesn't show a table of files, neither on view nor in the edit widget.

No time to closely investigate now, so I'm capturing this for later.

Comments

drumm’s picture

drumm
he/him
Location NY, US
commented
Assigned: Unassigned » drumm
drumm’s picture

drumm
he/him
Location NY, US
commented

I think I've found the problem. The files themselves are migrated, they do show up in the files_managed table. The one place where comment_upload is used again is http://drupalcode.org/project/project_issue.git/blob/HEAD:/includes/proj.... This query does not capture the node's vid.

The table's data is used at http://drupalcode.org/project/project_issue.git/blob/HEAD:/includes/proj.... This query looks for t.vid IS NOT NULL.

I think vid needs to be added to the first chunk of code.

drumm’s picture

drumm
he/him
Location NY, US
commented
Title: Issue file attachments not always being migrated to D7 » Issue file attachments and body not always being migrated to D7

I went over this with bdragon in IRC. I'm probably wrong on my ideas in #2, that behavior is intentional, ProjectIssueTimelinePhaseThreeMigration should handle getting the comment files and vids together, and does for other issues.

http://git7site.devdrupal.org/node/1801566 is also missing its issue node body. We're waiting on 7.devdrupal.org to do migration so we can take a look at the project_issue_migration_timeline table. We left things at

There's two revisions stuck on the end that are unsetting everything, I guess. Timestamps are 1364954860 and 1366165006. Hmm, so those are the last two comments...

dww’s picture

dww
we/he/they
commented

Probably unrelated, but I wanted to mention it here in case it's relevant:

The auto-close comment here:

http://git7site.devdrupal.org/node/1801566

apparently removed all the issue's tags.

That didn't happen on the live site:

https://drupal.org/node/1801566#comment-7309874

Seems to be happening on all the auto-close comments (at least the 3 or 4 I spot-checked). I'm happy to move this to a separate issue. Maybe it's just weirdness with whatever the last botched migration was. But, wanted to capture this while I saw it.

Thanks,
-Derek

drumm’s picture

drumm
he/him
Location NY, US
commented

I think this is all related. The MySQL stored procedure is behaving like it is being executed multiple times simultaneously, or having amnesia. The primary symptom is that (nid, delta) is not unique in the project_issue_migration_timeline table. This could certainly cause all sorts of problems.

dww’s picture

dww
we/he/they
commented

Yikes, that sounds scary. But, I'm glad you think it's related and that whatever y'all are doing to clear up the other problems will likely fix this, too. One less issue to have to open. ;)

Thanks for dealing with this stuff! At some point I should probably try to grok the migration code, but I'm happy to not have that be my problem at all for now. :)

Cheers,
-Derek

drumm’s picture

drumm
he/him
Location NY, US
commented

ProjectIssueTimelinePhaseThreeMigration does a bunch of logic right in the database as a stored procedure. I believe it is implemented this way a speedhack so our data isn't making a round trip to PHP and back. What it does is go through the revisions and comments for each node, and assemble a complete history. All attachments are accumulated as they are seen in each individual comment; creating a complete list for the issue by the last comment.

The problem is that the rows appear to be getting processed out of order for some reason. In the migration that is running right now, rows with nid = 118 have prev_nid values of 116 and 118, which are expected, but also 119, 121, and 125. These 3 unexpected rows reset the data for that node.

The query has ORDER BY nid ASC, sort_timestamp ASC, tl ASC, but it seems like the server is getting the rows out of order as they are processed. http://dev.mysql.com/doc/refman/5.5/en/cursors.html says

Nonscrollable: Can be traversed only in one direction and cannot skip rows

so I'm not sure what's really going on. We could be exercising some 5.5.27-MariaDB bug.

drumm’s picture

drumm
he/him
Location NY, US
commented

This problem is still happening on nid 1801566, but isn't being shown as obviously because delta reset before attachments are added. They do get to the final node, but there are likely other problems.

drumm’s picture

drumm
he/him
Location NY, US
commented
Project: Drupal.org customizations » Project issue tracking
Version: 7.x-3.x-dev » 7.x-2.x-dev
Component: Drupal 7 upgrade » Issues
bdragon’s picture

bdragon commented

Still debugging this..

drumm’s picture

drumm
he/him
Location NY, US
commented
Issue tags: -project

I got the DB fixed up enough and started a new full migration.

I did make one change to the migration which won't affect this, http://drupalcode.org/project/project_issue.git/commitdiff/75490df?hp=2d.... It should be a bit faster without a 1,674,636-row filesort and the receiving table shouldn't care about the order.

drumm’s picture

drumm
he/him
Location NY, US
commented
Issue tags: +project

.

drumm’s picture

drumm
he/him
Location NY, US
commented
Title: Issue file attachments and body not always being migrated to D7 » Issue being migrated to D7 out of order

bdragon tried a couple things to get the cursor to obey the order by and we've kinda hit a wall. The next step is to rewrite it back to regular PHP.

drumm’s picture

drumm
he/him
Location NY, US
commented
Title: Issue being migrated to D7 out of order » Issue data being migrated to D7 out of order
drumm’s picture

drumm
he/him
Location NY, US
commented
Status: Active » Needs review

I went ahead and replaced the stored procedure over the weekend and it is running now. Spot checking nid 118 look okay.

drumm’s picture

drumm
he/him
Location NY, US
commented
Status: Needs review » Fixed

This looks okay on 7.devdrupal.org now.

Automatically closed -- issue fixed for 2 weeks with no activity.