Upgrade UI

What I've got so far...

Some refinements - it now does requirements checking and does not present migrations which fail, and it writes the database connection to settings.php.

Here's a thought, dovetailing also with some of our dependency/requirements discussion... Add an optional module key to both the source and destination. checkRequirements() would take that as requiring the named module to be enabled, and the UI would present the list of distinct modules whose data is being migrated.

The requirements side should be a separate issue...

Thinking about how we would implement separate Import Configuration and Import Content buttons, a fundamental sticking point is, how will the UI know which migrations are considered "configuration" and which are "content"? A first guess would be keying off a destination plugin of "config", but of course some configuration is in entities. So, should we have an explicit flag in the migration config? Any other thoughts?

Here's a screenshot of where I am so far on the confirmation page - listing modules rather than each individual migration, with a note where there's a module that needs to be enabled.

Yes, agreed that the core upgrade path is at most a 3-click operation - submit the source credentials (using the same database form as the core install), import the configuration, import the content.

Frankly, I'm less interested in this than in the full contrib UI - my primary concern at this point is that the core infrastructure provides all the information and capability necessary to cleanly support the full UI in contrib, even if not all of it will be used in the core upgrade path. So, I'm trying to get going with that sooner rather than later to find the gaps - I actually began attempting it on Friday afternoon and was immediately blocked by #2202475: Registering/instantiating migrations.

Bojhan, I'm on IRC for the next few hours if you want to chat, as well as regular EST working hours (at least) tomorrow.

chx: As for identifying configuration vs. content with hard-code manifests - what about contrib modules? They will be providing config files for their own configuration and content, how will the core upgrade know which is which? I think we need to indicate that in the migration config files - something like type: configuration vs. type: content.

New patch. This is the bare minimum UI - single button configuration and content migrations. Unfortunately the content migration dies, but config looks pretty good. Screenshots to follow.

The full screens will just add menus and whitespace, but I can do that with my next iteration.

There is no finished page - for one thing, at the moment it crashes before finishing;). I'm not sure what the finished page should look like - certainly not a list of migrations. Just a "Congratulations, you've imported your Drupal 6 site, go to admin/content to review!"?

We need to think about restartability and "continuous migration" (i.e., doing an initial migration and a delta later). I'm thinking that:

1. While we're hardcoding the "migrate" connection, if it already exists skip the credentials screen and go (maybe) to the configuration button screen. But, this would preclude ever changing the credentials (without editing settings.php) or importing a second site (well, the latter isn't a use case we're looking to support in core anyway...).
2. If we've already run the configuration import, skip that and go straight to the content button screen.
3. If we've already run a content import, on the content button screen indicate that rerunning the content migration will import any new or changed content from the legacy site.

Thoughts?

Sorry, afraid I haven't had a chance to make much progress here, and I'm off for the next two weeks. If anyone wants to continue the work on this patch they're welcome.

A hui hou kakou!

Quick reroll to make migrate_drupal.module apply, and add menu_name admin to hook_menu_link_defaults (which makes drush cr happier). Doesn't actually import anything at the moment - @bdone, what's the status in terms of manifests?

Here's a new patch that mostly works (in the sense of running all the way through). There are a couple of separate fixes to make for it to actually run, I'll add those separately (dealing with missing profile_values table, and url_alias migration just plain broke). #2201699: Theme migration is still an issue - running the configuration step unthemes the site. Using my personal site, the configuration does appear to migrate successfully, nothing blatantly wrong there (other than creating a second Tags vocabulary). As for the content migration - the tags are migrated, but no users or nodes yet.

Note we're at the 30% stage here - looking for feedback on the broadest aspects of the UI. Like, where should it go at the end - certainly not back to the upgrade page as it is now, but maybe to admin/content? admin/config/system? Front page?

Just as an FYI, I didn't really do much UI-wise here, this round was mostly about making it work functionally using the manifests added to migrate_drupal.

@Bojhan: Going back to this comment:

I still this should be outside of the normal interface, and actually go into the same interface we use for maintenance. I'd like this to be part of the patches.

You're saying that the upgrade pages (which are under /admin/config/system) should be unthemed, correct? I'll need to research how to do that...

@Bojhan: It doesn't seem possible with the current D8 API to have the upgrade form be a normal page at admin/config/system and give it maintenance theming with no toolbar. I'm thinking it probably should follow the pattern of update.php - the address would be upgrade.php, and it would be a standalone file outside of normal routing. I'll start working on that and see how it goes - I'll post the latest stable version of what I've got at the moment for safekeeping in case that avenue doesn't work out.

Pointer to relevant core issue: #2218039: Render the maintenance/install page like any other HTML page.

Oops.

Checkpoint of the UI work. Added fields to support file migration on the initial form (although they're not yet used), and better feedback on what's happening. Content migration dies though, not sure why.

Sometimes it's good to sleep on things;) - I did start making an "upgrade.php" version of "update.php", but I woke up this morning and immediately realized that's dumb. The reason update.php is a special snowflake is that it can't count on pretty much any module but system being functional pre-update, so it needs to run without loading any (other) modules. Migration, on the other hand, is running in a fully-functional environment, and indeed needs all modules active to be able to save data. So, it should run in a normal page context - I'm going back to that approach.

I'm postponing attempts to use the maintenance theme and remove the toolbar until #2218039: Render the maintenance/install page like any other HTML page lands - it's generalizing stuff around the maintenance theme for installation and updating, it probably won't give us a direct solution but it should take a step in that direction and be a better basis for offering a patch enabling us to do what we need here. I'll focus for now on what happens within the form area of the page.

OK, looking step-by-step at the current state of the UI POC, keeping in mind we're at the 30% stage. Let's work out the high-level workflow, and what specific information we should (and shouldn't) present, then we can deal with details of layout etc. Note: it would be GREAT if interested parties could have a little brainstorming session at NYC Camp to discuss ideas around the UI, it would be something non-developers could (and should) get involved with - if this happens let me know and I'll see if I can join remotely.

First off, to what extent is this functionality to be presented as a one-time major version upgrade, versus a more general import from Drupal 6 or Drupal 7? In the former case, I don't think it should be a permanent part of the admin configuration UI, but perhaps at a /upgrade URI one needs to manually invoke (as they do with /update.php). In the latter case, then this should probably be renamed back to Migration (see bojhan's original screenshot).

The central piece here is a clone of install.php's database selector, for getting the pointer to the source database. The text fields at the top are new to this iteration of the UI - the database alone doesn't tell us how to find the files. Given the domain of the source site, we can find and scrape them (in theory - this is not yet implemented).

Private files, of course, we can't - those will have to be manually copied to a suitable place before beginning the migration process, and a separate pointer provided for those. Private files are relatively rare, though - maybe we could wait until the next screen (after opening and poking around the source database) and only prompt for them if they were configured in the source site.

So, what more (if anything) should we say here?

Progress!

...

So, here's where we need some thought on what information needs to be provided on what was just done. At the moment we've got a line for each D6 config file fed through the process - probably too much detail, but I'm not sure how we would consolidate it. Also note the message "Migration d6_contact_category did not meet the requirements". This is generated when a call to checkRequirements() fails, but we have no idea what the actual failing requirement was - we need a way to bubble that up. In this (and the other cases on this page), the issue is that the corresponding Drupal 6 module is not enabled, thus there is no data to migrate - so ideally we don't want to say anything. To do that we would need to bubble up the distinction between an OK "failure" like this, and a true failure which would require attention. Or, we could try to preprocess, run checkRequirements on each migration and silently skip any that fail...

Another thing that would be nice would be to report counts where relevant (more important for content below) - this wouldn't make sense for a lot of low-level of configuration, but the numbers of menus, vocabularies, node types etc. imported would provide a bit of a sanity check.

Should we encourage the administrator to review the imported configuration here before proceeding ("please visit admin/structure/types"...)?

Apart from the reports from the configuration step, anything else we want to say about the upcoming content step here?

...

File migration is not fully implemented (specifically, the part where we figure out where to migrate them from), hence the errors for now.

As with configuration, we need to figure out the right feedback to give here on what just happened.

We're landing on admin/config/system at the moment - is there a better landing page, like admin/content? Or (probably) should we stay on admin/config/system/upgrade for this page?

We should probably have links to typical places you'll want to go to review the migration results - admin/content, admin/structure/types, admin/people...

New patch for anyone who wants to try it out - you can also pull the ui_poc feature branch.

OK, let's talk about logging - what goes in the log? Do we just save a list of messages like those being displayed here, and dump them on a log page linked from the result pages? Maybe we can just use watchdog if that's the case?

I merged @ultimike's changes into the ui_poc branch, thanks.

It's going to be difficult to keep track of particular elements of the UI we're discussing here, I am creating more focused subissues.

Let's set a threshold for committing the basic UI to the sandbox, then we can more easily refine specific elements of it in their own issues - let's say, when the content migrations actually basically work.

Friday I reached a major milestone - I completed a full migration of my Drupal 6 site to my Drupal 8 development environment using the work in this issue. Of course, there are plenty of details left to work out, but right now the attached patch (with the addition of the branch at #2244555: Use copy() directly instead of file_unmanaged_copy() for the file migration to work) does support a basic D6->D8 migration through the UI.

Assuming we're agreed on the general shape of this UI, I'd like to see this committed to the sandbox soon - it will be much easier to work on refining individual details of the upgrade process working against the main branch rather than keep growing this big patch.

Looking at the screens step by step:

Note that private files aren't supported yet, let's do that as a followup issue once the main UI is committed.

[ui_poc-2.jpg and ui_poc-3.jpg aren't embedded here - they haven't really changed since last time, but they're attached if you really want to look]

Summary info, and a link to the log of operations performed.

The operation log for the configuration imports, stored in dblog. Let's discuss how to improve this in #2242611: Communicating migration results.

[ui_poc-6.jpg attached but not embedded - no change in how the batch API looks]

Like the install process, we land at the home page. We see migrated content present...

... and a little farther down the home page, we see a migrated node with an embedded image.

The operation log, after the content imports.

For this morning's run I used the ui_poc branch plus the patch at #2244555: Use copy() directly instead of file_unmanaged_copy() (the latter just makes the file migration work, shouldn't affect the presence of the UI). The patch is the diff between 8.x and ui_poc, so applying that patch shouldn't make a difference. But, let me try it in a fresh environment...

These are the steps I followed to build a fresh test environment to test the UI, it ended up going as smoothly as it had this morning for me.

git clone --branch 8.x http://git.drupal.org/sandbox/chx/2105305.git imp2
cd imp2
git apply -v ~/imp-upgrade_ui-2200379-48.patch
git apply -v ~/tmp/imp-use-copy-2244555-1.patch
ln -s default sites/imp2.local
# Add imp2.local to /etc/hosts, Apache vhosts
drush si --account-name=admin --account-pass=admin --db-url=mysql://user:pass@127.0.0.1:33067/imp2 -y
drush en -y migrate,migrate_drupal
# Login at http://imp2.local:8083/user
# Go to http://imp2.local:8083/upgrade

Now, that's migrating from my legacy D6 site - your mileage of course will vary according to what you have on your site. In particular, I see an error involving custom blocks in your comment - I don't have any custom blocks, so I wouldn't see any such errors. This almost certainly would have nothing to do with the UI - you might test this specifically at #2221815: Manual Testing (D6->D8): Custom Blocks and report any problems as a child issue to that one.

@awasson: A general thumbs-up is definitely good here! If you have any specific feedback relative to #2242605: Context for the upgrade form or #2242611: Communicating migration results, please add it to those issues. Apart from that, any problems you find that don't already have open issues in the queue, please open a new issue.

Thanks for running through this! It's always good to here something works outside one's own box...

OK, the last patch is committed to 8.x. Any specific problems that are found, or suggestions on changes to the workflow, should now be distinct issues.