Split large imports / exports into multiple requests
| Project: | Category |
| Version: | 6.x-2.0-beta3 |
| Component: | Category_legacy |
| Category: | feature request |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | active |
Jump to:
At the moment, category_legacy performs an import or export on all specified elements, and the elements within them, in a single request. This is not an ideal setup. The main problem with it is that when a large number of elements are involved, the request is likely to time out, and the import / export to terminate abruptly in the middle of processing. This can be worked around by increasing the maximum response time of your PHP setup, but many users are either unable or unwilling to do this, and rightly so. Another (related) problem is that with so much happening in a single request, there is a high risk of something going wrong.
The import / export process needs to be altered, in such a way that it is performed in multiple requests, with a few elements getting processed on each request. The way I envision it at the moment, is that each request will be made via AJAX, and that the current progress of the import / export will be visible through an implementation of Drupal's new AJAX progressbar widget. However, the request to process more elements, and the request for the current progress, will probably need to be two separate AJAX routines.
I'm thinking that the best way to do it, would simply be to have the whole thing degrade back to its current, single-request model, if the necessary JS support is not present. The number of elements to process per request will also need to be configurable on the 'admin -> settings -> category' page, although I'll need to work out an optimum value to put as the default (this is a figure that only tests will tell).
#1
Also related to this feature is the ability to recover from imports / exports that were terminated before they could finish. This would involve setting a flag somewhere (probably just in the variables table) to indicate that an import / export has begun, and then removing that flag when it finishes. If the user accesses the relevant category_legacy page, and any flags are set, the user then gets alerted that there are unfinished imports / exports that can be recovered from and resumed.
#2
Legacy import times out on a large site, unless I turn off the create menu item for that container. I agree that splitting the task up somehow, or perhaps using cron, would help here.
#3
#4
However old this issue is, we still need to have it fixed. Now, this falls into the 6.x version, where category_legacy still needs to be ported: #370631: Port category_legacy to D6
Drupal 6 core made this much, MUCH simpler for us, due to presence of Batch API (which provides all the necessary framework), so we only just need to split the actual processing into multiple function calls. Batch API even provides a storage, that we can use to keep our data during the process ($context['sandbox']).