Data export import

Seeking co-maintainer

Due to current work commitments I'm not able to give this module any TLC or improvements. If anyone would like to become a co-maintainer I would appreciate it - and could possibly arrange some time to go through the code and explain the structure of the module.

Overview

The purpose of this module is to be able to export nodes, taxonomy terms and users to data files and then be able to import these data files into Drupal sites. There may be many uses for this module but the main one would be that it can provide a way to export data from a live Drupal site and then to import that live data into an updated version of that site.

So, data can be exported and then imported - hence the name; Data Export Import.

Pullpush project

Please note - you may be interested in the Pullpush project which is over at Github at:

https://github.com/SSVC/pullpush/wiki

Pullpush describes how to set up a development server which can be used for Drupal projects. It also provides scripts which can be used to set up Drupal sites from scratch and to manage development and deployment.

Important note RE data integrity

The idea of this module is that when data is imported it is exactly the same and has exactly the same ID numbers as when it was exported. This is so that links/URL's/terms/etc will work in the same way on the site where the data is imported as the site which exported the data.

This means that the import of data works in a similar way to rsync WITH THE --delete SWITCH SET. I.e. If the site which exports the data has two nodes of type 'page' with ID's of 123 and 124 - then after importing the data the receiving site will have two nodes with ID's of 123 and 124. If the receiving site contained nodes of type page which had ID's of 125 and 126 (say some test nodes) then THESE WILL BE DELETED.

Also, if the the receiving site has nodes of another type (say article) which have ID's of 123 and 124 then THESE WILL BE DELETED and replaced by the imported nodes.

The important point to note here is this. The idea of this module is to export data from a site into datasets. These datasets are then imported into another site in such a way that the two sites end up with EXACTLY THE SAME DATA INCLUDING ID NUMBERS.

This also applies to taxonomy terms and to users.

As always, run backups, test on a test site first and check that the module is doing what it is needed to do.

Rules

When using this module it may be necessary to take into consideration any rule set up.

For example - a rule which sets a nodes title when it is created would probably have to be left in place. But a rule which changes values in fields of other nodes when a node is created may need to be disabled before importing a dataset.

Attached files

This module works with nodes which have files attached.

For Drupal 6 the module will handle files which are attached via the Upload module and attached via CCK fields.

For Drupal 7 this module will handle files which are attached to nodes as fields of type 'file' or type 'image'.

Moving data from Drupal 6 to Drupal 7

NB - This module has primarily been developed to move data between the same versions of Drupal - to move data between a live site and a development/beta site.

It may be possible that the taxonomy terms and users data files exported from Drupal 6 can be imported into a Drupal 7 site (or the other way around). This has not been tested yet and any feedback would be gratefully received.

Also, it may be possible to export nodes from D6 and import to D7 (or the other way around) provided the nodes do not have files attached. Again, this has not been tested yet and any feedback would be gratefully received.

Quick testing

Please note the easiest way to test this module is to use a single Drupal install.

* Install the module into a testing D6 or D7 site.
* Export current users, taxonomy terms or nodes to a dataset file.
* Alter either the users, terms or nodes.
* Go the the import tab and import the dataset file which was created.
* Check that the alterations which had been made are now reverted back to when the dataset was exported.

NB - By default the dataset files are put into sites/default/files/data_export_import - and the import functions look in this directory for file to import. To move the data from one Drupal site to another it is necessary to move the dataset files from sites/default/files/data_export_import on the site where they were exported and then put them into sites/default/files/data_export_import on the site which needs to import the files. When they are in that directory the module will find them automatically.

NB - When importing nodes it is important that the receiving site has the same content types available. The module will check that the receiving site has the same content types and fields available before carrying out the import. If the receiving site has extra fields then that's fine - it's if there are fields missing from the receiving site then the import will be halted.

Description

Moving data data between Drupal sites is not easy to achieve as Drupal stores configuration and data in the same database. Also, Drupal can be extended and customised which makes creating a standard data export/import module effectively impossible.

This module provides a framework for exporting datasets as files and also to be able to import these dataset files. That way if the module is installed on two separate Drupal instances it is possible to export data as a dataset file from one instance and import it into a different Drupal instance.

The module includes standard dataset profiles which which enable the following datasets to be exported/imported.

* Users
* Taxonomy terms
* Nodes.

When exporting a dataset is created which contains all the data. When importing the data is used in a way to make sure that the receiving Drupal instance ends up with matching data. Careful checks ensure that ID's match and that the data remains consistent.

This module uses the Druapl API to read data and produce rich data files which will contain all data needed to reproduce a particular dataset. It also uses the Drupal API to recreate data.

The primary usage for this module is envisioned to be exporting datasets from a live Drupal site and importing this live data into a new updated version of the site which developers have been working on. Once the import has completed then the newly developed site can become the new live site. This has the advantage of being able to fall back to the original live site if there is an unforseen problem between the new site and the imported live data.

Proper checks are carried out when importing datasets. For example, when node data is imported checks are carried out to ensure the receiving site has the correct content type and fields etc.

Why were no existing modules suitable

Used as part of a deployment plan

The focus of this module is to provide a tool to help with the way Drupal sites are developed, tested and deployed.

Most deployment plans push updates from dev to testing to live. This means that it is not possible to safely carry out large changes. If a site has been extensively redeveloped then there is a risk that when the changes are pushed on to live the site will break. Of course, it's possible to test on a copy of the live site as much as possible and it is possible to have a secondary copy of the live site to fall back on if the new live site breaks.

A better approach is to separately develop the new copy of the site as a beta copy - changes made by developers can be pushed up to a central beta copy of the site. This site can then be extensively developed and can be completely different from the existing live site. Using the Data Export Import module it will then be possible to export live data (users, nodes, taxonomy terms etc) from the existing live site and import that data into a copy of the beta site.

This merged site can then be set to be the new live site. The advantages are:

* The beta site can be extensively tested with a full copy of the live data before going live.
* The existing live site is untouched and can be re-enabled if the beta site has any serious issues.

Existing modules were not suitable

After very extensive research into the existing import/export modules it was found that none of them were able to export and import data in a suitable way.

Deploy module

The big underlying assumption with the deploy module is:

'Deploy 6.x-1.x assumes that you have two exact clones of you code and configuration across your staging site and your production site (expect the Deploy and Services specific stuff). You can't have anything that differs the sites from each other. Nothing.'

The Data Export Import module is designed to allow for data to be exported to dataset files which can then be imported into an different (e.g. beta) version of a site.

Also, Deploy transmits data directly via the services module. The Data Export Import module exports to dataset files which can be transferred in different ways - and data can be re-imported back in to the exporting instance which is useful for testing purposes. I.e. Developers can work on a site and make drastic changes to taxonomy terms and nodes etc and when testing is finished the original data can be re-imported from a dataset file.

Features module

The Features module is focussed on exporting/importing configuration settings. The Data Export Import module is focussed on exporting and importing content type of data such as nodes, taxonomy terms, users etc.

Stager

Focussed on pushing from staging to live. Also it is Drupal 7.x only.

Backup and Migrate

The Backup and Migrate module works at the database table level. Since there are so many interdependencies between tables it is not possible to cleanly extract datasets in a way which means they can be re-imported. The Data Export Import module works at the API level to recreate datasets exactly.

Staging

From the staging project page:

'This module do not work very well on sites that contains user generated content, on live site. So if you need this modules's name or url or what ever, let me know.'

Migrate

The primary focus of the Migrate module is the importing of data from non-Drupal sources. Also, it depends on several modules including, Dbtng, elements, autoload etc. It also relies on Drush and although Drush is an excellent tool it is not used by all developers.

The Data Export Import module is designed to export/import Drupal data and is designed to be a standalone module which just uses the Drupal API to carry out it's functions and should not need to rely on other modules. Also, the user interface will be set up as cleanly as possible to enable relatively inexperienced Drupal developers to export and import datasets.

The Migrate module also requires that module developers add extensions to their modules. The Data Export Import module does not need any other extensions to any other modules.