I am curious now that I have Drupal all set up and ready to go on my development server about best practices for moving to a live server... the theme portion of it seems pretty obvious to me, but what about modules, custom data tables they create, etc... is there a magic module out there somewhere that can assist with packaging everything up as far as data and settings go?
Are there any guidelines to follow when doing this? I can think of ways to do it myself, that involve numerous steps that have way too many variables and opportunities for error...so I figured posting here and asking flat out if there is anything to help would be a good idea. You guys have pampered me through my idiotic questions and been a very great community to deal with so far even when most of the time I got a solution to my problems for you I felt stupid for even asking...short concise answers, usually responded to with common-sense had I only thought to look in the right place or click the right link ;)
Tomorrow I plan to set up a subdomain on my development server and do 1 practice deployment to it just to see how it goes and what hiccups I might run into...
Thanks in advance!
Comments
Bosch
http://drupal.org/project/backup_migrate
i don't know about the module
i don't know about the module that i-sibbot mentions above, but i'd be skeptical of a module that claims to handle this entire process b/c it can certainly be quite complicated. there are a lot of smart people working on this, so definitely read up on related modules (off the top of my head, check out deploy and drush), but it's pretty crucial to understand the background.
essentially, your site has 3 components: the code (including customizations like your theme), the data, and any user created/uploaded files.
the code is not difficult to manage and deploy across multiple servers simply using SFTP, or better yet using a revision control system. the user files can also be archived and moved between servers easily.
the data is the tricky part, because so many configuration settings are made through Drupal's web-based UI and stored in the database. afaik, unless you have some serious development chops and resources at your disposal, the only real option at this time is to keep very careful notes of these settings and carefully apply them to your live site after testing them on dev/staging sites. i've seen talk of scripting these updates (like this article by Sacha Chua, or this old comment by sepeck -- you might want to check out that whole forum topic), but that all seems pretty intricate, difficult, and hypothetical to me...
my current approach is to keep a manually-edited, comprehensive notes file stored in my revision control system along with the code (ugh).
specifically for your task at hand (and making a few guesses/assumptions about your situation), something like this sounds reasonable:
* start by moving all code, data, and files from your dev server to your live server
* in the future, make code + config (data) changes to your dev site first, then manually reproduce these changes on your live site
* download db dumps + file archives from your live site to your dev site whenever you want to more closely synch the two
PS: for god's sake let's be sure this stuff is well-documented in the handbook!
i know thats in the
i know thats in the handbook...and as i said in my original post, moving the actual code/themes/files is the easy part. the data is the only bad part of it, and i am perfectly capable of using mysqldump to just drop the data out to a file and then pull it back into the new server.
it really isnt hard to do, i was just wondering if there were a more "automated" approach to it that could be installed and used directly from drupal itself. the above module should work fine, it is just taking a blanket dump of the tables in the database that a drupal installation is set to use i am sure.
the handbook tells me what i already know...i can go through some steps and get the job done myself, i was just curious as to whether or not someone else out there may have kept me from opening a term when i want to manually generate a backup of the data. ;)
ok, sorry for my rambling
ok, sorry for my rambling post then. i'm always trying to get my thoughts organized on these issues so i have some notes to refer back to -- i suppose we should all just focus on improving the handbook pages instead.
good luck.
No need to make a mountian out of mole hill :)
Ola,
I've moved many drupal sites, from differing dev, staging enviroments. The suggestion of "Noting down all your settings in drupal" is an approach that I would not take. You bound to get stuff wrong and create problems.
The steps I take are:
Move code base (all uploaded user files etc will be in this structure depending on how and where you've setup your file systems for uploads etc. check Admin/Site Configuration/File system).
Export your db (using whichever way you're comfortable, personally, I use phpMyAdmin to dump db into .sql file)
Create new database on live server with same name as your exported db and same user names permissions (import exported db .sql file)
Then test your site.
Things to be aware of, /sites/default/settings.php holds your connection info. If you've named you're new db the same as your old db and used the same user name with same permissions then this wont need changing.
Oh yeah, and 777 Chmod your files follow to!
Alos, on the Backup and Migrate module tip, it's nearly always in the top 10 suggested modules cos it's robust!
Export your db (using
and this works fine if you want to pull down changes from your live site to synch up dev/staging sites. but this approach does not help in the common workflow of developing and testing features on dev/staging sites, then pushing them out to a live site.
The biggest problem with the
The biggest problem with the mysqldump approach is when you're moving updates to an existing server that has user-generated content (comments, forum posts, user profiles, etc.) You can't just push the entire database from dev to live because you will lose all that stuff, but you can't necessarily just pull down from live either because you'll delete your current work. This is why you see a lot of reference to doing changes twice by hand and taking careful notes, etc.
If your site has no significant user-generated content to worry about then consider yourself very lucky :)
Been fighting the exact same issue.
Deployment from dev ->test->production seems to be a nasty octopus of challenges.
I've spent a bunch of time digging into this issue and there seems to be several Methodolgies used to tackle the beast.
1. Process mimicry: Do what you need to do on dev. If it works, do it again on production, exactly the way you did it on dev. -obviously this is awful, is error prone, and isnt really part of the dev/test/prod paradigm. But might be all you *really* need for super quick and dirty changes. And being a crappy way to do it doesnt stop people from trying.
2. Export db changes to disk: Put whatever changes (structural and data) are going to be made to the database in development to the filesystem. There are many, many ways to skin this cat. Indivdual modules have wonderful tools built in for exporting whatever they have added, like content copy and views export. But what about those modules who's admin pages have a lot of configuration data that get stored in the variables
dumpsterTable? You have to write a custom module with a series of hook_update_N() calls, but this requires getting into the guts of the modules to see what they are doing, and hopefully grok it well enough to duplicate the database updates in your own module. whee. So much for "time saved" by using contributed modules.There are some other projects that are attempting to develop a solution by making these export to disk structures a little more standardized (If I understand it correctly):
http://drupal.org/project/patterns
3. Database wizardry: Since the database holds information about who is using your production site (userdata), what is being looked at (content) and how it should all work (permissions, cck, views, module data, custom data etc) theoretically you should be able to drive a wedge between what is production sourced data, and what is data that comes from developmental changes. The main issue I've encountered with this methodology is key collision. Pointy hairs want to add a new category of stories in dev to see what they'd look like, but at the same time other nodes are being created on production... nid collision. Some techniques advocate hacks so that even keys are generated in prod, and odd keys in dev.
This project, which Im going to dig into next, seems to try and address that issue:
http://drupal.org/project/dbscripts - it gets around key collision by "making room" for dev sourced nodes by moving the starting node counter.
Maybe this project is getting mature enough?:
http://drupal.org/project/deploy
Thanks. Very helpful
Thanks. Very helpful comment.
I`m currently using 2nd method when possible and 1st in other situations.
I had read about all this import/export/migration modules but many of them are for Drupal 6 and I`m still on Drupal 5 :(
But I never heard about dbscripts module before.
I`m already use git for my Drupal installation and I think I will feel comfortable with such a CLI tool.
Does Drupal 7 improve the migration quandary?
It seems like this is a big enough issue that it should be a core feature. Or, perhaps not a feature, but a set of development paradigms, or rules, about where data is stored. Maybe that paradigm already exists, but there is lack of defined automation in place beyond "do this in dev then do it in staging and/or prod, manually". Or, does the Drupal data model need a seriously drastic change, where clearly defined silos of data exists (user data, module config data, core config data, non-static node data (i.e., products, docs)). Drupal is pretty awesome, but I've heard of two companies avoiding it as a potential platform because this is perceived as a long-term issue. Doing things manually doesn't scale, and it's too hard to version control configurations that are strewn around a database and/or co-mingled with non-config "content" data. How do other big companies, or orgs like whitehouse, that use Drupal handle such migrations? Any detailed case studies?
I kind of envision an operating system approach, where user data (and generated content) gets stored in a certain folder (user home folder), application configuration data is in a registry (like in Windows) or dot folders (like *nix). Then, when I want to migrate my OS, I know exactly which areas (folders) to backup and move, or, it's easy to cherry pick the data I want to migrate. The (enterprise) public tends to sort out which apps follow the best practice by storing things the right way, and simply not using those that don't.
MySQL Update Log approach?
This gets even worse when you're running Ubercart and have to worry about all your sales and inventory data! We're up against this now. I'm ok with process mimicry if I must, but we're theorizing on trying this approach:
1. Turn on logging on mysql on the PROD server
2. Let it cook for awhile with only customer activity, inventory activity.
3. GREP the logfile for the UPDATES and INSERTS - grab a list of the tables effected
4. Set up MySQL replication one way from PROD to STAGE for just the tables being written to in PROD.
5. Make your changes to blocks, CSS, etc. in STAGE.
6. When you want to publish from STAGE to PROD, take the PROD site offline for a few mins, do one more mysql replication to STAGE, then dump the entier STAGE db set into a newly created db on PROD, repoint your config to look at that db, then take the site online.
This assumes that the tables used for orders, inventory, user management, etc. are not shared with those used for blocks and other layout/presentaion state.
What do you think? Am I missing something obvious here? (I'm a relative noob, I'll admit it)
I really liked this approach.
I really liked this approach. Have you tested it yet?
I wonder how the content tables (especially those created with CCK) would be treated.
They are affected by the users, but are an important part of the development task.
Migrating Error
Has anyone seen this error before and no how to fix it?
Drupal 7.8, PHP5, MYSQL 5
PDOException: SQLSTATE[42000] [1044] Access denied for user 'dbuser'@'%' to database 'dbname' in lock_may_be_available() (line 167 of /backup/localhost/userdirectory/includes/lock.inc).
Thank
Migrating Error
Has anyone seen this error before and no how to fix it?
Drupal 7.8, PHP5, MYSQL 5
PDOException: SQLSTATE[42000] [1044] Access denied for user 'dbuser'@'%' to database 'dbname' in lock_may_be_available() (line 167 of /backup/localhost/userdirectory/includes/lock.inc).
Thank
The error is because of the
The error is because of the MySql user name and password between your development and production server is different.
You need to create mysql user name and password on your production server according to development server,
or change mysql login and password in settings.php according to your production server.
Thanks