Create a Test Site

Last modified: August 19, 2009 - 00:35

Glossary Of Terms:
Production site: The live website where all your web visitors go.
Test site: A copy of your live site, at a different URL, where you can make experimental changes without affecting your prod site

Drupal and its modules are in pretty constant development. So things are changing pretty fast and sometimes not completely documented. In practice, that means making a change or adding/removing a module on your prod site can have rather surprising results - surprises you'd rather your visitors didn't have to live with. That's why I recommend making all changes to a test site, and if they work out, promote the changes to your prod site.

Here's how I built a completely parallel site on my hosted, shared server account.

First I need to mention some things about my site, so the rest of this article makes sense:

  • My site is called example.com
  • it's hosted on a shared server
  • I have shell access to my hosted files. Without shell I wouldn't really want to attempt all this.
  • I am allowed to have two mysql databases on my account
  • all site files are at /home/user/public_html
     (there's also an alias, /home/user/www, which points to public_html)
  • in the above, user stands for my username, which I'm not telling you
  • I keep my production drupal database in mysql; that db is named mydb_prod
  • Apache is set up so that http://www.example.com and http://example.com will point to /home/user/public_html

All of this is pretty standard stuff, but you may have to make a few changes to this process if your site is set up differently. My test/prod workflow looks generally like this:

  1. Make a new (empty) database, and name it mydb_test
  2. Copy your prod db into mydb_test
  3. Copy all of your prod files (from public_html) into your test directory
  4. Make three settings changes to connect the test site to the test db
  5. Make changes to your test site.
  6. (optional) replicate test site's files and db back into the prod site to make your changes live

So how to actually do all this? Don't worry, it's not that hard once you have the right commands handy. Really, I do steps 1-4 in about 2 minutes. This document is long only because I wanted to be verbose enough for all skill levels!

Step 0 - work with your host provider

The first thing I did was ask my hosting provider to create a new directory, /home/user/test and then set up apache and their DNS servers so that http://test.example.com and http://www.test.example.com would point to this new test directory. It was important to ask them to make a new VirtualHost entry in the apache configurations, NOT just set up a rewrite rule. They should know what you mean by this - and they might charge you extra for it. Mine didn't, but even if this costs you a couple of extra bucks, it's worth it. Trust me when I say that using a rewrite rule or subdirectory of your existing site will make the rest of this much harder to do.

When this is complete your directory structure will look something like:

|-home
  |-username
    |
    |-public_html
    | |
    | |-database
    | |-files
    | |-includes
    | |-misc
    | |-modules
    | |-scripts
    | |-sites
    | |-themes
    | |-(drupalfiles)
    |
    |-test
      |
      |(empty because we haven't put anything here yet)

/home/username could in your case be /var/www/username, or something else entirely. The point is to have a test directory which is not part of your normal html directory, and which acts as a separate site in all ways.

You'll also need a second mySQL database to house your test site's data.

Step 1 - Create an Empty Database

I use the commands

mysql -uuser -ppassword -e "create database mydb_test;"
mysql -uuser -ppassword -e "grant all privileges on mydb_test.* to user@localhost identified by 'password';"

to create my database and set the permissions. The I broke a single mysql command into the last two lines because drupal.org wasn't printing it all as a single line. If you are restricted from doing so, you may need to create the empty database via cPanel, Plesk, or whatever control panel your provider gives.

Step 2 - Replicate the database

mkdir ~/temp
mysqldump -uuser -ppassword --add-drop-table mydb_prod > ~/temp/prod-db
mysql -uuser -ppassword mydb_test < ~/temp/prod-db
rm ~/temp/prod-db

Above you see the commands I enter. You'll need to change things user and password to your username and password; also mydb_prod should be replaced with the mySQL database name for your prod site. Knowing that, I bet you've already figured out that you need to replace mydb_test with the name of your newly-created test database.

The first line just creates a directory to hold our temporary dump file.

The second line, starting with mysqldump, is where we dump the entire prod database to a textfile. We use the --add-drop-table option because this essentially creates a script we'll run later; this option makes sure that script will erase old data before importing new data. Note that you need to supply the proper username and password for your production database. I specify the database name mydb_prod - you will need to change this to the name of your prod site's database. The textfile created by this command is actually a full script which will create tables and populate them with data.

The third line, starting with mysql, is where we do that. Again, be sure and replace the italicized parts with the username, password, and database name you set up in step 1.

Finally I delete the dump file (~/temp/prod-db); it's not needed anymore now that we have loaded it into mySQL.

Step 3 - Copy the web files

rm -rf ~/test
cp -R ~/public_html/. ~/test

Here I remove the test directory, then recreate it, and copy everything from ~/public_html into it. I remove the test directory and all contents because I may copy of the test site in there, and I don't want it to pollute my fresh new exact replica of my prod site.

Step 4 - Edit Settings

nano ~/test/sites/default/settings.php

This starts the nano editor (or use whatever you like) and loads drupal's settings file. Here you will need to change two lines:

$db_url = 'mysql://user:password@localhost/mydb_test';

$base_url = 'http://test.example.com';

These lines are not together in the file. Set the database connection line, pointing to the test database you created in step one and populated in step 2. (in other words, you need to replace the words user, password, and mydb_test with information that's correct for your test database.) The base_url line sets the URL you will point your browser at to use the site. Save the changes.

Now, in your web browser, navigate to the site - it should be working fine. But I also change the Name field at administer --> settings to TEST.example.com (all caps) so its easy for me to remember I am looking at the test site.

Step 5 - Make your changes

This is where you make you changes. Change settings, add/remove modules, muck with the code - it's up to you. Remember you are working in a copy of your site - one that users don't know about - so you don't have to worry that you are creating problems for the users.

Killes has written a very nice and very quick way of testing your site at node/11521

And if you don't like the changes, or if you mess up your test site
completely - no worries! Just re-do steps 2-4 to bring a new copy of your
production site and database into your test environment.

Step 6 - Promoting changes to production

Now that you have your test site the way you want it, you really have two choices.

Method 1: You can re-make all the same changes manually on your production site.
Method 2: You can basically migrate the test site into production, with a very brief outage for the users.

I usually use method 2, but have been known to do it both ways. To use method 2, I'm essentially doing everything mentioned above, but this time copying the test site into the production location. So I'll skip over it more quickly this time around:

  1. mysqldump -uuser -ppassword --add-drop-table mydb_test > ~/temp/test-db
    mysql -uuser -ppassword mydb_prod < ~/temp/test-db
    rm -rf ~/temp/test-db
    rm -rf ~/public_html/*
    cp -R ~/test/. ~/public_html
  2. nano ~/public_html/sites/default/settings.php (here you'll edit the db_url and base_url to the prod-site settings)
  3. Also be sure to open the prod site, navigate to administer // settings and change the name back to the way it should be for the prod site.

You should probably test your site again using the Killes method linked in step 5.

There are a couple of additional cleanup steps I do, once I know my site is in good shape:

  • rm -rf ~/test/* (just cleaning out test dir for next time)
  • rm ~/bash_history(because it has my database password in it)
  • Remove the test database (if space is a concern)
  • Further automation, and an observation on workflow

    neiljt - September 22, 2005 - 01:08

    Many thanks for sharing this. I feel much more relaxed about the idea of experimenting now that I can do so in a sandbox! Here's a minor enhancement for the benefit of lazy people like me, followed by a note about the workflow.

    1. Rather than changing the credentials manually in the settings.php file, it's possible to use sed. This example edits files after copying LIVE to TEST data:

    typeset -r LIVE_SETTINGS="/path/to/live/sites/default/settings.php"
    typeset -r TEST_SETTINGS="/path/to/test/sites/default/settings.php"
    sed -e '/^\$db_url/s/live/test/g' -e '/^\$base_url/s/live/test/g' < ${LIVE_SETTINGS} > ${TEST_SETTINGS}

    Obviously, this idea can be extended to include other items (e.g. passwords) which vary between the 2 areas.

    2. Whilst it may be valid, even desirable, to replicate the live database in a test area in order to work in a sandbox, e.g. to mess with templates, styles, etc., it's probably unwise to promote this copy back to the live database when work is complete. Real world changes may have occurred to the live data in the meantime, and these would be overwritten. Thus I have created 2 scripts as follows:

    • l2t - replicate DB and HTML (LIVE to TEST)
    • t2l - promote HTML only (TEST to LIVE)

    I have 1 further script, just in case I ever need it:

    • t2l_db - promote DB only (TEST to LIVE)

    Of course the above works for mods to HTML, but if working on content, user maintenance, moderation, and so forth, this must ultimately be performed directly against the live DB (whether or not a dry-run has taken place against the test DB).

    3. There may be a good reason that I am missing, but I am denied access to Killes' site testing method when I try to follow the link.

    You could also edit

    graemec - September 24, 2007 - 21:23

    You could also edit 'settings.php' to auto-detect the environment based on the path or url. For instance:

    if current path = './test' then $db_url = 'test'
    else if current path = './prod' then $db_url = 'prod'

    OR based on url

    if url = 'test.domain.com' then $db_url = 'test'
    else if url = 'www.domain.com' then $db_url = 'prod'

    Now when you backup and restore your test site it will detect which configuration is needed. It's worth the overhead for me... but it would be nice if drupal supported different environment configs.

    ... Also, to differentiate the two sites, I setup a new "Block" within the drupal installation, called "Stage Site Indicator", with the following sample php-html:

    if( getcwd() == "/path/to/stage/site" ) {
    echo '<img src="stage.gif" style="position:absolute;z-index: 1;left: 0px;top: 0px;">';
    }

    ... Now anytime I view the stage site, a banner appears diagonally across the top left showing me that I'm definitely on stage...

    mail redirection

    graemec - October 2, 2007 - 15:50

    Some other things to consider with a test site that may be based on production data:
    - accounts are still active
    - outgoing emails still sending

    My stage site is a localhost install which means external users cannot access the site. In the future I plan to create a module or script that blocks all accounts except admin, or alternatively, resets passwords to a master key.

    The outgoing email was more of an immediate concern so I hacked together a very simple module to detect the stage install, and redirect the email to a hard-coded email address.

    'module/mail_redirect/mail_redirect.module'
    -----------------------------------------

    <?php
    function mail_redirect_mail_alter(&$mailkey, &$to, &$subject, &$body, &$from, &$headers) {
        if(
    getcwd() == "/path/to/drupal_stage" ) {
           
    $body .= "\n\nMessage redirected from $to";
           
    $to = "email@domain.com";
           
    $subject = "[STAGE] - ".$subject;
        }
    }
    ?>

    'modules/mail_redirect/mail_redirect.info'
    -----------------------------------------
    name = Mail Redirect
    description = "Redirect all mail to specified address"
    version = "5.x"
    project = "mail_redirect"
    datestamp = "1180970107"

    Avoid temporary database dump file with...

    BladeRider - April 28, 2008 - 19:46

    Lovely guide that's concise and easy to follow, thanks for that.

    One little polish would be to avoid needing a temporary directory/file for the database dump in step 2 by piping the output of mysqldump directly into mysql like this:

    NOTE: these code examples are one-liners - the formatting here may break the lines

    mysqldump -uuser -ppassword --add-drop-table mydb_prod | mysql -uuser -ppassword mydb_test

    ...and similarly for step 6, part 1:

    mysqldump -uuser -ppassword --add-drop-table mydb_test | mysql -uuser -ppassword mydb_prod

    Extending permanently using symbolic links

    daleu1 - July 19, 2008 - 13:04

    I just found this article and thought I'd weigh in. The parallel environment you described can stay intact allowing you to make "what if" changes at any time. Being an old mainframe guy I borrowed a concept from the 70's called "copy down, move up" meaning you copy files "down" from production into test but then move them back up (they don't stay in test).

    To use this concept you must be using an OS that allows symbolic links (any Unix or Linux derivative). For background, a symbolic link is merely a special file that actually points to a real file and in our case we want all of the files in the test area to point to the files in the production area using symbolic links unless they are being modified. You'll need to add an Apache directive called FollowSymLinks on the test side only so if a file is not "down" in the test area being modified, apache will pick up all the production files.

    The links in the test environment will look like this:

    index.html --> ../production/index.html (a simplified example, for the example above it would be a bit more complex)

    Once this is in place for all files, you then can create a couple of shell scripts that "check out" production files and copy them down to test.

    CheckOut script function
    - Remove symbolic link in test
    - Copy production file to test
    - create "index.html.o" in production so you know it is currently checked out

    CheckIn script function
    - Verify that "index.html.o" exists
    - (optional) Push a copy of the existing production file down into an archive so you can quickly back out changes
    - Move the test version to the production area, overlaying the former production file
    - Create the symbolic link pointing to the new file

    That's it. You can now easily check out any file and make a small change. When you go to your test URL, Apache will pick up the new file from test and all of the other files from production.

    As for the test db versus production you will have a couple of static "where am I" files that direct your db connections to the correct db. I definitely put these down into a private directory for security reasons.

    I've built several large scale applications this way and it works wonderfully.

    D

    afraid of command lines

    sukioki - September 9, 2009 - 20:42

    okay, so I am not a programmer and I have a healthy fear of terminals.

    Is there some way I can do the above by just copying over the files in my Cpanel file manager and then changing some (?) *.php file?
    My organization can not afford anything but novice me, and I have been known to have finger dislexia, so if there's some way I can work through a GUI, that would really help me and other novices out a LOT!

    I use the combination of git, drush, phpmyadmin and backup_migra

    bibstha - October 9, 2009 - 06:16

    I use the combination of git, drush, phpmyadmin and backup_migrate module.

    1. Create a git repository inside the production environment.
    2. Use backup_migrate to create a new database dump.
    3. Clone the git repository to a testing folder.
    4. Create new database using phpmyadmin or mysqladmin
    5. Setup drupal in test (installation),
    6. Enable backup_migrate
    7. Restore the database dump from production.

    At step 7 I have an exact clone of production website. The advantage of this from using mysqldump is that, unnecessary data like sessions information, watchdog information are not migrated to testing.

    8. Any changes made to style or theme or new modules are migrated to production using git push.
    9. Any changes made to the database (new page, new view, new panel, edited content) are migrated depending upon the website.
    9a. if the changes are to an almost static website, we create a dump of test and put it to live server
    9b. If the server is dynamic with almost minute wise changes, I make atomic changes step by step to the production server.

     
     

    Drupal is a registered trademark of Dries Buytaert.