remote (offline) drupal installations data sync with main online site?

By sander-martijn on 4 Jun 2013 at 22:25 UTC

I'm an experienced drupal developer (tons of sites in versions 5-7 over the years, all custom themes, some custom modules etc), but I'm coming across a use case where I'm not sure how to approach it, looking for suggestions on where to start, whether there are modules that will help with this (multi-site will not for example solve this issue).

What I want to have is one central install of drupal maintaining a core data set. Other installs of drupal (eg. on laptops) can download sections of data from that main application, then go offline and run with that data locally, which may get updated during that time. Later, when the remote instance(s) return online they will automatically sync any changed data to the primary application.

This will be easier with a use case. Traveling salespeople/pharma reps or the like - you have a central drupal site which tracks all customers and their orders. In the morning a salesperson opens up a local instance of drupal running on their laptop and selects the customers they'll be visiting that day, whose profiles, orders etc get downloaded to their laptop db. While they're out they may need to update a customer's address, phone or other info, and add new orders. When they return to the office or anywhere else with internet, the central system is automatically notified of these changes and additions. At the end of the day they can remove those customers from their system (in this case not removing them from the central server) and start with a new list the next day. The remote sites would use a different (simpler) theme and have different functionality, so all we're synchronizing are specific pieces of information. The salesperson would generally never access the main site - that would be accessed by people in the office who need to view all orders from all their remote salespeople.

That's not my exact use case (which I'm not at liberty to discuss), but is functionally equivalent.

Any ideas on how this might be approached?

Comments

slight correction

sander-martijn commented 4 June 2013 at 22:28

the syncing at the end of the day does not need to be automatic - i would be fine with them having to actively "report in" to trigger the sync. in fact that would be better, since someone might have internet access (knowingly or even unknowingly) while they're in the middle of updating information.

In summary, it sounds like

zeezhao commented 7 June 2013 at 08:18

In summary, it sounds like you will need a custom module that will keep track of daily activity (just like heartbeat module or similar) on each salespersons' system. Then on the press of a button from a master, retrieve each log from local systems, to initiate node updates in master database.

May need to keep timestamps on each local copy to know when master sync last run.

Getting a local copy each day with only data required may be tricky. Easier to dump master and load onto local systems each day...

Maybe leverage migrate module?

update:
See http://drupal.org/project/migrate_d2d
http://drupal.org/node/1888620 -- sounds like this may help
http://drupal.org/project/deploy

Also see http://drupal.org/project/pouchdb

Interesting suggestions

sander-martijn commented 12 June 2013 at 20:14

Thanks,
Those are interesting suggestions. I will look into the links you posted as well.

One thing is that at least in my case for security reasons (theft of a laptop for example) we cannot have the master downloaded. It MUST be the required subset for the day and nothing more. A data compromise of that sort would be catastrophic. At least if we can limit it to a loss of that day's data it becomes manageable risk.

_

drupestre commented 2 July 2013 at 08:20

Hello,

... Some ideas :

I have to deal with a similar issue for a client.
My approch is quite different cause the sync should be made from phone to the main site as well.

The services module handles data update with using REST API. So, your remote device could interact with this service.

Do you really need drupal on your remote device ? Cause a desktop application could use a webservice to synchronize the data.

I think this is the way I will follow in order to sync data between phone and the main site.

Hope it helps
Benoit

Building an offline platform for an educational NFP

bohemier commented 24 November 2013 at 13:56

Hi!

Which approach did you take? I am looking for a similar solution for an educational platform I am building for a non-profit organization. Basically, teachers on the road would be able to edit and review their students reports and exams while offline. I was thinking of making one drupal install per teacher on the server using multisite along with the main website. Then running a local copy of each of these teacher install (using MAMP or the like), synchronized with the online server using drush rsync/sql-sync. The main website would then synchronize content to/from the teachers install.

I was looking through HTML5 localstorage/indexeddb/websql but I don't think these technologies would be appropriate for loading a node edit form while being offline...

Any thoughts would be welcomed!

J-F Bohémier
Angelicode.com

pouchdb

zeezhao commented 14 May 2015 at 15:52

Even though this posting is old, please did anyone implement using any of the suggestions above?

Thinking of using pouchdb to sync master drupal database when using a POS terminal (e.g. commerce_pos) in location where internet connection is flaky for cash only payments.

Thanks

Not so far

sander-martijn commented 14 May 2015 at 16:19

I went through several different possibilities but never found an acceptable solution. I ended up deciding that the remote system should be tablet based for my needs (so not drupal, though it may still communicate with a drupal server when syncing up), and it's a pet project with no deadline. I'm not an app developer (and no interest in learning it), so I'd need funding which I don't have - so for me at least the project got shelved.

but to answer an earlier comment, yes in my case the data has to be able to be synced in, available without internet, and updated at a later point - because the actual data input will be done while very offline.

I still want to revisit this, but it has to happen when I have downtime and money - a combination I don't see as often as I'd like to.