make rsync transfer only changed files

That is true - rsync will transfer all files again on every run if you don't use -u option. See example: https://gist.github.com/838524

I think I disagree with your stated assumption:

without --upadte, you _always_ transfer all files, no matter what.

The manual says otherwise:

       -u, --update
              This forces rsync to skip any files which exist on the destination and have  a  modified
              time that is newer than the source file.  (If an existing destination file has a modifi‐
              cation time equal to the source file’s, it will be updated if the sizes are different.)

              Note that this does not affect the copying of symlinks or other special files.  Also,  a
              difference  of  file  format  between the sender and receiver is always considered to be
              important enough for an update, no matter what date is on the objects.  In other  words,
              if the source has a directory where the destination has a file, the transfer would occur
              regardless of the timestamps.

              This option is a transfer rule, not an exclude, so it doesn’t affect the data that  goes
              into  the  file-lists,  and  thus it doesn’t affect deletions.  It just limits the files
              that the receiver requests to be transferred.

I have been using rsync for a long time, and rsync doesn't sync everything just because -u isn't there. A clear example:

anarcat@angela:a$ ls
test
anarcat@angela:a$ rsync -t --stats test test1

Number of files: 1
Number of files transferred: 1
Total file size: 10240 bytes
Total transferred file size: 10240 bytes
Literal data: 10240 bytes
Matched data: 0 bytes
File list size: 18
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 10308
Total bytes received: 31

sent 10308 bytes  received 31 bytes  20678.00 bytes/sec
total size is 10240  speedup is 0.99
anarcat@angela:a$ rsync -t --stats test test1

Number of files: 1
Number of files transferred: 0
Total file size: 10240 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 18
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 25
Total bytes received: 12

sent 25 bytes  received 12 bytes  74.00 bytes/sec
total size is 10240  speedup is 276.76

Notice how the first run sync'd all the data, while the second didn't. So -u is not the answer here. -c could have the same result, as it uses checksums to compare files, but -t is usually enough and less CPU-intensive.

To quote the manual again:

       -t, --times
              This tells rsync to transfer modification times along with the files and update them  on
              the remote system.  Note that if this option is not used, the optimization that excludes
              files that have not been modified cannot be effective; in other words, a missing  -t  or
              -a  will  cause  the  next  transfer to behave as if it used -I, causing all files to be
              updated (though rsync’s delta-transfer algorithm will make the update  fairly  efficient
              if the files haven’t actually changed, you’re much better off using -t).

@anarcat - rysnc does copies *all* files every time your run it without -u [edit: or -t], please see this simple example: https://gist.github.com/838524

However, if you will use -t (without -u) with first rsync, the next rsync will of course transfer only changed file and the next attempt to rsync even without -t option will still copy only changed files.

Example: https://gist.github.com/842122

But if you will rsync without -t or -u option every time, it will transfer *all* files again on every attempt, even if only one file has changed in the source directory.

Example: https://gist.github.com/842121

Hmm.. now I'm not sure - does the -t by adding times is more reliable than -u option in this case? What if the local and remote servers have different system time (or even date)? Will this break anything?

@octopus - I read the example you mention (https://gist.github.com/838524) before replying - did you read the documentation about -u? I do not think it is what you think it is:

This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file.

Why a file on the remote server would have a modified time **newer** than the source is beyond me, and -u doesn't *update* the timestamps, so you are bound to reupdate the files anyways, or to update the wrong files. It also operates only on the receiver end and *skips* files. For example, if the remote file has been "touched" while the local file needs to be synced up, -u will forbid that to happen, which I don't think is what we want.

I agree that not using -t, -u (or -c) is sub-optimal - but it works right now, where as using -u would actually introduce a bug (ignoring some remote files). -t is the usual way this is done (ie. it's in -a).

If times are not sync'd you're screwed anyways, whether you use -t or -u because well, they rely on times. :) Although it will affect -u more, because -t sets the time as seen from the source server, while -u relies on the target's notion of time.

To put this shortly: I recognize we can optimise this. -u is not the right way, -c or -t is.

@anarcat - I was confused a bit since I never used rsync without options, always with some mix or just -a, but you are right - the -u option would introduce a new layer of chaos/bugs in the already enough hard to understand magic introduced by this remote stuff in Aegir. Thanks for clarify it!

So, at the moment we just play this safe, and we get rsync to check the contents of every file. This does NOT mean we actually transfer each file though.

Marking as postponed until someone wants to pick this up and come up with a good solution/benchmarks.

Rsync does NOT "always transfer files" without -u. Straight from the man page, rsync uses a "quick check" algorithm to determine what files have updated by looking only at the file size and timestamp. If either of those are different for a file, then it does a full comparison using the delta change algorithm. The delta change algorithm compares files on the source and destination by performing a checksum on each file chunk, and does not transfer the entire file while doing this comparison.

When you run it from the command line you will see it print out every file name, but that does not mean it is actually transferring the file data. The only data that gets transferred are the information required to compare the files, such as the file list, timestamp, size, etc..., and the data chunks for any file that has changed or new files. The bandwidth usage that is reported includes any speedup gained by having partial files on the destination side, so it may appear to be doing something that it's not.

This ticket should be closed.

The point here is that we should use -t to make sure the remote timestamps are updated when we sync, which optimizes the transfer. Again quoting the manual:

Note that if this option is not used, the optimization that excludes
files that have not been modified cannot be effective; in other words, a missing -t or
-a will cause the next transfer to behave as if it used -I, causing all files to be
updated (though rsync’s delta-transfer algorithm will make the update fairly efficient
if the files haven’t actually changed, you’re much better off using -t).

But I agree we could just close this ticket already, as even if -t is not used, rsync will only transfer what is needed. That can be easily seen with --progress, on big files... So, closing.