My words on free/open source software and Mac

Monday, October 10, 2011

Using rsync to migrate data between Linux and Apple Mac OS X

I got a new MacBook Air running Mac OS X (10.7.1) "Lion", and I managed to migrate my archive data (around 1.3TB) from an old Linux laptop running Debian Testing "Wheezy." The process was not smooth and I hit several blocks. Here I document several points that you need to pay attention to when migrating data across operating systems:

Zeroth

Use the latest rsync (3.0.8 from MacPorts was used here). The old version of rsync is buggy on Mac. The version of rsync on Linux doesn't matter much because rsync on Linux has been quite stable since long ago.

First

Disable "Ignore ownership on this volume" (you can use Finder or from command line). If not rsync won't be able to set file permission correctly. I think if you don't instruct rsync to preserve permission and mode then you might not need to do this but I'm not very sure.

Second

Disable antivirus software like McAfee because they interfere with rsync's time preservation.

Third

Use rsync's "--iconv=UTF8-MAC,UTF-8" option (only available after rsync 3.0), because Mac's HFS+ decompose UTF-8 before storing it so the filename would be different from that of Linux if no conversion is done right.

If you saw "iconv_open("UTF-8", "UTF8") failed", try swapping the parameters of iconv.

Fourth

This is not directly related to rsync nor Mac, but a general rule for handling archive data: use another piece of software to do the verification. Your rsync might be buggy, the OS might be buggy, the disk or USB cable might be kaput, you might have used the wrong options or settings. Therefore you should always use another piece of software other than rsync to verify the data before and after the migration. I used AIDE to do file checksum and time verification.

9 comments:

Neo said...

You might want to have a look at The Fink project.

Yan Li said...

Thanks for the comment. Will try it.

DDD said...

I prefer homebrew. Though homebrew will not include duplicated tools of apple shipped ones, there a homebrew-alt can support rsync-3.0.9
https://github.com/adamv/homebrew-alt/blob/master/duplicates/rsync.rb

Yan Li said...

Thanks DDD. Yeah, I remember I read somewhere that the rsync 2.x shipped by Mac is buggy and may crash during copying. Using a later version is always recommended.

Gary S. Weaver said...

To get latest homebrew version of rsync:

brew install https://raw.github.com/Homebrew/homebrew-dupes/master/rsync.rb

AndyB said...

If you have two Linux files whose names are identical except for case (e.g. /tmp/TEST and /tmp/Test), only one of them will show up on the Mac. (One will overwrite the other because OS X ignores case when opening files).

If you run rsync twice in a row, you'll see the file being transferred each time.

Yan Li said...

@AndyB Good point. Thanks!

mahasiswa teladan said...


hi...Im student from Informatics engineering nice article,
thanks for sharing :)

Diego Menéndez said...

Thanks a lot! I installed rsync 3.1.0 with:
brew install homebrew/dupes/rsync

About Me

My Photo
Santa Cruz, California, United States