My words on free/open source software

Sunday, October 09, 2011

Using rsync to migrate data between Linux and Apple Mac OS X

I got a new MacBook Air running Mac OS X (10.7.1) "Lion", and I managed to migrate my archive data (around 1.3TB) from an old Linux laptop running Debian Testing "Wheezy." The process was not smooth and I hit several blocks. Here I document several points that you need to pay attention to when migrating data across operating systems:


Use the latest rsync (3.0.8 from MacPorts was used here). The old version of rsync is buggy on Mac. The version of rsync on Linux doesn't matter much because rsync on Linux has been quite stable since long ago.


Disable "Ignore ownership on this volume" (you can use Finder or from command line). If not rsync won't be able to set file permission correctly. I think if you don't instruct rsync to preserve permission and mode then you might not need to do this but I'm not very sure.


Disable antivirus software like McAfee because they interfere with rsync's time preservation.


Use rsync's "--iconv=UTF8-MAC,UTF-8" option (only available after rsync 3.0), because Mac's HFS+ decompose UTF-8 before storing it so the filename would be different from that of Linux if no conversion is done right.

If you saw "iconv_open("UTF-8", "UTF8") failed", try swapping the parameters of iconv.


This is not directly related to rsync nor Mac, but a general rule for handling archive data: use another piece of software to do the verification. Your rsync might be buggy, the OS might be buggy, the disk or USB cable might be kaput, you might have used the wrong options or settings. Therefore you should always use another piece of software other than rsync to verify the data before and after the migration. I used AIDE to do file checksum and time verification.


Neo said...

You might want to have a look at The Fink project.

Yan Li said...

Thanks for the comment. Will try it.

DDD said...

I prefer homebrew. Though homebrew will not include duplicated tools of apple shipped ones, there a homebrew-alt can support rsync-3.0.9

Yan Li said...

Thanks DDD. Yeah, I remember I read somewhere that the rsync 2.x shipped by Mac is buggy and may crash during copying. Using a later version is always recommended.

Gary S. Weaver said...

To get latest homebrew version of rsync:

brew install

AndyB said...

If you have two Linux files whose names are identical except for case (e.g. /tmp/TEST and /tmp/Test), only one of them will show up on the Mac. (One will overwrite the other because OS X ignores case when opening files).

If you run rsync twice in a row, you'll see the file being transferred each time.

Yan Li said...

@AndyB Good point. Thanks!

Diego Menéndez said...

Thanks a lot! I installed rsync 3.1.0 with:
brew install homebrew/dupes/rsync

Prologic Corporation said...

This is a good article & good site.Thank you for sharing this article. It is help us following categorize:
healthcare, e commerce, programming, multi platform,inventory management, cloud-based solutions, it consulting, retail, manufacturing, CRM, technology means, digital supply chain management, Delivering high-quality service for your business applications,
Solutions for all Industries,
Getting your applications talking is the key to better business processes,
Rapid web services solutions for real business problems,
Web-based Corporate Document Management System,
Outsourcing Solution,
Financial and Operations Business Intelligence Solution,

Our address:
2002 Timberloch Place, Suite 200
The Woodlands, TX 77380


About Me

My photo
Santa Cruz, California, United States