Yan Li's Words

My words on free/open source software and Mac

Monday, October 10, 2011

Using rsync to migrate data between Linux and Apple Mac OS X

I got a new MacBook Air running Mac OS X (10.7.1) "Lion", and I managed to migrate my archive data (around 1.3TB) from an old Linux laptop running Debian Testing "Wheezy." The process was not smooth and I hit several blocks. Here I document several points that you need to pay attention to when migrating data across operating systems:

Zeroth

Use the latest rsync (3.0.8 from MacPorts was used here). The old version of rsync is buggy on Mac. The version of rsync on Linux doesn't matter much because rsync on Linux has been quite stable since long ago.

First

Disable "Ignore ownership on this volume" (you can use Finder or from command line). If not rsync won't be able to set file permission correctly. I think if you don't instruct rsync to preserve permission and mode then you might not need to do this but I'm not very sure.

Second

Disable antivirus software like McAfee because they interfere with rsync's time preservation.

Third

Use rsync's "--iconv=UTF8-MAC,UTF-8" option (only available after rsync 3.0), because Mac's HFS+ decompose UTF-8 before storing it so the filename would be different from that of Linux if no conversion is done right.

If you saw "iconv_open("UTF-8", "UTF8") failed", try swapping the parameters of iconv.

Fourth

This is not directly related to rsync nor Mac, but a general rule for handling archive data: use another piece of software to do the verification. Your rsync might be buggy, the OS might be buggy, the disk or USB cable might be kaput, you might have used the wrong options or settings. Therefore you should always use another piece of software other than rsync to verify the data before and after the migration. I used AIDE to do file checksum and time verification.

Thursday, February 17, 2011

openSUSE Build Service (OBS): Break the Link Between a Branched Package and Its Source

We are using openSUSE Build Service (OBS) for the development of MeeGo. I have an old package, which was branched from a Trunk package long ago. But every time the source pakcage in Trunk is updated my package would be broken because the changes between them are so large that it's become meaningless to keep the link between them any more. So I think I should just remove the link between them.

The simplest way may be just removing my package and start a new package from scratch, however, this action not only removes the package history but will also reset the cumulative release version number, which is unacceptable since that would make it impossible for client machines to follow updates to this package.

I have carried out a long search but failed to find a hint to this issue, and after some trail-and-error I think I have found a way:


  1. Check out the project's unexpanded files by using "osc co -u PRJ/PKG"

  2. Remove the "_link" file by running "osc rm _link"

  3. Copy in the latest package files since the link is now broken

  4. Check-in the changes



I've done a test and the steps above worked. However, it seems that by those steps the release version number will also be reset.

Update: darix from freenode/#opensuse-buildservice said 'osc in git has "osc detachbranch" exactly for that.'

Sunday, December 26, 2010

Easy way to align Linux partitions for SSD and myths debunked

Recently I got a laptop from my employer with an Intel X25-M SSD drive in it. I have spent some time trying to figure out the best and easiest way to install Linux onto it while retaining the Windows 7 (so I have to use MBR instead of GPT). There are numerous discussions you can find on the net, some of them are quite complex and some are simply misleading (if not wrong). In fact this task is quite easy with modern Linux and here's a documentation of what I've done. This blog is not meant to be a full discussion on this topic, but only intended as a cheat sheet for those experienced Linux users (if you want a more detailed and correct discussion on this topic, I recommend Ted Ts'o's blog: Aligning filesystems to an SSD’s erase block size.)

For those impatient readers, here's an overview: the disk geometry (or C-H-S value) you see from partition programs as well as the warning about partitions are not aligned to cylinder boundary don't matter at all so there's no point to spend time on tuning loose knobs. New GParted (I used 0.7.1 from SysRescueCD, not to be confused with "GNU parted") is a very easy to use tool and it now defaults to create partitions aligned on 1 MiB (I heard that Ubuntu 10.10 and later can do partition alignment automatically too but I haven't tried that).

Here's the detail:

The myth: Does the C-H-S value still matter?


A lot of discussions were about checking the disk geometry after partitioning. However, in fact no modern operating system pays attention to those values any more (see the comment by Ted Ts'o). And those "-H xx -S xx" options passed to fdisk are only meant to make that fdisk program happy so it can create partitions aligned with cylinder boundaries. Although the C-H-S values are stored in MBR, they are not used by the Linux kernel who runs the whole system. Instead of paying attention to choosing or checking those C-H-S values, you should only need to make sure that the start and end sectors of partitions are multiples of the size of your erase block. Of course you can use any good partition programs to do this.

So you don't have to use that fdisk program


and you can simply chose any program that you are familiar with. For me I'm comfortable with sfdisk and GParted. This time I tried latest GParted and was very happy to see it had supported partition alignment on 1 MiB boundary. So my work is done in one minute without using a calculator. After that I also carefully checked the partition table with "sfdisk -uM" and was sure that those partitions were aligned on MiB boundaries. The extended partition was not aligned but I thought that didn't matter as long as all logical partitions in it were aligned correctly.

A few more tips:



  • If you want to have TRIM/DISCARD support you can use a 2.6.33+ kernel with Ext4 file-system, mounted with -o discard.

  • When making the Ext4 file-system, make sure you use the stripe-width option and let it align with erase block boundary too (see Ted's post above).

  • Don't use LVM or dm-crypt since they don't support TRIM/DISCARD command yet.



compilebench test result:


Intel's X25-M is very smart that even if you don't align the access correctly the drive will do magic under the hood so the initial benchmark data doesn't differ too much, however, the performance will degrade much more quickly as the internal mapping tables getting more and more complex due to unaligned writes. Here I post the file-system benchmark result when it's still new. I'll try to do benchmark again after several months to see if there's any performance degrade (although my usage is not heavy).

I used compilebench v0.6 because I think it mimics daily desktop usage very well.

Machine spec:
System: Debian Squeeze (testing)
Kernel: 2.6.37-rc5-amd64 from Debian experimental repo
Test command: sudo ./compilebench -D t -i 10 --makej (the same option used by Phoronix Test Suite)
File system: Ext4, mount option: rw,noatime,discard

compilebench Initial Create test result: 105.24 MB/s (on 2010-12-27).

Tuesday, December 21, 2010

MeeGo: Auto-Start an Application on Session Start

For modern Linux desktop, it's a common way to auto-start an application in a user-session by placing a ".desktop" file into /etc/xdg/autostart. But if you are writing or porting an application to MeeGo and found that the application is not auto-started after you put in a seemingly correct .desktop, please check this:

If there's an "OnlyShowIn=" item, it must contain the string X-MOBLIN-NB. For example:

OnlyShowIn=GNOME;X-MEEGO-NB


Another caveat is that MeeGo's uxlaunch doesn't honor AutostartCondition key yet, for the sake of simplicity and fast-boot.

For code under the hood, check MeeGo's uxlaunch/desktop.c.

Saturday, December 18, 2010

Very hard to make bootable USB drive from Ubuntu 10.10's CD Image

I was spoiled by MeeGo's super convenient USB live image so I was very surprised when I found that it was very hard to make a bootable USB drive from Ubuntu 10.10's CD-ROM image for my wife.

On Linux, Ubuntu only officially supports making bootable USB drive from its CD-ROM image by using "ubuntu-usb-creator", which is not available in any Linux installations I'm using now (MeeGo, Debian, CentOS). Ubuntu's official help document mentions one alternative way -- to download a small image and combine it with the iso file -- doesn't work, and community posts pointed out that that methods only worked with "alternative iso" (not desktop iso).

Finally I found a way (and the only way) to make a usable USB drive for installation is from this post.

So what? I think Canonical really should release official USB images since most netbooks shipped today have no CD-ROM drive.

Thursday, July 15, 2010

Google Chrome (Chromium) Sync Server Address

If you want to know the host name and address of the sync server Google Chrome (or Chromium) uses, see here:

http://src.chromium.org/svn/trunk/src/chrome/browser/sync/profile_sync_service.cc

For now, the sync service's URL is

https://clients4.google.com/chrome-sync

Sunday, June 27, 2010

HOWTO: Run Plants vs Zombies™ in Linux under Wine (includes how to Register!)

I have to admit that I found PopCap's Plants vs Zombies™ (referred to as PvZ below) interesting. It runs on Windows, Mac and iPhone but not on any Linux yet.

You can download the trial version from the official site above, and it runs amazingly well on a recent Wine, without any special configuration. Well done, Winehq!

Now comes the problem, if you decide to buy it after 60 minutes play, the registration process is a bit hard with Wine. The reason is that PvZ requires an Internet Explorer or Firefox to finish the registration process. Obviously it has to gather some local hardware/system information and upload it to their activation server so that they can know how many machines you are running PvZ on, therefore you won't to able to give away free copies of the game to all your friends. I think this whole thing is OK, though instead of writing codes in PvZ to let it communicate with their server, instead, sadly, they choose to use ActiveX running in IE for this.

Some early posts reported that the registration process can be finished if you install an IE into the same Wine directory. Running IE on Wine is not impossible but hard, especially when IE4Linux project is no longer maintained.

The HOWTO


Now I have found that you can also finished the registration process using not IE but Firefox instead. Thanks for their being in the same free software camp, Firefox runs very well on Wine. Just grab a Windows copy and install it into the same Wine directory.

Then run Firefox in your Wine, go to PopCap's Registration Page, follow the instruction there to download and install the Firefox plugin. After that, close your Firefox (for some unknown reasons, the Firefox Wine process may still linger in the background after its window is closed, run "ps -ef | grep -i firefox" and kills the linger process if need).

Now starts PvZ by running its in your winefile. Because the PvZ and Firefox need to be started in a same Wine instance so they collect the identical hardware/system information. Register it following the normal process. It may fail at the first attempt to start a web browser, keep clicking the "Retry" button until it shows it's waiting for the registration process to finish (it's waiting for web browser plugin to return registration key). Start a Firefox instance from that same winefile program. Go to PopCap's Registration Page, now if your plugin is installed successfully, it should show a web page asking for your POP Order Number, give it and the registration process should finish successfully and the PvZ program should proceed to the game.

It's not rocket science, just nuisance.

PvZ runs very well in Wine during my 2 days play. I can say it's perfectly running on Linux for now.

Known Problems



  1. Sound effects may not work on some machines with some old Wine versions (try upgrade to latest Wine.)

  2. The full-screen mode will be always "full-screen," which means that if you have a non-4:3 screen, it will be slightly torted.

  3. The between-battle preview screen may be a bit slow sometimes, but as soon as the battle begins the speed returns to normal (not sure if it shows similar issue when running on a real Windows.)

Running Wine on MeeGo

One of the good thing of MeeGo is that it contains a standard X server along with full 2D/3D acceleration. Having a full X stack running on your netbook enables the possibility to use a good variety of exist fun Linux programs. Such as Wine.

In order to play Plants vs Zombies™ and Zuma™, I have installed Wine on MeeGo. Wine is not in MeeGo's official repo (yet), so I just downloaded all wine-*.rpm from Fedora's rawhide repo and the installation was a breeze. I hope MeeGo will have it's own Wine package soon, it should be quite easy to repackage with minor tweaks from Fedora's packaged Wine. Thanks Fedora project! Anyone in the community want to volunteer?

About Me

My photo
Santa Cruz, California, United States