Automated Backups on Tiger Using rsync
Pages: 1, 2
If you want to record shell command errors, perhaps for debugging the script, change the command to:
do shell script
"rsync -aE --delete ~ /Volumes/FW200/Backups 2>>~/rsyncErr.txt || echo -n"
This will append the error message to the file rsyncErr.txt in your home folder. This change is only of use if you need to see the shell error message.
As noted in the Tiger Terminal article, several characters such as spaces, quotes, apostrophes are command-line metacharacters. That is, if you type them in the command line they are interpreted as commands and not part of the name. If your backup target folder contains any of these characters, you will need to escape or quote the characters appropriately. Fortunately, AppleScript includes a quoted form command that escapes a text string appropriately for the command line. You would use it like so:
do shell script
"rsync -aE --delete ~ " & quoted form of "/Volumes/FireWire 200/My Backup ƒ"
Note the space between the tilde and quote in the above command. You need the space, in unquoted form, to separate the source and target file names. Remember, you only need to do this if the target folder name has a command-line metacharacter.
Try running the script you just made; it should back up your entire home folder to the target folder of your external drive.
Running Commands Automatically
We need a way of running the backup command automatically, without a person doing it. The easiest way of doing this in Tiger is to create a repeating alarm in iCal.
Launch iCal and create a new event. Set the event's "from" field to the time you want your backup to take place, probably early morning or some such time when you are not using your Mac. You will make the event repeat and trigger at the start, so the dates and "to" fields are not important.
Set the "repeat" field to "every day" and the "alarm" fields to "Run script," your script file, and "0 minutes before." Note that you will have to select "Other..." from the alarm file popup menu to select the script file.
Now iCal will run your backup script automatically at your event "from" time (for example, 2 a.m.). You must leave your external drive mounted and iCal running to trigger the backup script.
You can add other commands you wish to be run daily to your backup script. For example, you could add more rsync commands to back up other mounted volumes.
Richard Hough is a web developer for a Vancouver, Canada educational software company.
Return to the Mac DevCenter
Post your backup questions and comments here.
You must be logged in to the O'Reilly Network to post a talkback.
You must be logged in to the O'Reilly Network to post a talkback.
Showing messages 1 through 60 of 60.
-
LBackup
2009-01-14 17:43:36 lucidsystems [Reply | View]
If you wish to backup a Mac OS X machine, then consider taking a look at LBackup
-
applescript error
2007-03-12 10:00:34 jacob'o'reilly [Reply | View]
Hi there,
I'm getting an error when I try to run this from applescript "Can't make current application into type string." It work fine from the terminal but not from applescript, here is the script
do shell script
"rsync -aE --delete ~/backup_test /Volumes/JKEENAN/backup || echo -n"
Any advice would be great
Cheers
Jack
-
Whats wrong with my syntax?
2006-04-06 20:19:17 machine117 [Reply | View]
I am trying to rsync using the command found on page one, but constantly get this error:
My commandrsync -aE --delete ~/Desktop /Volumes/WD250/BU
rsync: -E: unknown option
rsync error: syntax or usage error (code 1) at main.c(1023)
What am I doing wrong? I noticed in the rsync man that it uses -e with the --delete flag, not -E, but when I use that, I get mixed results. Files copy over, but with don't keep any settings such as what program opens them, and applications come out useless. Any help appreciated! -
Whats wrong with my syntax?
2006-12-17 22:58:23 valdi [Reply | View]
Im running the following command:
rsync -aE --delete /Users/MacBook/Desktop/me/ /Volumes/HD/ 2>>~/rsyncErr.txt || echo -n
and its working fine, but my problem is the volume HD when i drag it to terminal sometimes it shows HD, sometimes it shows HD-1 and sometimes HD-2, im really confused .. cause if its hd-1 and its written in the command as hd-2 it will not take a backup.
note that HD is a folder located in the Xserve raid and its connected to my computer through the ethernet network!
how can i avoid this problem!
thanks -
Whats wrong with my syntax?
2006-04-06 20:34:39 machine117 [Reply | View]
Double correction: I found the error of my ways. After reading a different article, I remembered that a while back I had installed and subsequently removed RsyncX, not knowing it changed my syntax. All is well with this instead:
-a --eahfs --delete
-
rsync backup script
2006-04-01 08:55:07 TheBronx [Reply | View]
Hi all,
after been a couple of times to lucky with restoring my apple it's time for a automated backup. As addition to the ver usefull articel I want to post a simple script. It's not the best and nicest but it will do the job!
- Also I have read a lot of articles about mounting and unmouting the external harddisk, but it won't seem to work fine. I don't want my partinioned harddisk on my desktop all the time. So with this script it will fire him up and do the backup, gives an result output on my desktop (to check if the beackup went ok, and unmounting the external HD, and restoring the silence.
p.s. check with the command "diskutil list" for the right external disks. And change the volumename with you partition name.
Well have fun, and if you guys have additions let me know.
cheers,
Henk
do shell script "echo ==rsync Backup script == >>~/Desktop/rsync.txt"
do shell script "date >>~/Desktop/rsync.txt"
do shell script "ls /Volumes/"
if result is not {"Macintosh HD", "volumename"} then
do shell script "diskutil mountDisk disk1s7"
do shell script "echo ==Disk Mounted== >>~/Desktop/rsync.txt"
do shell script "echo ==start rsync logging== >>~/Desktop/rsync.txt"
end if
do shell script "rsync -aE --progress /Users/volumename/ /Volumes/volumename/rsync >>~/Desktop/rsync.txt || echo -n"
do shell script "echo =rsync Backup Ended== >>~/Desktop/rsync.txt"
on idle
return 60
end idle
do shell script "diskutil unmountDisk /dev/disk1s7"
do shell script "diskutil unmountDisk /dev/disk1s3"
do shell script "diskutil unmountDisk /dev/disk1s5"
do shell script "echo =====Ready!===== >>~/Desktop/rsync.txt"
-
rsync backup script - now Incremental with little storage impact
2006-12-04 14:18:08 Inspector71 [Reply | View]
Hi TheBronx,
Thanks for your script - it really helped. I was wanting to do a similar thing, but I found out you can do incrtemental backups that don't copy the files across each time - only the hard links (so you need very little extra storage space); so I used yours and amended it as follows:
I got it to check if the HDD was available or not before continuing (I'm using a MacBook so I can't guarantee that the backup drive will be plugged in) and to exit if not.
I also implemented incremental backups so that I have copies of the state of my source for the last 9 times it was backed up (in case I backup an error). I liberally used the code on this page: http://www.macosxhints.com/article.php?story=200310240137579
Lastly, I fiddled with some of the shell scripts to exclude .Trashes and Volumes folders as well as the Downloads folder on my system. The /Volumes exception is important as the first time I did this I not only backed up my internal HDD but also the other partition on my external one, as rsync followed the symbolic links.
I also found that Spotlight kicked in and started indexing my backup very quickly, with enormous performance consequences. I recommend you start off by disabling indexing on your backup drive by following the advice here: http://www.thexlab.com/faqs/stopspotlightindex.html, then try running the script.
As TheBronx suggests, find out what the name of your disk is using diskutil list and replace it into here. My partition for backing up is called Backup and I have folders 01-09 for increments within that. I have it set to backup my entire internal HDD (/).
The only problem I am having now is that unless I run it as root (which I haven't done yet) I lose all file ownerships and everything reverts to my user.
Here it is, have fun:
do shell script "echo ================================ rsync Backup script ================================= >>~/rsync.log"
do shell script "date >>~/rsync.log"
-- Make sure Firewire HDD is available, otherwise quit
do shell script "ls /Volumes/"
if result does not contain "Backup" then
do shell script "echo ==Mounting Backup Disk: >>~/rsync.log"
do shell script "diskutil mountDisk disk1s2"
do shell script "ls /Volumes/"
if result does not contain "Backup" then
do shell script "echo : Could Not Mount Backup Disk - Exiting =============!!!============= >>~/rsync.log"
return
else
do shell script "echo : Backup Disk Mounted >>~/rsync.log"
end if
end if
-- Rotate Backup dirs, deleting the oldest
do shell script "echo ==Rotating Backup Directories== >>~/rsync.log"
do shell script "rm -rf /Volumes/Backup/09"
do shell script "mv /Volumes/Backup/08 /Volumes/Backup/09"
do shell script "mv /Volumes/Backup/07 /Volumes/Backup/08"
do shell script "mv /Volumes/Backup/06 /Volumes/Backup/07"
do shell script "mv /Volumes/Backup/05 /Volumes/Backup/06"
do shell script "mv /Volumes/Backup/04 /Volumes/Backup/05"
do shell script "mv /Volumes/Backup/03 /Volumes/Backup/04"
do shell script "mv /Volumes/Backup/02 /Volumes/Backup/03"
--do shell script "cpio -pdl /Volumes/Backup/01 /Volumes/Backup/02"
do shell script "cd /Volumes/Backup/01 && find . -print | cpio -pdla /Volumes/Backup/02"
do shell script "echo ==Directories Successfully Rotated== >>~/rsync.log"
-- Do rsync
do shell script "echo ==start rsync logging== >>~/rsync.log"
do shell script "rsync -aE --progress --delete-after --exclude=Volumes/ --exclude=Downloads/ --exclude=.Trashes / /Volumes/Backup/01 >>~/rsync.log || echo -n"
do shell script "echo =rsync Backup Ended== >>~/rsync.log"
on idle
return 60
end idle
do shell script "diskutil unmount disk1s2"
do shell script "echo =====Backup Complete===== >>~/rsync.log"
Cheers,
Marcel
-
rsync backup script - now Incremental with little storage impact
2006-12-04 14:25:36 Inspector71 [Reply | View]
OK, so I think the last script will work OK if you are only backing up stuff from within your home dir, but I decided to convert my script to a shell script so I could run it as root from crontab.
I have also updated the rsync options a bit to preserve permissions and access dates. The cpio command needs a little work as it seems to change perms on the files it 'copies' (or relinks to).
Anyway, here is the shell script (saved in /etc/backup.sh, called through crontab):
echo ================================ rsync Backup script ================================= >>/var/log/rsync.log
date >>/var/log/rsync.log
if [ -d /Volumes/Backup ]
then
echo ==Mounting Backup Disk: >>/var/log/rsync.log
diskutil mountDisk disk1s2
if [ -d /Volumes/Backup ]
then
echo : Backup Disk Mounted >>/var/log/rsync.log
else
echo : Could Not Mount Backup Disk - Exiting =============!!!============= >>/var/log/rsync.log
exit 2
fi
fi
echo ==Rotating Backup Directories== >>/var/log/rsync.log
rm -rf /Volumes/Backup/09
mv /Volumes/Backup/08 /Volumes/Backup/09
mv /Volumes/Backup/07 /Volumes/Backup/08
mv /Volumes/Backup/06 /Volumes/Backup/07
mv /Volumes/Backup/05 /Volumes/Backup/06
mv /Volumes/Backup/04 /Volumes/Backup/05
mv /Volumes/Backup/03 /Volumes/Backup/04
mv /Volumes/Backup/02 /Volumes/Backup/03
cd /Volumes/Backup/01 && find . -print | cpio -pdla /Volumes/Backup/02
echo ==Directories Successfully Rotated== >>/var/log/rsync.log
echo ==start rsync logging== >>/var/log/rsync.log
rsync -aEp --progress --delete-after --exclude=Volumes/ --exclude=Downloads/ --exclude=.Trashes --exclude=/dev --exclude=/tmp --exclude=/private /Volumes/Backup/01 >>/var/log/rsync.log || echo -n
echo =rsync Backup Ended== >>/var/log/rsync.log
sleep 2m
diskutil unmount disk1s2
echo =====Backup Complete===== >>/var/log/rsync.log
One problem I am having is that the HDD won't unmount afterwards; I think a process must be hogging it. If anyone can clean this up a bit the help would be appreciated :)
Cheers
Marcel
-
One more bug: over-writes files of different type
2005-10-04 21:01:35 bioinfotools [Reply | View]
One more bug to add to the list.
The current version of rsync on OS-X 10.4.2 (and presumably earlier versions too) has a bug where --backup doesn't if the files have the same name but are of different type. For example, if the source has a directory "mythings" and the destination has a file of the name, rsync tries to place the directory in place of the file (as it should), but applying --backup will not cause a backup copy of the file to be made, instead the file will get silently over-written and lost. The situation occurs in reverse for files replacing directories. These situations may be rare but they do occur and it does cause a complete loss of the destination data involved.
Note also that rsync considers empty directories not to be worthy of being retained (potentially nasty if its a directory that some program expects to be present that is occasionally empty, like a cache directory or the like).
Apparently the latest version of rsync has the --backup bug fixed, but this version doesn't handle resource forks, etc.
Personally, I off looking for something with fewer issues. Let's hope there is something out there... Any suggestions?...
-
7 day incremental backup
2005-09-30 06:54:17 annibee [Reply | View]
Hi
I love this tutorial and I've had a lot of fun scheduling various backups from applescript in ical. Its funny how the MACOSX wrapping seems to make everything easier than say using perl in UNIX ;)
Anyway my question is ..... How do I implement a full backup every Sunday and then incremental backups everyday at midnight using applescript and ical or should I just script the whole thing as listed on http://rsync.samba.org/examples.html
Thanks in advance
Anna :) -
7 day incremental backup
2005-10-01 09:26:01 Corvus [Reply | View]
Hi Anna. I guess I wasn't clear on howrsyncworks; it creates
full backups by copying only the data that has changed since the last backup.
You don't need to do "full" backups ever again.
Suppose you ranrsyncfor the first time on Sunday and it
made a copy of your entire 80 GB hard drive. You work on some projects;
changing a few bytes in a multi-megabyte database, add a few lines to a
text file, add and delete some records from a contact list. Whenrsync
runs again that night, it will compare each file on your hard drive, byte-for-byte,
with that on the backup drive, and only copy over the bytes that have changed.
Only the few bytes changed in your database get copied to the backup.
The new lines get added to the backup's copy of your text file, the new
contacts get added to your contact list and the old ones get deleted, without
affecting any of the other records.rsyncmakes the fewest
changes to the target files so they are identical to the source files. It really
obsoletes the old full/incremental backup method. -
7 day incremental backup
2005-12-01 07:10:34 KevinStout [Reply | View]
Hi Corvus,
You seem to be missing the point of an incremental backup. The point is to be able to do a restore to a point in time. For example you deleted / changed a critical file on tuesday, but do not realize it until friday. From waht I understand, if you rsynced everyday, the orginal or unchanged file is now lost forever. -
7 day incremental backup
2005-12-01 07:10:21 KevinStout [Reply | View]
Hi Corvus,
You seem to be missing the point of an incremental backup. The point is to be able to do a restore to a point in time. For example you deleted / changed a critical file on tuesday, but do not realize it until friday. From waht I understand, if you rsynced everyday, the orginal or unchanged file is not lost forever.
-
rsync applescript quoted form
2005-09-23 09:42:22 annibee [Reply | View]
If I type the following at the command line
rsync -aE /Volumes/FMRILDAP\;SAMBA-2/archive-list ~/BACKUP
It works perfectly.
However if I include the same in an applescript I get the error Expected """""" but found unkown token and the ; is highlighted. I tried the suggeseted fix for the metacharacters but I get the same error.
Thanks
-
Beware - this script deletes before it copies
2005-08-25 09:09:38 thomaslunde [Reply | View]
As written, the proposed rsync command line deletes files which are not on the backup volume before it writes over new files. If you are like me, you rename new versions of files -- and therein lies the danger. If there is a problem with the backup drive in mid-backup, you can lose your old safe version and have the new changed version not written out.
To fix this, just use the
--delete-after
option to rsync.
-
Rsync part works fine, but what's up with Applescript?
2005-08-22 08:28:14 bbushman [Reply | View]
I'm happily able to run these scripts in the terminal window, but am trying to use them via applescript as mentioned in this article. Same error I've seen noted earlier here - the script will compile, but when I run it I get back this error "can't make current application into type string".
I've read this tech note at apple to help investigate the problem:
http://developer.apple.com/technotes/tn2002/tn2065.html#TNTAG3
Coping stategies have included:
1. keep the do shell script out of any tell blocks
2. putting it within a "tell me" block
3. running it all as one quoted command
4. storing my file paths as string vars in POSIX form (ie. "/Volumes/MyHD/Library/WebServer/etc.) and then concatenating them into the shell script
5. storing my file paths as string vars in applescript form (ie. "MyDisk:Library:WebServer:etc.) and then concatenating them into the shell script using "quoted form of the POSIX path"
6. adding /bin/bash -c to make sure they run in the same shell as my terminal instead of in sh
7. Kicking and screaming like a little girl while shouting obscenities
Also note: I've installed for now the RSyncX version, as recommended in this forum by "mechanist" on this forum. Again, works peachy in terminal, no dice in Applescript.
What am I doing wrong here, folks?
Current version of the script look like this:
-- initial vars
set buildCart to "TheLoraxG4PB-HD:Users:bbushman:DEV:brandsourcedev:website:trunk:src:cart"
set siteCart to "TheLoraxG4PB-HD:Library:WebServer:Documents:brandsourcedev"
set mySrc to quoted form of the POSIX path of buildCart
set myTgt to quoted form of the POSIX path of siteCart
-- sync 1
do shell script
"/bin/bash rsync -a --eahfs " & mySrc & myTgt & " || echo -n"
-
Rsync part works fine, but what's up with Applescript?
2005-09-23 09:58:15 annibee [Reply | View]
The same thing happened to me and so I took the carriage return away from the end of do shell script and then it worked. :) -
Rsync part works fine, but what's up with Applescript?
2005-08-22 10:51:38 bbushman [Reply | View]
Scratch that - I put it in the middle of a function and now the applescript portion works fine... and rsync appears to be running, because it's making my error file in my user folder. But: nothing actually sync's. What could be wrong with my shell script? -
Rsync part works fine, but what's up with Applescript?
2005-08-22 08:40:09 bbushman [Reply | View]
PS: just read the betalogue post related to this topic (thanks pierre!) Perhaps adding /usr/local/bin/ to the beginning will help after I stop getting my other error, but for now, I'm still (of course) getting the AS error.
Of note (which I'm assuming is obvious to anyone who knows what they are doing) is that in my event log in script editor, I"m left with this after I try running the script:
tell current application
do shell script current application
"Can't make current application into type string."
Anyone?
-
Discussion on rsync bug with creation/modification dates
2005-08-22 05:59:46 Pierre Igot [Reply | View]
Readers of this article and discussion might be interest in this post and discussion on Betalogue:
http://www.betalogue.com/2005/08/18/more-on-automated-backups-with-tiger-using-rsync/
-
A potential problem
2005-08-04 09:58:49 EdHagen [Reply | View]
Here is a potential problem with this strategy: If your backup drive is disconnected, then rsync will create a folder on your main drive and copy all the files to that. This can use up all your drive space, causing all kinds of problems.
One way to avoid this problem is to save the backup script on the backup drive. That way it can't run if the drive is disconnected. Another strategy would be to add some applescript or shellscript to detect if the drive is connected before running rsync. -
a solution
2005-08-04 15:44:03 garbanzito [Reply | View]
i use a nice script called rsyncbackup that covers this problem and more; the documentation at http://rsyncbackup.erlang.no/ covers the details, but essentially, the rsyncbackup user can define a test that must be satisfied before proceeding; i use traceroute to server as my test; rsyncbackup also allows setting default parameters, defining named backup sources & destinations, mailing of error logs, etc.; i use it with the RsyncX version of rsync on Panther with great success
for scheduling, i would consider anacron as well as iCal
-
also having errors running script
2005-07-25 11:46:11 KJC [Reply | View]
I keep getting the ".....into type string" error in Script Editor. Any ideas? I'm trying to copy the contents of "Source" to the folder "Target". Both folders are on my Desktop
Here is what I have in Script Editor:
do shell script
"rsync -aE --delete /Users/kevinc/Desktop/Source/ /Users/kevinc/Desktop/Target/>>~/rsyncErr.txt || echo -n"
-
Also having errors running script
2005-07-25 19:01:34 rdomidion [Reply | View]
You need a space and a "2" before the ">>". Your command should look like:
do shell script "rsync -aE --delete /Users/kevinc/Desktop/Source/ /Users/kevinc/Desktop/Target/ 2>>~/rsyncErr.txt || echo -n" -
Also having errors running script
2005-08-22 11:40:07 bbushman [Reply | View]
problem solved. Had some concatonation errors in my shell script, and setting the result of the script to a var took care of the applescript error.
if anyone can tell me why that is, and why several demos online don't have this problem, I'd be grateful.
-
With ssh it looks better
2005-07-25 00:05:55 Alexander_Lamb [Reply | View]
Hello again.
I tried using ssh to perform a backup (instead of mounting a remote drive). Well, apart from some issued because I don't have a DNS server (so I had to put an IP address) and a small bug with the -b option (path is wrong in that case), the rsync went to the end without any problem.
Therefore I believe there is a real problem with rsync when mounting a drive from the network (smb or afp).
Regarding my remark on another reply, saying things were horribly slow while using rsync, well, it could be related to Spotlight. Indeed, although rsync is not supposed to change anything on your files, I have the feeling it does which triggers a tremendous activity from Spotlight.
Alex
-
More issues with rsync
2005-07-24 09:33:36 Alexander_Lamb [Reply | View]
Hello,
Just tried again between a powerbook and an iMac both running latest Tiger OS. I mounted a share, typed the password, etc... and ran the following command:
sudo rsync -Eavb --delete --backup-dir="/Volumes/DATA-1/alamb_bck/rsync_bck" --exclude="alamb.sparseimage" /Volumes/DATA/Users/alamb "/Volumes/DATA-1/alamb_bck"
Goes well for one or two minutes (but still copying too much accross) and then crashes with:
alamb/Library/Caches/com.apple.AddressBook/MetaData/72F62263-1445-11D8-8E70-000A95CC81E4:ABPerson.abcdp
Invalid file index 1913126912 (count=29712)
rsync error: protocol incompatibility (code 2) at /SourceCache/rsync/rsync-20/rsync/sender.c(159)
rsync: writefd_unbuffered failed to write 103 bytes: phase "unknown" [generator]: Broken pipe (32)
rsync error: error in rsync protocol data stream (code 12) at /SourceCache/rsync/rsync-20/rsync/io.c(909)
I also had many of those:
rsync: chgrp "/Volumes/DATA-1/alamb_bck/alamb/Library/Caches/Java Applets/cache/javapi/v1.0/file/bApplet.class-1195b694-2c622a6a.class" failed: Operation not supported (45)
Ok, I should have excluded the cache, but the symptom is the same.
Alex
-
More issues with rsync
2006-04-19 03:51:15 gulivieri [Reply | View]
i use rsync command on os x 10.4 to backup data on network volume (Lacie XP). the command was launch continuously (every ~300 s) by a script. In all the client machine the script gone rigth for some day, afterword the new file don't appear on Lacie disk. However from terminal i found on /Volumes two directory named DATA (the old one) and DATA-1 (new one), where:
DATA don't contain all the folder but only the one relative to a single machine
DATA-1 contain all the folders
i had to rename the destination folder on the script, but i don't understand what's happen
some advice? -
More issues with rsync
2006-03-28 18:33:56 avgill [Reply | View]
Hey Alex, I'm seeing the same error on a Powerbook with RsyncX 2.1 archiving data from the laptop to a mounted SMB share.
Did you ever find out what causes this? Just for the record, I'm using sudo to run the backup script so permissions shouldn't be an issue. Google hasn't been of much help so far besides just leading me to this discussion =/
username/Desktop/dir1/contacts/oct13_2005.rge/Calendar/0ODKOK22JTCNLST5N2KKBAQBXJ
Invalid file index 1970210816 (count=39213)
rsync error: protocol incompatibility (code 2) at sender.c(562)
rsync: writefd_unbuffered failed to write 69 bytes: phase "unknown": Broken pipe
rsync error: error in rsync protocol data stream (code 12) at io.c(666)
Thanks,
Avtar -
More issues with rsync
2005-07-24 11:49:26 Corvus [Reply | View]
Hi Alexander_Lamb. I have had similar problems when I try backing up files that I don't have read access to. The solution was running the rsync command as root, but it appears you are doing that withsudo. Can anyone think of another solution?
I wanted to reiterate iikka's excellent suggestion given below to exclude cache files; that didn't work for you either?
-
Automating the backup
2005-07-24 07:21:03 mrchucho@yahoo.com [Reply | View]
Try using /etc/daily.local instead of iCal. The script will run every night around 3am with output going to /var/log/daily.out.
-
rsync to, e.g., Linux server with -E
2005-07-24 01:04:31 iikka [Reply | View]
The rsync with -E (--extended-attributes) does not work directly
with stock rsync included in most Linux distros. This is confirmed
by the rsync man pages for the rsync on Tiger, which state the
following:
-E, --extended-attributes
Apple specific option to copy extended attributes,
resource forks, and ACLs. Requires at least
Mac OS X 10.4 or suitably patched rsync.
So, in order to use rsync to make backup from Mac to Linux, the Linux-side
rsync must be patched to support the -E attribute. Here's how to do that
(all commands given on the Linux machine):
1. Get rsync source code
curl -O http://samba.org/ftp/rsync/old-versions/rsync-2.6.3.tar.gz
(This is the version supplied with Mac OS 10.4, the patch may well
work on newer versions, but I haven't tried that.)
2. Get the required patch files from darwinsource:
curl -O http://www.opensource.apple.com/darwinsource/10.4.1/rsync-20/patches/EA.diff
3. Unpach the rsync source
tar -zxvf rsync-2.6.3.tar.gz
4. Move to the unpacked directory and apply the patch
cd rsync-2.6.3 && patch <../EA.diff
5. Configure & compile
./configure && make
6. Install (as root to replace the current rsync installation)
su
make install
All this takes less than 10 min. It requires, that you have root
access to the Linux box, though. I made an alias like the following:
alias syncme rsync -vaE --exclude=Trash --exclude=Library/Caches
~/* my.backupserver.com:backupdir/
The command could also be easily added to a cron job.
Also, please note if rsync is used on Mac OS X from a mounted file
system to another, i.e. neither the source or destination path
contains a colon (as described on the original article), it works
regardless of the mounted filesystem type and location. That is,
backup to Linux would also be possible without the above steps by
mounting the target filesystem by, e.g., SMB or NFS, and following
the advice on the original article. The patch is only required,
when two copies of rsync are involved in the process (i.e. the local
and remote copy), in which case both need to support the -E flag.
-
rsync to, e.g., Linux server with -E
2005-07-29 20:09:26 mrkahuna [Reply | View]
just an FYI, I had to change step 5 to:
./configure --enable-ea-support && make
as the default appears to be to not enable -
rsync to, e.g., Linux server with -E
2005-07-24 11:29:23 Corvus [Reply | View]
As you say, that would only be required if you want to sync to a remote machine, one not mounted as a local filesystem. The article only describes backing up to a locally-mounted filesystem. Backing up to a remote machine is quite an involved project, worthy of an article in itself.
However, your post mentions one brilliant idea which I urge everyone to adopt. The--exclude=Library/Cachesoption will skip cache files, which are huge and do not require backing up. This can be incorporated into the original command as:
rsync -aE --delete exclude=Library/Caches ~ /Volumes/FW200/Backups
Would it not be better to use--exclude=Library/Caches/, with a trailing "/", to include the (admittedly, unlikely) case of a Library file named "Caches"?
In any case, this simple change will save me hundreds of MB in backup space. Thanks iikka!
-
Will it work on SMB, NFS or AppleShare backup volumes?
2005-07-23 17:00:04 rberger [Reply | View]
Will rsync or RsyncX work if one had an SMB or other network mounted directory for the backup volume (ie instead of an external drive to keep the backups on)?
Particularly will it work on Network drives that are not OS X based (ie on a Linux server)?
I'm concerned about the metadata being properly handled on the non OS X drives. -
Will it work on SMB, NFS or AppleShare backup volumes?
2005-07-25 07:18:29 tlaurenzo1 [Reply | View]
It should work. Just be aware that the slick rsync algorithm that differentially copies files will cause an excessive amount of bandwidth to be utilized if copying to a locally mounted remote filesystem. (Do to the nature of the algorithm, this should result in virtually the entire target file on the remote file-system being sent back to the server for any file where the time and/or date are different -- Then the differences are calculated and sent the other way).
I haven't tried it, but you probably will find that it lowers your bandwidth needs by specifying the "-W" option.
-
Error running script?
2005-07-23 09:21:39 ljhardy [Reply | View]
When I try to run this script out of the script editor I receive the following error:
Can't make current application into type string.
I cut and pasted the rsync command from the article and changed the Volumne name accordingly.
What did I screw up here? :-) -
Error running script?
2005-07-24 04:57:33 Corvus [Reply | View]
"... into type string" errors are often a problem with quoting. Are you usingquoted form? You'll have to show us the entire script; just copy it into a reply.
-
how do you restore the files?
2005-07-23 08:39:21 thomasyung [Reply | View]
Great article on backing it up. Can you write up something for how to restore those backups? Verifying that the backups and their restored versions are the same presents its own issues as well. -
how do you restore the files?
2005-07-24 05:37:03 Corvus [Reply | View]
Good point thomasyung, I apologise for not including this in the article.
The backup files are just copies; to restore a single old file, simplty drag it normally from the Finder.
To restore everything from backup, run rsync in reverse. In my case:
rsync -aE --delete /Volumes/FW200/Backups/rdhough/ ~.
Note the "/" at the end of the backup folder name! This means to back up only the contained files, not the enclosing folder. You need to do this since your machine will already have a home folder on it.
This is a rather dangerous command to experiment with. If you just want to test it on a sample folder, do something like:
rsync -aE --delete /Volumes/FW200/Backups/rdhough/Documents/SampleFolder ~/Documents.
-
Beware rsync's huge bug...
2005-07-22 16:18:17 mechanist [Reply | View]
Rsync is fantastic for backups... except if you use the version included with Tiger, as described in this article, you'll likely encounter a big honking bug.
Beginning with Tiger there's the "-E" option that specifies "extended" attributes such as resource forks should be copied. Great, right? But if you use "-E" and "-a" (archive) together, as you would for a backup, the modification dates of the copied files will be incorrect.
With "-a" the modification date of the copied file should be the same as that of the original. But with "-a" and "-E", and when the file has a resource fork, the modification date of the copied file will match the date OF THE COPY, not of the original file.
I reported this to Apple for 10.4; they marked it as a duplicate. The bug is still present in 10.4.2.
For now you're better off using the version of rsync included with RsyncX. -
Beware rsync's huge bug...
2005-09-28 02:02:32 maceis [Reply | View]
This is true, but only files which have a ressource fork are affected. -
Beware rsync's huge bug...
2005-07-23 23:35:44 KonradHosemann [Reply | View]
The rsync od RsyncX has the same problem, as well as hfsrsync from darwinports, which is the same as the RsyncX rsync, I guess.
Do you know an rync which definetly doesn't have this bug? -
Beware rsync's huge bug...
2005-07-26 15:00:58 mechanist [Reply | View]
In my experience, the version of rsync included with RsyncX definitely does not have this bug. I use it all the time and have observed no problems like the one I described.
With the RsyncX version the command you'd use is:
rsync -a --eahfs SOURCE DEST
This should be equivalent to "rsync -aE" with Tiger's stock rsync.
In my usage the modification dates are correct, both for copying to a different drive on the same system and for copying to a remote system over ssh. -
Beware rsync's huge bug...
2005-07-22 16:31:15 Corvus [Reply | View]
Thanks Mechanist; I've been using rsync for months and never noticed that. rsync compares file data, it doesn't use the modificatin date, but still...
I'll try it with RsyncX. -
Beware rsync's huge bug...
2005-07-23 15:10:41 joseph p.muscara [Reply | View]
I've been using RsyncX for ages, but the problem I've had lately is that I can't get it to run properly as a cron job. It consistently causes a kernel panic on my 10.4.2 Server. I was hoping that switching to rsync would solve my problems.
If anyone has any suggestions about using RsyncX as a cron job, I'd appreciate it; I did use the fix for the bug in the script generator to make sure the cron job was using /usr/local/bin/rsync. -
Beware rsync's huge bug...
2005-07-24 05:06:54 Corvus [Reply | View]
Hi Joseph. Is your problem with rsync or cron? If it's the latter, my suggestion is to use iCal events as shown in the article, not cron.
I know Linux users will be aghast, but I have found iCal events much easier to create, manage, control, and understand than cron. And there is negligible difference in performance if you're just running the command once a day or so. -
Beware rsync's huge bug... (and more)
2005-07-24 02:49:48 Alexander_Lamb [Reply | View]
Several remarks here:
1)
I use file vault. To avoid backuping the sparseimage in addition to the decrypted data, I use this:
sudo rsync -Eavb --delete --backup-dir="/Volumes/alamb_bck/rsync_bck" --exclude="alamb.sparseimage" /Volumes/DATA/Users/alamb "/Volumes/alamb_bck"
However, since I don't want to leave an unencrypted backup on a machine somewhere on the network on on my portable drive, I create an encrypted sparse image that I mount for the destination.
2)
I managed this mounting from a Windows SMB volume. Doing this you avoid the ressource forks issues. (since you are backuping to hfs+ and not ntfs).
3)
There is a bug (Apple calls it a feature) which makes rsync recopy systematically ALL the resource forks of the files.
4)
I consistently had "broken pipe" errors when I had my source volume (where my home is) as hfs+ journaled WITH case sensitivity.
5)
Some files (other than resource forks) get systematically backed up (iPhoto photos for example). No way to tell why. Obviously it takes ages.
6)
I never managed to have rsync run consitently with crashing on broken pipes or other issues. For me, it is not ready to prime time on MacOSX (RsyncX had the same problem).
7)
It looks like there are fewer problems when backuping through rsh with ssh.
8)
There must be a squeduling issue with rsync. I have a power book with 1 gig ram but when I am runing rsync I hardly can move the mouse or type a text in text edit!!
9)
Someone on a forum suggested it was the --delete creating problems. But how should I backup correctly without --delete and without at least archiving what is deleted??
Well, I am at the point where the time I wasted trying to make all this work cost me more than a 50$ sync program such as ChronoSync, SynchronizeProX, TriBackup or YouSynchronize.
Any hint as to where I should go to now??
Alex -
Beware rsync's huge bug... (and more)
2009-04-23 19:11:22 lucidsystems [Reply | View]
The version of rsync shipping with Mac OS X (10.5.6 and earlier) has various issues. However, if you install the latest version of rsync ( currently 3.0.5) with the appropriate patches for Mac OS X, it is a great backup utility.
The URL below contains instructions for downloading, patching, compiling and installing the latest version of rsync on Mac OS X. Once you have developer tools installed, this e process will typically take less than five minutes. Details at the following URL.
http://connect.homeunix.com/lbackup/developer/rsync_hfs
I hope this helps.
