Note : You follow this guide at your own risk. I’m in no way responsible for any issues that arise as this is meant to be a guide that you can use to come up with your own backup plan for your own circumstances.

If you are a photographer you’ve probably either already got a robust backup workflow on the go or have at least been thinking about it. This short article will give you a little idea about how to backup your photography using a tool called rsync which is available on OSX and Windows. This little guide will focus on OSX.

Let’s take an example of a situation where you might like to backup some photography. Say for example that you have a folder on your machine that you would like to backup to your external drive. This can be a hassle doing it manually; finding files and folders, dragging them into your external hard drive, double checking that you have not missed anything etc etc. The problem with this approach is it is very easy to end up with a folder on your machine that is out of sync with the backup folder on your external hard drive. Synchronisation is a term you’ll have encountered before if you’ve used Apple, Google and Microsoft products. The idea is that you keep two things identical through synchronisation. Rsync is a perfect tool for this. I’ve used it in the past for maintaining web servers and other nerdy things (like we’re not already nerding out with this article.)

Rsync lets you specify which folder to sync from and where to sync it to. In our case our ‘Photography’ folder on our computer into our ‘Photography’ folder on our external hard drive.

Firstly we need to check that we have the latest version of rsync available to us. Apple has a problem when it comes to keeping GPL software up to date, so the first thing is to go into Terminal and type

rsync

This will output some information, scroll to the top of the window and look for the version number. If it is anything less than 3.1.0 then you are out of date. Using the latest version will always be beneficial and version 3 has much greater memory performance than version 2.9+.

Updating Rsync

If you are on an old version of rsync you’ll want to follow these instructions on how to upgrade rsync.

1. Install Homebrew.

Homebrew is a software package manager. This will enable us to get the latest version of rsync. Follow the instructions on the Homebrew website for installation.

At the time of writing I did the following in Terminal

ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go/install)"brew help
brew doctor

2. Install rsync, in Terminal

brew tap homebrew/dupes
brew install rsync

3. Update the paths file in your private folder

The private folder is hidden so you will need to run the following command in Terminal

sudo chflags nohidden /private/

This will expose the private folder in Macintosh HD.

Look for the paths file (/private/etc/paths) and edit, moving /usr/local/bin ahead of /usr/bin. This will allow your system to pick out the latest rsync version and ignore the old version that came preinstalled with your OS. Save the file and close.

Set the private folder to hidden again.

sudo chflags hidden /private/

Automator and Rsync

Now we can start looking at some rsync commands and executing them with Automator.

Automater is an application that comes with OSX that enables you to do all sorts of amazing things in an automated way, hence the name. In this example we’re going to use it to run rsync commands and then save this little commandlet as an application that we can run whenever we want to sync our photography folders.

In this example we are going to sync our /Documents/Photography folder with our Photography folder on our external hard drive. The hard drive is called Secondary. The rsync command will look something like this, of course yours might look different depending on where you are syncing from and to.

rsync -aE --delete ~/Documents/Photography "/Volumes/Secondary/Photography/"

You can run this in terminal and it will effectively delete anything that is in /Volumes/Secondary/Photography/ that is not in the folder on your hard drive (/Documents/Photography) and also move across anything that is in /Documents/Photography which is not in /Volumes/Secondary/Photography/.

By using Automator we can make this a much easier and less fiddly process.

1. Open Automator, choose New Document and select Workflow.

Create workflow

Create workflow

2. In the Variables panel search for Run Shell Script and double click it.

Run Shell script

Run Shell script

3. In the right panel type in the command detailed earlier.

Take a look at the screen shot below to see what our Automator window should now look like.

backing up using rsync and automator

backing up using rsync and automator

4. The next stage is to save this as an Application file which will enable us to run it like an application.

Save as Application

Save as Application

You can save it wherever you like but it’s handy to save it on the backup disc because that is where you are synching to.

To test it, double click on the file that you have just created and in the top right hand corner, on your status bar you’ll see a little cog running, click on it and it’ll tell you that a process is running. The cog will disappear when the process has been finished.

Before doing this for real of course I would recommend using some test folders before running this on anything important. I’m using this successfully and the same process can be used for external hard drive to external hard drive. With some reading and experimentation you can do some great things with rsync and Automator.

Credits

I referenced a couple of articles in order to put together this guide. Articles that I read when trying to put this together for myself. There were bits and pieces across different blogs and I’ve used that information to create this article which will hopefully provide a more streamlined tutorial on getting something like this to work in OSX Mavericks.

How to speed up rsync on your Mac (and stop the dreaded “building file list…”)
rsync + Automator = free and easy backups for your Mac
Brand new rsync for OSX

Issues

During the refinement process of my own workflow I discovered an issue with synching files that had extended attributes. Read the wikipedia page for information on what extended attributes are. The -aE flag means we’re wanting to archive files and extended attributes, however rsync seems to have problems with extended attributes of certain files. By removing the E I managed to get a sync to work. It seems certain movie files have meta information stored in other files. A quick Google search throws up a fair few results and I have yet to find a better solution.




5 responses to “A Photographers Guide to Backing Up Using RSYNC”

  1. ilium007 says:

    the latest rsync from home-brew no longer uses -E for extended attributes, use -x instead. See the man page for more.

  2. Where you edit “/etc/paths”, also edit /etc/manpaths. That way a “man rsync” at the Terminal window gives the man page for the ‘brew’ installed version of rsync and not the OS X version.

    You don’t have to chflags on /private if you are using the Terminal window and ‘vi’ to edit these files. The hidden flag only applies to Finder. Terminal windows see all of the files. Just edit /etc/paths and /etc/manpaths to place the /usr/local path ahead of the other paths in the file, one path per line.

    See my article for some more info on backup strategies.

    http://www.travelphotographers.net/article.php/photo-storage-management

  3. JollyD says:

    PLEASE NOTE: comment 1. is spam and install an insidious piece of adware that will install deep into your system and destroy it when you try to uninstall.

    Please remove that comment.

  4. KJ says:

    Hello. Thank you for attracting my attention to this. I’ve removed the offending comment.

  5. Benoit says:

    Nice little write up KJ. one word of caveat: you are using rsync to synchronize folders. This does not constitute a backup strategy as anything deleted by mistake on the source will be deleted on the target as well.

Leave a Reply

Your email address will not be published. Required fields are marked *