Synchronizing remote folders

I’ve been experimenting with my latest Raspberry Pi and Sense HAT for a while and having got everything working I just wanted to copy over some files from my desktop.

Since the Raspberry Pi had SSH installed I could easily just use ‘scp’ to copy over the files in a single folder but in this case I wanted to copy files in multiple folders and replicate the directory structure. It didn’t look like this was going to be possible using ‘scp’ so I had to resort to searching to see if I could find an answer. After a while I found a couple of links, one of which showed me how to use ‘rsync’ to replicate a folder structure (and all the python files), and the other showed me how to use ‘rsync’ with ‘scp’ to copy files to a remote machine, so all I had to do was combine these two methods to give me what I wanted.

When using ‘rsync’ be careful to make sure you get any trailing forward slashes in the right place, otherwise the results may not be quite what you expect. For example to copy everything from ‘~/documents/letters/’ to ‘~/archive/letters/’ you could use either of the following commands:

$ rsync -a ~/documents/letters ~/archive/
$ rsync -a ~/documents/letters/ ~/archive/letters/

Both commands do the same thing but since ‘rsync’ will create the destination folder if it doesn’t exist the difference between them is that in the first case ‘~/archive/’ folder will be created if necessary and in the second case it must already exist.

$ rsync -a ~/documents/letters ~/archive/letters/

The command above won’t work as it will actually copy everything in the folder ‘~/archive/letters/’ to ‘~/archive/letters/letters/’.

Tip – To see what files and folders would be copied without actually running the command use ‘-n’ to perform a dry run and ‘-v’ to show what files or folders would be copied. In the case of the command above this would have shown that is was not going to work, since the first thing that gets copied is the ‘letters/’ folder itself, and not the first file in that folder…

$ rsync -a -n -v ~/documents/letters ~/archive/letters/
sending incremental file list
created directory ~/archive/letters

To selectively copy files and folders using ‘rsync’ you can use filters to include and exclude certain files. The following example uses a filter to include all folders and combines this with a second filter to exclude all files in order to replicate the folder structure from the source folder to the destination folder:

$ rsync -a -f"+ */" -f"- *" source/ destination/

Note – You can’t exclude all the files before including the folders as an '*' by itself will match every file and folder, but by including every folder first using '*/' we get the result we want.

Adding additional filters allows you include other files so since I wanted to copy all the python and shell scripts as well I needed to add two additional filters before the final filter that tells ‘rsync’ not to copy any files.

$ rsync -a -f"+ */" -f"+ *.py" -f"+ *.sh" -f"- *" \
> source/ destination/

This will only copy all the python and shell scripts but it includes all the folders, so I added an additional filter exclude hidden folders.

Note – Since '*/' will match all folders even f they are hidden you need to put the filter to exclude hidden folders before the one that includes all folders.

$ rsync -a -f"- .*/" -a -f"+ */" -f"+ *.py" -f"+ *.sh" -f"- *" \
> source/ destination/

You can also include or exclude files using ‘--include’ and ‘--exclude’.

$ rsync -a --include="+ *.py" --include="+ *.sh" -f"- .*/" -f"+ *" \
> source/ destination/

To get ‘rsync’ to copy files to a remote system using ‘ssh’ you need to specify the remote shell.

If you wanted to copy everything from ‘~/documents/letters/’ to ‘~/archive/letters/’ in the another user’s home folder on a remote machine with an IP address of you would need the following command:

$ rsync -a -v ~/documents/letters -e ssh username@

So finally to copy all the folders and sub-folders from one user’s home folder to another user’s home folder on a remote machine including any Python or shell scripts I needed to combine the remote shell with the appropriate filters:

$ rsync -a -v -f"- */*/*/" -f"- .*/" -f"+ */" -f"+ *.py" -f"+ *.sh" -f"- *" \
> -e ssh ~/ username@

I have added an additional filter to exclude any folder that matches '*/*/*/'. This will exclude any folder more than two levels deep which will limit the depth of the recursive copy.

Note – By default ‘rsync’ doesn’t copy files whose modification date and size match, to force it copy these files use ‘-I’.


Raspberry Pi is a trademark of the Raspberry Pi Foundation

This entry was posted in Debian, Linux, Raspbian and tagged , , , . Bookmark the permalink.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s