sync dir s3

Sync whole directories to Amazon S3

Installation

$ npm install -g sync-dir-s3

Motivation

Whilst there are already a lot of S3 command line clients out there all I really wanted was something like rsync that I can use for backing up files from my laptop and servers to S3.

This command line tool provides some options but decides on a lot of things as well. The big assumption is that for the most part it will be interacted with and is not 100% automated. It doesn't rely on credentials being available globally (i.e. ~/.aws/ or similar) as it prompts you for these, and stores these in an encrypted format in your home directory.

Another important aspect is that it just decides how to name your files in an S3 bucket. For speed and ease you just specify the bucket and it will name everything like this:

{bucketName}/{hostName}/{directoryTree}/{filename}

So for example, if you were to run it in the following directory (assuming a bucket name of my-backups)...

$ hostname
lenovo-ideapad
$ cd /home/user/MyFiles/
$ ls
file1.txt   file2.txt   file3.png
file4.xls
$ sync-dir-s3 # interactions skipped in this example

...you'd get the following in the my-backups bucket:

lenovo-ideapad/home/user/MyFiles/file1.txt
lenovo-ideapad/home/user/MyFiles/file2.txt
lenovo-ideapad/home/user/MyFiles/file3.png
lenovo-ideapad/home/user/MyFiles/file4.xls

This way (as long as your different machines have different hostnames) you don't have to worry about name structures and everything is kept predictable and consistent.