s3kup: the s3 backup tool
I’ve created a tool creating backups on a s3 bucket, I called it s3kup. It’s written in GO and very straightforward to use. It’s somehow a rewrite of an old ruby gem I had for the same purpose.
It’s a command that can take any input from a shell pipeline and store it in a versioned way in a s3 bucket. s3kup has 3 basic commands: push, pull and list.
push
It will store your backup, e.g.:
pg_dump | bzip2 -c | s3kup push --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name database-bkp.bz2 --keep 5
tar cz /app-storage/* | s3kup push --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name app-storage.tar.bz2 --keep 5
cat my-cat-picture.png | s3kup push --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name my-cat-picture.png --keep 5
It should work with anything you send as input through a pipeline.
s3kup will create a structure on the bucket like this:
my-backups/
|- database-bkp.bz2/
|- <timestamp1>
|- <timestamp2>
| ...
|- <timestampN>
The --keep flag will enforce how many versions you want to keep for the file.
s3kup will automatically remove older versions when the keep limit is reached.
list
List stored versions for a backup:
s3kup list --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name database-bkp.bz2
* 1429439081775798266 119.5K Sun Apr 19 07:00:10 2015
* 1429439081773590206 38.7K Sat Apr 18 07:00:08 2015
* 1429439081772235435 38.6K Fri Apr 17 07:00:07 2015
* 1429439081771233242 35.2K Thu Apr 17 07:00:08 2015
* 1429439081763423420 33.2K Wed Apr 16 07:00:08 2015
pull
Pull will output the stored version to the STDOUT. By default it will fetch the last pushed version:
s3kup pull --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name database-bkp.bz2 > database-bkp.bz2
s3kup pull --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name database-bkp.bz2 | bunzip2 > database.sql
But it’s possible to specify a version too:
s3kup pull 1429439081761232345 --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name database-bkp.bz2 | bunzip2 > database.sql
Custom endpoints
By default s3kup will push to the us-east amazon s3 endpoint (https://s3.amazonaws.com), but it’s possible to specify the endpoint for another region or a s3 like service:
s3kup push --endpoint-url "https://s3-eu-west-1.amazonaws.com" ...
Download / Install
Using go to build/download:
go get github.com/tscolari/s3kup
go install github.com/tscolari/s3kup