s3kup: the s3 backup tool

Posted by Tiago Scolari on Apr 18, 2015
In Development, Go, Server

I’ve created a tool creating backups on a s3 bucket, I called it s3kup. It’s written in GO and very straightforward to use. It’s somehow a rewrite of an old ruby gem I had for the same purpose.

It’s a command that can take any input from a shell pipeline and store it in a versioned way in a s3 bucket. s3kup has 3 basic commands: push, pull and list.

push

It will store your backup, e.g.:

pg_dump | bzip2 -c | s3kup push --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name database-bkp.bz2 --keep 5

tar cz /app-storage/* | s3kup push --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name app-storage.tar.bz2 --keep 5

cat my-cat-picture.png | s3kup push --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name my-cat-picture.png --keep 5

It should work with anything you send as input through a pipeline.

s3kup will create a structure on the bucket like this:

my-backups/
|- database-bkp.bz2/
   |- <timestamp1>
   |- <timestamp2>
   | ...
   |- <timestampN>

The --keep flag will enforce how many versions you want to keep for the file. s3kup will automatically remove older versions when the keep limit is reached.

list

List stored versions for a backup:

s3kup list --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name database-bkp.bz2
* 1429439081775798266       119.5K      Sun Apr 19 07:00:10 2015
* 1429439081773590206        38.7K      Sat Apr 18 07:00:08 2015
* 1429439081772235435        38.6K      Fri Apr 17 07:00:07 2015
* 1429439081771233242        35.2K      Thu Apr 17 07:00:08 2015
* 1429439081763423420        33.2K      Wed Apr 16 07:00:08 2015

pull

Pull will output the stored version to the STDOUT. By default it will fetch the last pushed version:

s3kup pull --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name database-bkp.bz2 > database-bkp.bz2
s3kup pull --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name database-bkp.bz2 | bunzip2 > database.sql

But it’s possible to specify a version too:

s3kup pull 1429439081761232345 --access-key aws_key --secret-key aws_secret --bucket-name my-backups --file-name database-bkp.bz2 | bunzip2 > database.sql

Custom endpoints

By default s3kup will push to the us-east amazon s3 endpoint (https://s3.amazonaws.com), but it’s possible to specify the endpoint for another region or a s3 like service:

 s3kup push --endpoint-url "https://s3-eu-west-1.amazonaws.com" ...

Download / Install

Using go to build/download:

go get github.com/tscolari/s3kup
go install github.com/tscolari/s3kup

Or download the binary.

Github Repository