Splitting Backups into 5GB Chunks

A common problem with cloud storage of files is that many restrict the filesize to make things more manageable. A good way to solve this problem is to use split to reduce the filesize and create multiple smaller files from a large archive.

There are two primary use cases – note that these commands split into 5GB chunks and number them starting with 00;

1) split on the fly, as the backup is being taken (this example takes a directory and creates an archive based on that directory):

 tar -cpz ${directory} | split -d -b 5000m - ${directory}.tgz.

2) split after the fact, reducing an existing archive

split -d -b 5000m source-file.gz destination-file-split.gz.

Once the archive has been split, it can be safely uploaded to your cloud storage of choice.

When the archive is retrieved and ready to be restored, concatenate the files back together and uncompress.

For a tar.gz archive:

cat *source-file*.tgz.* | tar xzv -

For a gz archive:

cat *source-files*.gz.* | gzip -d > destination