cloudsync

This project is obsolete and inactive. I recommend to use rclone as a much more powerful alternative. If you are interested to takeover the ownership. Feel free to contact me.

Sync a local filesystem on linux, windows and osx incremental and encrypted with google drive simliar to rsync. You can also restore the encrypted data back to a local filesystem. It works as a complete backup solution for your private data.

Other compareable backup solutions like duplicity upload one big encrypted 'base' archive with additional delta archive files. After a few months you must upload a new fresh 'base' archive to avoid hundreds of delta files. This approach, while widely used in various backup solutions, is problematic for private async DSL connections. To solve these issues each file is encrypted and uploaded separately.

To get a first impression you should take a look at this screenshot.

The encryption is based on OpenPGP with AES 256 (optional) and a passphrase. It is possible to decrypt uploaded files with a normal OpenPGP compatible tool like 'gpg' or 'gpg2'.

Encrypted data includes:

• filetype [folder,file,symlink]
• filecontent, filename and original filesize
• createtime, modifytime and accesstime
• owner, group, posix permissions, acl entries and fat32 attributes
• md5 checksum

Filechanges are detected by comparing the file metadata. It uses a local cachefile to speedup the incremental update. The local cachefile is completly restoreable by analysing the serverside archived metadata.

Supported Services are:

• Google Drive (stable)
• Remote mounted filesystem (beta)
• Dropbox (alpha preview)

To provide additional cloud targets like Amazon Cloud Drive or Microsoft OneDrive just implement 6 functions from the interface Connector.java.

Requirements

1. Java >= 7
2. Java Cryptography Extension (JCE) (optional to support AES256)
3. Maven

Install

git clone https://github.com/HolgerHees/cloudsync.git
cd cloudsync
mvn install

Config

To use, copy 'config/cloudsync.config.default' to 'config/cloudsync.config' and set the following;

• PASSPHRASE: your master password used for encrypt/decrypt

There are two authentication options, either using an Installed Application or Service Account.

Both options require that you enable the Google Drive API.

Installed Application

Follow these instructions to set the following;

• GOOGLE_DRIVE_CLIENT_ID
• GOOGLE_DRIVE_CLIENT_SECRET

Service Account

This process requires more configuration however will ensure you can run unattended.

1. Create Service Account. You must ensure that you delegate domain-wide authority to your service account.
2. Download P12 keyfile.
3. Set the following in the configuration file.

• GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL - ie.. <some-id>@developer.gserviceaccount.com.
• GOOGLE_DRIVE_SERVICE_ACCOUNT_USER - The Google Drive user, ie.. [email protected].
• GOOGLE_DRIVE_SERVICE_ACCOUNT_PRIVATE_KEY_P12_PATH - The full or relative path to your P12 file.

Usage

To create a backup of '/data', call:

./cloudsync --backup /data --name dataBackup

to restore a backup into '/restore', call:

./cloudsync --restore /restore --name dataBackup

for a complete list of options, see below:

usage: cloudsync <options>
 -b,--backup <path>                            Create or refresh backup of <path>
 -r,--restore <path>                           Restore a backup into <path>
 -c,--clean <path>                             Repair 'cloudsync*.cache' file and put leftover file into <path>
 -l,--list                                     List the contents of an backup
 -n,--name <name>                              Backup name of --backup, --restore, --clean or --list
    --config <path>                            Config file path. Default is './config/cloudsync.config'
    --followlinks <extern|all|none>            How to handle symbolic links
                                               <extern> - follow symbolic links if the target is outside from the
                                               current directory hierarchy - (default)
                                               <all> - follow all symbolic links
                                               <none> - don't follow any symbolic links
    --existing <stop|update|skip|rename>       Behavior on files that exists localy during --restore
                                               <stop> - stop immediately - (default)
                                               <update> - replace file
                                               <skip> - skip file
                                               <rename> - extend the name with an autoincrement number
    --history <count>                          Before remove or update a file or folder move it to a history folder.
                                               Use a maximum of <count> history folders
    --include <pattern>                        Include content of --backup, --restore and --list if the path matches the
                                               regex based ^<pattern>$. Multiple patterns can be separated with an '|'
                                               character.
    --exclude <pattern>                        Exclude content of --backup, --restore and --list if the path matches the
                                               regex based ^<pattern>$. Multiple patterns can be separated with an '|'
                                               character.
    --permissions <set|ignore|try>             Behavior how to handle acl permissions during --restore
                                               <set> - set all permissions and ownerships - (default)
                                               <ignore> - ignores all permissions and ownerships
                                               <try> - ignores invalid and not assignable permissions and ownerships
    --nocache                                  Don't use 'cloudsync*.cache' file for --backup or --list (much slower)
    --forcestart                               Ignore a existing pid file. Should only be used after a previous crashed
                                               job.
    --noencryption                             Don't encrypt uploaded data
    --dry-run                                  Perform a trial run of --backup or --restore with no changes made.
    --progress                                 Show progress during transfer and encryption.
    --retries <number>                         Number of network operation retries before an error is thrown (default:
                                               6).
    --waitretry <seconds>                      Number of seconds between 2 retries (default: 10).
    --network-error <exception|ask|continue>   How to continue on network problems
                                               <exception> - Throw an exception - (default)
                                               <ask> - Show a command prompt (Y/n) to continue
                                               <continue> - Show a warning and continue
    --file-error <exception|message>           How to continue on blocked files or permission problems
                                               <exception> - Throw an exception - (default)
                                               <message> - Show a error log message
    --logfile <path>                           Log message to <path>
    --cachefile <path>                         Cache data to <path>
    --min_tmp_file_size <size>                 Minimum file size <size> in bytes to use tmp files (default: 134217728)
 -v,--version                                  Show version number
 -h,--help                                     Show this help