Sync Two Drives with Jenkins

Most of us know that Jenkins is an awesome continuous integration tool, however it’s also a great generic task runner. In my case, photography is one of my serious interests too, I have hundreds of gigabytes of RAW images over the years and it’s my habit to keep it in multiple hard-drives and if possible synced automatically.

In the past I have used Microsoft SyncToy to do simple folder syncing, but since then I have moved my image backup to a Linux server running 2 drives and in addition backs up the images via FTP server as well. Today I’ll share how I manage those tasks using Jenkins. I am using Ubuntu for my backup server by the way, so the instructions will be based on Ubuntu.

The idea is that if I used Jenkins, I can initiate the sync process either by schedule or with one click of a button from the web browser at home.

As always, it’s a good idea to backup your files first before attempting this tutorial. Start with smaller directories first if you can. In any case, have fun!

Install Jenkins

First things first, we have to install Jenkins first. The steps are quite simple, open your console on your terminal and enter the following commands:

The above should get Jenkins up and running automatically too. When you go to your server URL at http://localhost:8080 you should see our Jenkins interface.

Note that we are doing this for a personal in-house server, therefore I won’t go into Jenkins security; that is a topic for another day.

Allow Jenkins to Sudo

The next step is to allow Jenkins to sudo in order to mount external drives if needed.

So open your /ets/sudoers using visudo and add Jenkins information at the end the file.

Sync to 2 Local Drives

Let’s add our first task, we want Jenkins to be able to sync to 2 local drives, so in case one of the drives fails the other will still be good. My current setup is one external drive and an internal drive for backup. The main drive for me is the external since it’s easier to take anywhere should I ever need to copy files directly.

The basic of the synchronisation is done using the good ol’ rsync tool. The command that I use is as follows:

Now let’s get Jenkins to run the above command for us.

Go to your Jenkins server address, in my case it’s http://192.168.0.9:8080. That port is the default for Jenkins installation.

Once the page is loaded select:

1. New item from the left hand side
2. On the next screen, let’s call the job Image – Personal Backup
3. Then select Freestyle project, so that we can run any commands

You will then be taken to the next step which is the job build details. If we were to build  a source code, we would pull things from the repository, build and release to different server. This is the essence of Jenkins CLI, however today we are using this in a totally different manner, therefore all we need to do is to specify the build step.

So scroll down the page until you see the Build section.

1. Click on add build step, choose execute shell
2. Then insert our rsync command, but this time we prefix it with sudo. The directories are mounted under different users so we need jenkins to use the sudo command. Which is why we specify Jenkins to be able to run sudo without asking for password.
3. Your end settings should look like this screenshot below.

Schedule It

In addition to the setup that we have above, we can also tell Jenkins to run the job every so often. This is done by entering the cron style rule under the Build Triggers section.

In my case I opt to have the job to be run every 3pm on Sunday, so I will add the following rules:

You can replace the 00 with H if you wish to so that Jenkins can prevent multiple jobs from running at the exact same time if you have more than one jobs.

The end configuration result should look like this:

Sending Completion E-mails

Yet another cool thing that Jenkins can do is sending completion email after a job is done. This is done by entering your email address under the Post-build Actions. 

It’s important to understand that for my Ubuntu installation, I need to install mailutils so that Jenkins can send the emails. The command to install is as simple as:

Closing

I hope this makes your life easier when backing up your files such as you precious images between 2 drives. If you have more than 2 drives, simply add similar task as the one that we have right now. This is of course not an exhaustive tutorial of how to use Jenkins but hopefully this will get you going for the first time.

Categorised in: Tools, Ubuntu