Cloud services

Setting up my own cloud service

  • Sharebar

Setting up my own cloud service has become my current hobby

I own a Motorola Razr and together with motocast is simply awesome because it syncs all of my files directly to my pc. I can stream music from my computer, pictures, videos and modify files, you know all that beauty.

I love motocast so much that I even have a computer with Windows installed on it (granted there are other reasons for it but that is almost a sin for a Linux guy like me)

Recently though I have been searching for a solution that will allow me to sync any phone with my computer regardless of the operating system. I tried dropbox, owncloud, Ubuntu one, box and skydrive. None of them were able to do quite what I needed and how I needed it, Own Cloud is freaking awesome but can’t stand the idea of wasting a whole web server and database server just for it.

Being a developer I think I can fix the world if I am given the source code so I started some little scripts to accomplish this a simpler and cleaner way.

This of course it is not a full blown solution, matter of fact is in the pre pre pre alpha stages (trust me on that)

Why set up your own cloud service

Setting up my own cloud service gives me the ability to have as much disk space as I need or want. I am at 20GB at DropBox for free (friends are awesome :-))

My other reason is that I don’t like dropbox or any of the self called cloud service keeping my files or being the medium to sync them. Except for my domain. I mean think about it, would you want to store your private documents in a service that can be compromised?

You got me, what do we need to do to get rolling

Ideally you should do this with your own domain, one that supports ssh but they are other alternatives like using DropBox, Ubuntu One, Box or even email.

You can use a free domain with FTP but it will be up to you to figure out how to do it because I am a security freak and don’t like FTP.

What you need:

  • A computer that will be your server
  • An static IP (we are going to bypass it – seriously)
  • Access to your router, we’ll need to open some ports
  • Unison (file sync – apt-get update && apt-get install unison)

Why Unison and not Rsync?

Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other. …(read more here)

It sounds like black magic to me, the kind I need to get my files synced without worrying about it too much.

Let’s do magic… let’s get rid of the static IP requirement

In order to set up my own cloud service I decided to use a domain instead… however here is I how I did it first.

Whatsmyip.com offers this a URL that returns only the external IP we are visiting it from:

http://automation.whatismyip.com/n09230945.asp

So if we are if we set up a cron job say to get our IP every 4 hours

*/4 * * * curl http://automation.whatismyip.com/n09230945.asp > my_external_ip.txt

Then we are good to go and don’t need the static ip. Now keep in my mind that if that url goes dead, well we need to start considering the alternatives.

So let’s start with the server:

#!/bin/bash
#script in the server
IP='my_ip.txt';
OLDIP='old_ip.txt';
PORT='22';

if [ -e $IP ]
    then
	    mv $IP $OLDIP;
fi
#download your current ip
curl http://automation.whatismyip.com/n09230945.asp > $IP
scp -P$PORT $IP user@domain:/path

#alternatives
#cp $IP ~/Dropbox/
#*/4 * * * curl http://automation.whatismyip.com/n09230945.asp | mail -s "my ip" email_id

#upload your old ip needed to update the services later
if [ -e $OLDIP ]
    then
	scp -P$PORT $IP user@domain:/path
fi

Here is a link to the script: http://ubuntuone.com/3TlzBkhH4Cyr6KMBiBvXS9

Let’s move on to the client

#!/bin/bash
#script in the server
IP='my_ip.txt';
OLDIP='old_ip.txt';
PORT='2222';

if [ -e $IP ]
    then
	    mv $IP $OLDIP;
fi
#download your server ip

scp -P$PORT user@domain:/path/$IP $IP 

#replace any instances of your old ip
#may require sudo depending where you use it
if [ -e $OLDIP ]
    then
        IP=$(cat $IP);
        OLDIP=$(cat $OLDIP);
        sed -i 's/$OLDIP/$IP/g' ~/.ssh/config
fi

Here is a link to the client: http://ubuntuone.com/1O5BifJOdAaOXfvmv0sEEN

So now, let’s setup the actual programs

In all the clients we just need to the following:

vim ~/.ssh/config

Add this:

Host MyOwnDropBox
HostName SERVER_IP
Port SERVER_PORT
User USERNAME

Set up your own private/public ssh keys

ssh-keygen -t rsa 

You can either copy the contents of the ~/.ssh/config/id_rsa.pub to the server ~/.ssh/authorized_keys or you can do this:

ssh-copy-id username@server

Now this here comes the fun part, let’s sync the clients and server:

#!/bin/bash
# set PATHS / dirs
_PATHS="/home/username \
/etc/"
 
# binary file name
_unison=$(which unison)
 
# server names 
# sync my client with rest of the server in cluster
_REMOTE_SERVER=$(cat my_ip.txt)
 
# sync it
for r in ${_REMOTE_SERVER}
do
	for p in ${_PATHS}
	do
        	${_unison} -batch "${p}"  "ssh://${r}/${p}"
	done
done

Script link here: http://ubuntuone.com/0Ri4ncQRrUO3JfG463JS1r

Simple right… if you have your static IP or domain you can cut the setup to half of it.

So… what did we get

Unison is great because it is available for almost everyone (Windows, Unix, Debian, RedHat) and it is awesome because it is resilient to failure. It is careful to leave the replicas and its own private structures in a sensible state at all times, even in case of abnormal termination or communication failures.

For the Android phones it’s been a bit complicated to implement because I have resorted to using http://kevinboone.net/kbox.html and there is not a unison client but there is support for rsync, problem is that I still can’t handle bi-directional syncing correctly. But hey a least I have my computers in sync which is progress icon smile Setting up my own cloud service

Besides the obvious features of adding a good interface to avoid command line (seriously why do I keep using bash?), a more automated setup(gosh it kills me), selective syncing and a more consistent way of syncing maybe a domain just for that and different accounts, what do you guys think should be on the wishlist.

Keep in mind this is 3x pre alpha icon smile Setting up my own cloud service and there are better solutions out there but I like to keep my privacy and most important ensure that my files are not contained somewhere in the cloud.

So what’s next

Well I’d like to get from pre alpha to maybe alpha software and move from bash to vala… don’t get me wrong, I love bash but sometimes you need a higher level program to get things done quicker. We won’t sacrifice performance but I am still scratching my head on how I am going to do multithreading in bash something it is trivial in Vala. Maybe even add a GUI to it but more importantly I want it to turn this into a push notification system and make an android version of it so it syncs automatically as well, you know nothing fancy.

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)
 Setting up my own cloud service

About Luis Tineo

Husband, Father, performance improvement junkie, biker and video gamer, Linux user and in my day job I'm a Systems Architect at Blue Acorn.

Tags: , , ,

  • http://www.mageaustralia.com.au/ Magento Australia

    Or you could use http://owncloud.org/

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • http://www.kingletas.com letas

      Yes I have before but it pains me to need a webserver just to sync files and folders, it seems like a waste of resources to me… Owncloud it’s pretty awesome just too much for what I need and want.

      VN:F [1.9.22_1171]
      Rating: 0.0/5 (0 votes cast)
      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)