I am going to do this in 2 parts. Part 1 is the proof-of-concept and generally how it works and Part 2 I’ll do when I get some more hardware in later in the week. That will be the proper, production installation. It will be step-by-step on how to implement this, properly and securely.
Quick overview – this is about using a Raspberry Pi and the open source product ownCloud to create your own version of DropBox, or offsite cloud storage service.
In recent years, there has been this new idea of “cloud storage”. This is where companies, out of the benevolent goodness of their heart, let you store many gigabytes of data on their hard drives hosted in the internet. You can access your files via the web, via mobile apps, and best-of-all, there are Windows/Mac apps that “sync” your cloud storage to a local drive. Some examples are: DropBox, Copy, OneDrive, Google Drive, iCloud, etc…
Sounds great, what’s the problem?
Well, there is a catch. If you read the terms of service, they let you store your data there, but they are going to rummage through every single byte of data you store. They are going to search, catalog, and categorize this data to give you a “better advertising experience”, by selling all of these keywords to advertisers and marketing people – whoever will give them some money. It turns out that’s the reason they were give you this for free!
Certainly, you have some files and folders you don’t care about. But they are going to open and read (legally) every file you put out there. Some people don’t like that and would like some alternatives.
By the way, just to explain some buzz-phrases you might have heard around this. If you are syncing your data to this cloud drive, is it not protected locally by default. Someone who steals your computer can likely read those files. See this article on how to easily fix this.
Communication to and from all of these servers is done over SSL. That’s means it’s as secure as opening a browser and connecting to a bank. Well, the communication is secure. However, once your data reaches the cloud service provider, it is free for others to view – including that company, their partners, the NSA, other governments, and other nefarious users. So once your data arrives at the cloud storage, that’s when the feeding frenzy begins!
You’ll hear these companies claim that your data is encrypted – yes, from some other people during transit, but it’s not encrypted from them or their data consumers!
A Solution – OwnCloud:
I ran across https://owncloud.org/ which is an open-source, PHP-based cloud storage service. The main way that it’s different is that it has a website, mobile, and desktop clients to view the data. However, YOU actually host the website – and the data is encrypted while it’s storing on YOUR hosting device! So, it’s all the good parts of cloud storage – but the actual storage of data, you manage, which makes this VERY ideal from a privacy and security standpoint.
One way you could use this is to host a site on WinHost or Azure, and run ownCloud from there. However again, I prefer to not have my data on external systems at all. What if I could host this is a secure way, where I didn’t store any data on external hardware?
What if I hosted ownCloud on a Raspberry Pi, with a USB drive for storage (like 128GB of “cloud storage” capacity)? I could expose it to the internet with some dynamic DNS, an SSL certificate, and port forwarding on my router. However, I still have the problems of: what if my house burns down or what if my house is broken into and it’s stolen?
Then it dawned on me that once this is configured, this could run from ANYONE’S internet connection. So, I am and would like to work with more people who want to do this . I’ll host your Pi + ownCloud at my house – and expose it via my router, and I’ll have you host my Pi + ownCloud at your house – and you set up port-forwarding so that I can get to it.
In other words, imagine this is a secured Pi server running Apache, PHP, ownCloud, dynamic DNS, etc – and I asked you to host this at your house. All you have to do is plug in the power and plug in the Ethernet port. It can just fall behind your desk out of sight.
I don’t know if this diagram helps or hurts:
At Person A’s house, they have Person B’s Raspberry Pi, preconfigured with OwnCloud and with a high-capacity USB thumb drive. That Pi periodically updates a dynamic DNS service with it’s public IP. Also, at Person A’s house, they have port-forwarding enabled so that any traffic coming in on port 44300, will be forwarded to port 443 on the Pi.
This means that at Person B’s house, they can find the location of the Pi by name (via dynamic DNS) – and secure traffic to port 44300 will forward inside to the Pi (via port forwarding in their router). That results is Person B at their house being able to sync their folders to the Pi and use the web interface. Meanwhile, away from home, it basically works the same way. A phone using a browser or the OwnCloud phone app would look up the Pi’s DNS address, and just connect to it the same way except over a cell network.
The benefits of this whole project are:
- Offsite storage – I have an offsite cloud storage service to which I have complete control. The “offsite” part is at a trusted friend or family member’s house, hidden behind a desk with some other wires.
- Encryption – My data is always encrypted and only I and the Pi have the private key. So, if I use strong passwords – and unless the Pi is stolen from my buddies house, the data is secure. If the Pi is lost or stolen, I have my synced copy of my data at my house.
- More storage – I get WAY more storage than those cloud providers. I am setting mine up with a 128GB thumb drive. Most free-tier cloud storage is 5 to 15GB
- Cost – All told, the Pi+Case was around $50, and 128GB thumb drive is ~$40. So, this entire setup has a one-time cost of around $90 and no recurring costs.
I did some stress-testing with this product too. Here is what I found:
- I was able to sync a single 3.9GB file – it took like a day, but it eventually synced successfully. I think that that another sync check would kick off – see the large file changing, and try to start another sync, then realize a sync was going on… etc, etc. But again, it eventually synced.
- While syncing from my PC, I did everything in the world to the Pi. I stopped the Apache server. I pulled the power plug. I tried to spin up the CPU. In times when the sync process lost communication, it went into a failure state and kept checking for the Pi to come back. When it regained communication, it started syncing again automatically.
Bottom line, I could not do anything to break the sync process. Whenever the web server was back, the sync process was quietly resume! As far as file size, I could sync at least up to a 3.9GB file – it took a day, but it did work without error. So, as a product – it seems VERY resilient!
Setting up the Raspberry Pi:
The good news is, I already covered some of this in other posts. It’s really nothing complicated at all. First, here’s the run-down:
- Get a Raspberry Pi (I have Raspian Wheezy, which is Debian v3.12.28) boot it up with a network cable plugged in.
- Update and upgrade apt-get
- Install the Apache web server, PHP, and MySQL
- Copy the OwnCloud web application to where Apache hosts the website
- Set permissions and ownership of the files
To update the repository lists, run:
sudo apt-get update
To upgrade all existing packages that are installed, run:
sudo apt-get upgrade
To install apache, PHP, and MySQL, run:
sudo apt-get install apache2 php5 libapache2-mod-php5
sudo apt-get install mysql-server mysql-client php5-mysql
the default website is hosted in /var/www/. I created a subfolder called owncloud and copied/unzipped the website to there. I need to talk to some Unix people about the next part, I changed the owner to the account that apache runs as, www-data using this:
sudo chown -R www-data /var/www/owncloud
then, so the files can execute – and I’m sure there is a more secure way (which I will cover in Part 2 of this blog post), I just made it wide-open for now, just to get it working:
sudo chmod -R 7770 /var/www/owncloud
If you are a Unix person and these two commands made your head explode – PLEASE let me know what I should do instead!
At this point you can connect to your website (use http://127.0.0.1/owncloud/ if you are connecting from the Pi, itself). Before you set up the site, you might want to add some more storage for this app. I tried a USB-powered hard drive, but there wasn’t enough power – the drive kept failing. So, I tried a few higher capacity USB thumb drives and those work just fine!
Adding the USB thumb drive:
So, before you set up OwnCloud (the landing page of the website walks you through it) – you may want to add more storage. I plugged in a USB thumb drive into one of the Raspberry Pi USB ports. From the console, I ran:
dmesg | grep “sd”
the USB flash drive will likely be /dev/sda or /dev/sdb – just look in the output for something like that. IN my case, it was “sda”. So, I need to format the device, mount it, and then tell the OS that I want to mount it every time it boots.
To view the partition table on the device, run:
sudo fdisk /dev/sda
from the cryptic prompt, type “p” and <enter> to “print” the current configuration. From there you can delete a partition, add a partition, etc. When done, use “w” and <enter> to write the changes to the device. I deleted all existing partitions and create 1 big one.
Now that the partitions exist on the USB device, we need to format it. To do that, we run:
sudo mkfs.ext4 /dev/sda
Finally, to mount it – we need a directory somewhere. I created /mnt/usb/ – so to mount that USB flash drive to /mnt/usb/ I run:
sudo mount /dev/sda /mnt/usb
You can verify that it worked by doing a:
ls –la /mnt/usb
and you should at least the lost+found. OK. Last step is we want to tell the OS that we want to mount the USB device there every time. For that, we edit the fstab file:
sudo pico /etc/fstab
In there, I added an entry at the end like this (here is definition for each column in the fstab file):
/dev/sda /mnt/usb ext3 defaults 0 3
Now, when I do a
or reboot, the USB drive is mounted there every time!
At this point, we have a wildly-unsecured Raspberry Pi with wide-open permissions running OwnCloud. You can now navigate to http://127.0.0.1/owncloud and run the wizard – point to /mtn/usb/OwnCloud/ for your “data directory”.
This was a proof-of-concept which worked great, but this is not how I want to run this.
I want to have a locked-down installation of Pi with good passwords and only necessary software installed. Once the OS is secure, I need to set up a dynamic DNS service. On the Pi, I need to run:
sudo apt-get install ddclient
and configure this to periodically update the dynamic DNS. Then, I need to get an SSL certificate with that FQDN name, figure out how to install that on Apache, then set up port-forwarding to allow traffic through for the Pi on exactly one port. Well, this last part will be done off-premise. But I should be able to bring over or send my preconfigured, secured Pi to someone else’s house. All they need to do is set up port forwarding and this should just work.
This post was showing how I did the proof-of-concept. Part 2 of the blog post will be later in the week and it will cover how to do this properly and securely. Meanwhile if you have any thoughts, opinions or ideas, please let me know!