Running your own email server (2015 megapost edition)

Of all of the professional development things I’ve done in the last few years, this has been the most time-consuming, and educational, and tedious, and frustrating… and tedious.. So, in this post I’ll cover all the things I’ve learned – primarily for Future Robert, because he’ll forget!

Why own your own email server:
I almost don’t know where to start. E-mail comes with a LOT of baggage and history. If you forget for a moment what you know about e-mail, and let me describe a new messaging system I’m writing – a system where people can send messages back and forth over the internet. Here are the key points:

  • It will be insanely difficult to setup and endlessly tedious.
  • The actual payload of the messages will all be in clear-text, and it will be really difficult to do end-to-end encryption.
  • This means that sysadmins can pretty easily read all messages of all users (and/or sell them to marketeers, governments, etc)
  • This also means those messages are susceptible to hackers getting them too – and they will be able to read them all, including all attachments.
  • This messaging system will be used to send, generally, private communication meant specifically for a limited group of parties. It may also contain highly sensitive things like links to reset a password, actual credentials, bank account numbers, etc.

This sounds, nothing short of horrible. Yet, this is basically what e-mail is (plus many more bad things!). So, why then would you want to own your own email server? This sounds like a trainwreck; e-mail in general is a pretty awful technology. Welp, here are a few reasons:

  • YOU would be the “owner” of the the content. If a marketing person or government wanted YOUR data, they would come to YOU! No one would else would incidentally have access.
  • You could more easily implement end-to-end encryption, and have safer (more secure) backups
  • Your data wouldn’t be going through Google, Microsoft, and Apple – where their terms of service basically say “if we can see your data, we will capture your data”. (foreshadowing: “Or would it have to go through those anyway?”)

Or summed up another way, why wouldn’t you? For convenience? Sure, Google/Microsoft/Apple all give you free e-mail, but that’s because YOU are the product. They capture every byte of your data. They sort, organize, collate, and then sell this data as part of your advertising profile – and/or let governments have it – and/or occasionally they are hacked and hackers have it. Your data is not even approximately secure, and it’s really nothing more than a chewtoy for every: company, government, and hacker out there. Wouldn’t you rather own your own data?

Who cares! It’s just e-mail!
But it’s not “just e-mail”. E-mail is really the key to everything in your electronic life. Need to reset your password at your bank or credit card website? They send you an e-mail. Use two-factor authentication? Most sites still use an e-mail (amongst other methods). Sending something in writing, of importance, to anyone else? Chances are you’ll use e-mail.

E-mail is the circulatory system of our private digital lives – yet it’s not even a little bit secure.

Not only is it inherently not-secure, we use it through companies that ADMIT they are going to observe what we send/receive and can keep a copy of this data! So, if e-mail is at the center of our electronic lives, doesn’t it deserve at least a little though about how secure it is, and how we back up this data?

Why own your own e-mail server, as an IT professional?
This is probably more of what sent me over the edge on this adventure. For several projects I’m working on, I have several domains. I’d like to have a contact e-mail for a few of them. However, the cheapest/easiest option is for me to pay $5/user/month. There are a zillion places (including gmail and outlook.com) that offer this sort of deal. I would like e-mail, but not that bad – I just want an easy/simple way for people to communicate with me, for things regarding that domain. Put another way, if I want an e-mail for 4 domains, I’m looking at $20/month just for “just-in-case” e-mail addresses? That’s crazy. It’s the year 2015, I have some infrastructure at home and in Azure, there must be an on-prem way I can do this easily.

What I’m describing below is all FREE, except for your time. You can host unlimited mailboxes for unlimited domains on one “server” which could be, in this case, a Raspberry Pi?!

Whether you want to own your own data or if you just want to save some money, this might be worth pursuing.

What are my options for e-mail servers?
There might be other options, but in the Windows and Linux world, there exists a few options:

  1. Use Gmail/Outlook.com/etc and pay $5/month/user
  2. Use Microsoft Exchange server (like, if you have an MSDN subscription – it’s cost-prohibitive otherwise)
  3. Buy “all-in-one” mail server software – these are not cheap, and the price skyrockets quickly for even basic features.
  4. Host your own mail server (like, on Linux – like below).

Each one of these is fraught with problems. For example:

  • Gmail/Outlook.com (cloud) – just a waste of money for “nice-to-have” e-mail accounts. Several domains and few accounts can add up quickly.
  • Exchange server (on-prem) – the server MUST be part of an Active Directory, and is already insanely complicated. More on this, below.
  • Pay mail server (on-prem)http://www.iredmail.org/, the “free, open source email server” which isn’t free, or https://www.axigen.com/ which gets really expensive, really quickly.
  • Host your own (on-prem or cloud) – this is not easy, not straight-forward, but is free and completely configurable.

In case you haven’t guessed, I went with this last option – and this blog post will explain exactly how to set it up.

Getting started…
To start, let me say that e-mail comes with a LOT of baggage. It was invented in the early 1970’s, and has been added-on-to every since. Now, nothing is simple, it’s a quilt of relatively simple parts that all work together. They seem pretty fragile – and some pieces are – but they somehow manage to work. Not knowing anything about mail on Unix/Linux, I wasn’t stuck on a single problem for more than maybe :20 minutes.

Make no mistake though, even the simplest setup will be remarkably complexticated!

Now, I have/will do this setup three times. The first time, I spent probably about 50 hours now total researching, experimenting, and figuring this all out. I did this on my dev, Ubuntu 15.04 Server.

image

Next, I’m doing this again from scratch on a Raspberry Pi running Raspbian. Why? Primarily because people are always saying “I don’t know what to do with my Raspberry Pi!” – and here is a completely reasonably option. You can host your own mail server and have many gigabytes of mail storage, and it would be completely portable too, in addition to being safer for your data.

image

The third time I will do this will be on my production Ubuntu 15.04 Server, and that will be (I hope) where I run my e-mail going forward. Both Ubuntu and Raspbian are Debian-based distributions of Linux, so the steps, and locations of config files should be exactly the same (in this case, they are exactly the same). This just goes to show you how amazing a Raspberry Pi is – you can do the same thing you can do on a big Linux server!

Consider this also: your production server could be a Raspberry Pi 2 at your house with a 128GB USB thumb drive (giving you 128GB of mailbox storage) – or you could rent a standalone ARM-based server for $3.33/month, and pull nightly backups from it.

image

This might be ideal because if your mail server is compromised, ONLY that server is compromised. If you host at home and your mail server is compromised, then so is your internal network!

Concept: How does it all work?
This in itself took some time to gather. I knew pieces of this, but I wasn’t really solid on the whole thing. So, now having been down this road, here is how e-mail works nowadays, as I now understand it:

image

In summary, you need:

  • DNS – your DNS server must have an “MX” (mail exchanger) which points to your SMTP server.
  • SMTP server – this receives inbound mail for your domain, and you use it to send mail to other SMTP servers.
  • IMAP server – a way that you can use an e-mail client to view your messages, mark them read, move them between folders – all of that is handled by IMAP.
  • Web mail – optionally, you may want to offer a webmail client where users can log in and see their mailbox.

The DNS part is just a configuration change done once, and that’s it. As for the other 3 parts, these can all exist on one server – your e-mail server.

Step 1: Sending an e-mail
When you are in your favorite e-mail client and you click “Send”, you submit your message to your SMTP server. Then, that SMTP server does a request to the DNS of the recipients and asks for the “MX” record(s) for the domain. MX stands for “mail exchanger”. This is the name of an SMTP server which accepts e-mail messages for that domain. You can do this from a Linux computer with:

$ nslookup –type=MX gmail.com

On Windows, the nslookup help shows that you can do this too, but for the life of me, I couldn’t get it working. Anyhow, that results in something like this:

Server:         192.168.1.213
Address:        192.168.1.213#53

Non-authoritative answer:
gmail.com       mail exchanger = 5 gmail-smtp-in.l.google.com.
gmail.com       mail exchanger = 10 alt1.gmail-smtp-in.l.google.com.
gmail.com       mail exchanger = 40 alt4.gmail-smtp-in.l.google.com.
gmail.com       mail exchanger = 20 alt2.gmail-smtp-in.l.google.com.
gmail.com       mail exchanger = 30 alt3.gmail-smtp-in.l.google.com.

Authoritative answers can be found from:
gmail-smtp-in.l.google.com      internet address = 74.125.21.27

This tells me that if I want to send e-mail to a gmail.com e-mail address, that I should connect to any of those Mail eXchangers (MX). Note the number after them is the preference, so prefer the 5 more than the 40. This tells clients WHICH mail exchangers to prefer, or try first.

So, when you click send, you send the message to your SMTP server, who then does this MX lookup – then your SMTP server connects to the other SMTP server, and delivers the message.

By the way, if you don’t have DNS tools like “nslookup” and “dig” on your Linux machine, you’ll need to install a package. On Debian-based distributions like Raspbian, it’s:

$ sudo apt-get install dnsutils

On other distributions, this package might be called “bind-utils” too.

If you connect to SMTP on port 25, the whole conversation is done in clear-text. If you connect to port 465, then the communication is encrypted via TLS (similar to how https works), but the payload is still in clear-text. Here is an example conveyance of an e-mail message – “>” are messages you see from the server, “<” are things that you type in (“helo” is not a typo by the way):

> 220 some_mailserver.net ESMTP Sendmail 8.13.8/8.13.8; Thu, 23 Apr 2015 12:34:07 +0100
< helo localhost.localdomain
> 250 some_mailserver..net Hello ???? [???.???.???.???] (may be forged), pleased to meet you
< MAIL From:myaddress@example.com
> 250 2.1.0 myaddress@example.com… Sender ok
< RCPT To:somone@example.com
> 250 recipient <somone@example.com> OK
< DATA
> 354 enter mail, end with line containing only “.”
< Hello Fred, can you call me?
< .

> 250 message sent
< QUIT
> 221 goodbye

That’s more-or-less how a client gets an e-mail handed off to your mail server.

Step 2: Storing the e-mail
Remember, e-mail was invented 30+ years ago, even before relational databases. Plus, it was mostly available on Unix where “everything is a file” – so e-mails are stored as files. There are at least two general formats: MBOX, and MAILDIR. One uses directories for folders, and e-mail messages are one per file. The other uses files to represent folders, and inside of that one file are multiple messages. Maildir (the directories and one-message-per-file) approach seems to be the preferred method. I even bring this up because your SMTP server is just part of the equation. If you are going to use an IMAP server to read your mail – that needs to speak the same language! That will be covered below, but first, here’s how the MAILDIR format looks (using the Linux “tree” command to view the directory structure):

image

new messages are stored under “new”, and then the e-mail client moves them to “cur” after they’ve been read. An IMAP client can also create folders and move messages that way too. So, each one of those “144156…” files under the “cur” and “new” folders are individual e-mail messages.

Notice too that the structure of the directories is: /var/spool/mail/domain.com/user – the approach below will show you how to support unlimited domain and unlimited mailboxes without, without adding any new Linux user accounts. All mailboxes are “virtual” and have their own authentication.

The permissions are significant too because there are potentially 3 different actors who need access to these folders: postfix (the SMTP server who delivers the mail), dovecot (the IMAP server that gives you access), and squirrelmail (or other webmail client, which needs to give you access to your mail). We address that with a group that gives all 3 of these access.

Step 3: Accessing messages from an e-mail client (Outlook, smartphone, Thunderbird, etc)
In order to connect to your mailbox, see messages, create folders, and move messages – you need an IMAP server. One of the most popular choices is “dovecot”. Despite it being a very different technology than postfix, it works with postfix quite well. Postfix just accepts e-mails and puts them into domain/user folders – but doesn’t know about any usernames/passwords.

Dovecot then (or any IMAP server), is what needs to support “virtual users” and provide a way for them to authenticate AND map them to the correct domain/user folder. Luckily, this is just some configuration.

All you need do is create the new mailbox on the server (I have a script which does this, see  below), and then the end-user can connect and see their mailbox.

Step 4: Access messages from webmail (Squirrelmail)
If you are going to offer a mail service, it’s inhumane to only offer IMAP. In the same way you have www.outlook.com or www.gmail.com in ADDITION to the ability to get messages on an e-mail client, that’s what we’re talking about. A web interface for checking your messages.

It would be ideal in addition to offering IMAP, to also offer a webmail interface too. That is, a website where the end-user can log in with their IMAP mailbox credentials and see their messages. In present day, everyone talks about Squirrelmail as being the de facto standard but um… I don’t get it. Despite that it was easy to setup, the interface is straight out of the early 2000’s! Not only is it not mobile-friendly, it still uses “frames”!!!! See below, yikes…:

image

I mean yes, it works, and it’s technically a free webmail client, but geez-Louise it’s pretty rough! There are many themes; you can change the fonts and colors, but that’s about it. For example:

image

I’m sure there are other clients out there, but yeah – this one does everything you need (except being able to use it on a phone). So for now, we’ll go with this.

The Plan/The Project:
What I am setting up here, and what you can easily do, is set up mail on a Raspberry PI, or on any Linux system you have laying around. This will be mostly specific to Debian, but honestly the only things that would likely be different across Fedora/CentOS, openSUSE, etc will be file locations.

There are a zillion ways you could set this up – here, we’ll stick to exactly one way. It will include:

  1. Setting up your DNS – including getting a dynamic DNS name, and pointing your mail.example.com to it, and establishing an MX record.
  2. Setting up port forwarding – mail network traffic will be coming to your home router. You should open up a port or three and have it direct to your mail server.
  3. Installing and configuring Postfix – the SMTP server (receives e-mail and stores it on disk; sends mail on your behalf)
  4. Installing and configuring Dovecot – the IMAP server (lets you connect to your mailbox via Outlook and other mail clients)
  5. Installing and configuring Squirrelmail – the webmail application (lets you view your mailbox from a browser)

By the way, implementing this is much easier than you might think. The concepts above are the difficult part, once you know the “why” behind each step, it makes it relatively easy to get each piece done. With that said, let’s dig in.

STEP 1: Setting up your DNS:
Since DNS settings take some time to propagate, it probably makes sense to get these going first. This is similar to pre-heating the oven in your kitchen while you prepare the rest of your meal. There are three things that you’d want to do:

  • (Optional) Get a Dynamic DNS name
  • Create a CNAME record which points to your mail server (or dynamic DNS)
  • Create an MX record which points to your mail CNAME

All about Dynamic DNS:
If you are going to host this at home, your might know your internet-facing IP address changes periodically. To account for this, you can register for a dynamic DNS entry via one of several free or paid services. Then, either with software on your computer, or in your router, that will go update the dynamic DNS whenever your public IP changes.

Wait, what? Why? If I want my publicly-facing IP address to be available to others by name (like: example.dnsalias.org), then I need to update that DNS entry every time my router gets a new IP address. I happen to be using www.dyn.com and they make it pretty simple (referral code: RFE1TR3KRZ to save 15%):

image

I’m using this pay-service for other reasons, but if you are looking for something free, check out www.changeip.com too. One of the reasons I used dyn.com is because my router supports it directly:
image

Above, what this means is that whenever my internet router gets a new IP address, it will log into DynDNS and upload the pointer for my dynamic DNS name.

Creating a CNAME record:
This is really just for convenience. Ideally, you want to be able to have everything connect to mail.example.com if example.com was your domain. You could point to that for your webmail, and when you are connecting Outlook, you could just point to mail.example.com for the IMAP and SMTP server too.

Now, this is technically not correct (using a CNAME record). Technically, “A” records should point from a name, to an IP address. A CNAME record should be an alias for an existing A record that you already own. However, if you are using dynamic DNS, there is no other way.

So, if you are using something like GoDaddy for DNS, you want to create a new CNAME record like this, where “yourhomeserver.example.com” is your dynamic DNS name, or if you hosting this as a proper server, just the A record of your already-defined mail server.

image

and if you are using Active Directory, that would look like this:

image

and if you are using Bind, in your forward zone file, add:

mail IN CNAME yourhomeserver.example.com

To test this (on Windows, MacOS, or Linux), you should be able to:

ping mail.example.com

if example.com was your domain. You should get a response back. If you don’t and it says it can’t find the host, you can clear your name resolution cache on Windows with:

ipconfig /flushdns
nbtstat –R

and then try again. It can take some time for DNS changes to propagate, so you can also just try again later.

Create an MX record:
Now we need to create the Mail Exchanger record. Again, on godaddy.com it will look something like this:

image

and if you are using Active Directory, then that would look like this:

image

and if you are using Bind, in your forward zone file, add:

IN MX 10 mail.hernando.division42.com

if your domain was “hernando.division42.com”. Now how do you verify an MX record? See above where we could either do:

$ nslookup –type=MX division42.com

or with the “dig” command:

$ dig division42.com MX

and again, nslookup on Windows supposedly supports this but I couldn’t get it to work!

STEP 2: Setting up port forwarding:
On most home routers, this concept is called “port forwarding”, however on this Arris, they call it “virtual servers”:

image

the concept is that when network traffic comes in from the outside on port 25 (unsecure SMTP), then forward that traffic to the internal address of 192.168.1.202 on port 25. This should be the IP address of your mail server.

image

As far as which ports to open, it depends on how secure you want to be. It would be ideal to only support secure transactions. However, for testing, it might be easier to use the non-secure techniques. So, here are all the ports that will be involved (see: SMTP and IMAP for more details):

Port Service
25 SMTP, unsecured (clear-text)
587 SMTP, secured for submissions
465 SMTP, secured, deprecated older standard but still widely used
143 IMAP, unsecured (clear-text)
993 IMAP, secure
80 HTTP, unsecured (for accessing webmail)
443 HTTPS, secured (for accessing webmail)

When configuring which ports are forwarded from the internet, you should really only forward the secure ports.

STEP 3: Installing and configuring Postfix (the SMTP server):
Geez, this has been a lot already! All of this was just prep work to actually get to point where we install a mail server. Believe it or not, that will be relatively simple. I’ve already wasted the dozens of hours figuring all of this out so that you don’t have to. This should hopefully be pretty turn-key. To get started, run:

$ sudo apt-get install postfix

that will kick off the installer, which looks like this:

image

Choose Internet Site and take all the defaults. Once installed, there are few things we need to do:

Tell postfix about the domains and mailboxes we want to support:
By default, Postfix is just meant to support mail for the local computer. If you want a new mailbox, you create a new Linux account. That isn’t really ideal for multiple domains and multiple users. So, Postfix (and Dovecot, later) support the idea of “virtual” mailboxes. These do NOT need a local Linux account. So first, edit the main configuration file:

$ sudo nano /etc/postfix/main.cf

Add the following:

masquerade_domains = division42.com robseder.com

virtual_mailbox_domains = division42.com, robseder.com
virtual_mailbox_base = /var/spool/mail
virtual_mailbox_maps = hash:/etc/postfix/virtual

assuming your domains are division42.com and robseder.com. Notice that “virtual_mailbox_maps” setting – that points to another file. That file, is where you establish your mailboxes for each user. So, go edit that file:

$ sudo nano /etc/postfix/virtual

and add something like this:

rseder@division42.com division42.com/rseder/
rob@robseder.com robseder.com/rob/

that means that it is going to route e-mail to rseder@division42.com to /var/spool/mail/division42.com/rseder/ – consider too that the /var/spool/mail/ is already the default maildrop location. The above  settings just split that out more. As you can tell, this means that you could support multiple domains and multiple users. See a script at the end of this post to automate adding these entries.

Postfix isn’t any smarter than this. It’s simply going to see if the e-mail address domain is one that we support; if it is, then it splits the username and domain name, and puts the message in that folder.

Enable connecting to SMTP over TLS (the secure way):
Actually this is enabled by default (see the smtpd_tls* settings in the main.cf file). However, if you did want to regenerate your x.509 certificate, you can run something like this:

$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/postfix/ssl_private.key -out /etc/postfix/ssl_public.crt

and then to create a .pem file, you combine the two files:

$ sudo cat ssl_private.key ssl_public.crt > ./ssl.pem

and then change those settings in the main.cf to point to the new certificate. However, this isn’t a great way to go because there are other steps you need to take to make this a “trusted” certificate (Postfix requires that the certificate be trusted).

Nowadays, SSL certificates are cheap or even free. If you are going to do this for real, you should probably get a certificate for like $9/year and be done with it!

Set up relaying:
Remember how we talked about using a mail server so that Microsoft, Google, Apple, etc don’t collect your data? Well it turns out there is yet another scam where they can and do anyway. Remember the terms of service from those vendors basically say “if we can touch your data, we can keep it”. Now, when I tried to e-mail a gmail.com or outlook.com e-mail address, I got a non-deliverable with this message:

host mx3.hotmail.com[65.55.92.184] said: 550 DY-001 (SNT004-MC4F4) Unfortunately, messages from xxx.xxx.xxx.xxx weren’t sent. Please contact your Internet service provider. You can tell them that Hotmail does not relay dynamically-assigned IP ranges. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. (in reply to MAIL FROM command)

What does this mean? It means that you can’t send an e-mail to major e-mail providers from your home mail server because they don’t allow that. You MUST send all of your e-mail (unsecured) through “acceptable” e-mail servers, like Microsoft, Google, etc.

Put another way, you need to “relay” your mail through another, more-trusted SMTP server: the owners of e-mail, or Big Email™.

One option I have, is I can relay through my internet service provider. If you can’t, you can relay through gmail or outlook.com, etc. The whole idea is supposedly to reduce spam. No one can submit a message on the internet without being vetting by the approved e-mail gatekeepers! Such a scam.

Anyhow, to set up relaying, go and edit the main file again:

$ sudo nano /etc/postfix/main.cf

and then add something like this:

# To enable relaying through brighthouse
relayhost = mail.brighthouse.com
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/saslpasswd
smtp_always_send_ehlo = yes

You might notice that this points to an external file for the username/password. So, go create/edit that file:

$ sudo nano /etc/postfix/saslpasswd

and add a line like this:

mail.brighthouse.com user@brighthouse.com:mypassword

Then, and this is important, run this command to make a “database” file for SASL:

$ sudo postmap /etc/postfix/saslpasswd

Now, restart the SMTP service:

$ sudo service postfix restart

and you should now be able to send e-mails, even to outlook.com or gmail.com addresses. Lastly, you probably want to apply permissions to that saslpasswd and .db file:

$ sudo chmod o-r /etc/postfix/saslpasswd
$ sudo chmod o-r /etc/postfix/saslpasswd.db

The “o-r” means remove the read-only permissions for world/everyone-else. So now, owner has read-write, group has read, and world has nothing – which should keep those credentials safe:

image

Lastly, we want to set the permissions that Postfix uses. Since there are 2 actors who need access to these messages (Postfix for delivering, and Dovecot for accessing), we can use the “mail” group to limit access. So, we can tell Postfix to set the permissions on the mailboxes as postfix (user account as the owner), and mail (group). To do that, in the main configuration file, add this:

# write messages as postfix:mail (uid:gid)
virtual_uid_maps = static:107
virtual_gid_maps = static:8

That is assuming that is the UID and GID of those accounts. I found that the group number was the same, but the UID may be different. To find them, run this:

$ cat /etc/passwd | grep postfix
$ cat /etc/group | grep mail

and that will show you the UID of “postfix” and the GID or “mail”.

Set the “mailbox format” to “MAILDIR”:
To be honest, I’m confused at this point about what the default is for postfix, so to take it out of the equation, you can explicitly set this it to use maildir instead of mbox format, add this to that main configuration file:

home_mailbox = Maildir/

That about does it. At this point, you should be able to receive e-mail messages and they should get stored in /var/spool/mail/domain/user where domain and user are your domain and your email name. Clients should be able to send e-mail out to any domain, even gmail.com and outlook.com too – however, we can’t test this until we can connect to a mailbox. To do that, we need an IMAP server.

STEP 4: Installing and configuring Dovecot (the IMAP server):
At this point, you should have a working SMTP server that receives mail and puts a message to jdoe@example.com into the /var/spool/mail/example.com/jdoe/ directory (well, technically under the “new” folder, under that). That’s great, but we don’t have any way to access these mailboxes – nor a way to authenticate users. That is where Dovecot comes in. To install that, run:

$ sudo apt-get install dovecot-common dovecot-imapd

If you are doing this on a Raspberry Pi, you will get some errors messages. If you do, modify this file:

$ sudo nano /etc/dovecot/dovecot.conf

and change the:

list = *,::

to just:

list = *

This means that it will only bind to IPv4 adapters. Now, re-run in your apt-get install and it should complete successfully. Next, in the /etc/dovecot/conf.d/ folder, there are several config files. There are just few things that are needed – you’ll need to modify: 10-mail.conf, 10-ssl.conf, and 10-master.conf.

$ sudo nano /etc/dovecot/conf.d/10-mail.conf

add/modify this file to include the following:

mail_location = maildir:/var/spool/mail/%d/%n

passdb pam {
  driver = passwd-file
  args = /etc/dovecot/passwd
}
userdb static {
  driver = static
  args = uid=postfix gid=mail home=/var/spool/mail/%d/%n
}

The mail_location is very important. First, it says that the “format” of the mailboxes is “maildir” (as opposed to mbox, from earlier) – which means that SMTP and IMAP will be expecting the same directory and file structure. Next, notice on that same line, that defines the path of the mailbox as:

/var/spool/mail/%d/%n

where %d is the domain portion of the e-mail address (e.g. example.com) and %n is the username portion (e.g. jdoe). Next, the passdb explains that we are going to have a password file for usernames and passwords. That location will be /etc/dovecot/passwd (more on this in a minute). Lastly, the userdb setting tells it which permissions to maintain on the files and folders (postfix:mail).

Also within that file set the following:

first_valid_uid = xxx
last_valid_uid = yyy
first_valid_gid = z
last_valid_gid = z

This sets permissions of who can access the mailboxes via IMAP. Typically, you’ll find that the “postfix” and “dovecot” users are created around the same time. On this system, postfix is UID 111 and dovecot is UID 112 (run “cat /etc/passwd” on your system to find the user ID). Likewise, on this system, the group ID for “mail” is 8 (run “cat /etc/group” on your system to find the group id). Therefore, xxx = 111, yyy = 112, and z = 8. Which means my config actually has:

first_valid_uid = 111
last_valid_uid = 112
first_valid_gid = 8
last_valid_gid = 8

But again, your system will be different, so you need to look up your own values! Next, within the master config:

$ sudo nano /etc/dovecot/conf.d/10-master.conf

just uncomment the following lines to enable secure connections for IMAP:

inet_listener imaps {
  port = 993
  ssl = yes
}

Lastly, modify the SSL config:

$ sudo nano /etc/dovecot/conf.d/10-ssl.conf

In this file, uncomment and set this to “required”:

ssl = required

this will make sure you can ONLY connect to IMAP securely (over port 993). You will likely need to add (uncomment and modify) the following lines as well:

ssl_protocols = !SSLv2 !SSLv3 !TLSv1
ssl_cipher_list = ALL:HIGH:!SSLv2:!MEDIUM:!LOW:!EXP:!RC4:!MD5:!aNULL:@STRENGTH

Also, just make note of the path of the ssl_cert and ssl_key. You should get a real certificate, or for testing, you can generate a new one like this:

$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/dovecot/private/dovecot.pem -out /etc/dovecot/dovecot.pem

At this point, you should be able to restart the service and get no errors:

$ sudo service dovecot restart

You should also keep an eye on syslog for errors. In another window, to constantly monitor syslog, run:

$ sudo tail /var/log/syslog –f

That will tie up that console, but will update automatically as new entries are written. use CTRL+C to cancel.

STEP 4.5: Creating user accounts and passwords:
At this point, the SMTP server accepts e-mail for a list of domains, but doesn’t know much more than that. The IMAP server knows how to find those folders, but it looks for username/password information in the /etc/dovecot/passwd file (which you need to create). So, first you need to encrypt the users password, like this:

$ doveadm pw -s SHA512-CRYPT –p MyGreatPassword

Which results in something like this:

{SHA512-CRYPT}$6$j4razhr3I0EdvWsX$xkcXIXYQNNIOOXEAw5ichj2EImXmW8Z3XrzmxbhOez/cW5o5q7lF4nAulsmkbZX2MrFXg1Ordd0kExFMpvt5O1

Now, the format of that /etc/dovecot/passwd file is something like this:

jdoe@example.com:{SHA512-CRYPT}$6$j4razhr3I0EdvWsX$xkc…cW5oXg1Ordd0kExFMpvt5O1
ssmith@example.com:{SHA512-CRYPT}$6$w3qz0s.vRkwNnPT/…uQyBjRsm2yGiFjL9Z3pVYM8u0

I truncated the password for readability, but you get the idea, it’s e-mail address, then “:”, then the complete result from encrypting the password. To put all of this together: when your IMAP client connects, your e-mail address and password hash will be matched in this file. If it’s a match, then your username and domain name will be used to go to your mailbox directory – where you will have access to your messages.

For each new user, you’d need to run the above command to encrypt that password, and then append a new line in that password file.

STEP 5: Installing and configuring Squirrelmail (the Webmail client):
At this point, we are getting mail, and we can connect to our mailbox via an IMAP client (like Outlook or Thunderbird). Now, as a nice-to-have, it would be ideal to also have a webmail interface. We’ll use “Squirrelmail”. To install this:

$ sudo apt-get install squirrelmail

this is a pretty big install as it installs the Apache2 web server and many, many supporting libraries. Once installed, there are just a few things you need to do.

Set up HTTPS:
For this example, I’ll create a self-signed certificate. In real life, you should buy a real SSL certificate. So for testing, here is how you can create a self-signed cert:

$ sudo mkdir /etc/apache/ssl
$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/apache2/ssl/apache.key –out /etc/apache2/ssl/apache.crt

then, to protect the private key, take away “read” permissions from “world”:

$ sudo chmod o-r /etc/apache2/ssl/apache.key

Next, enable the SSL module in Apache:

$ sudo a2enmod ssl

Then enable the default SSL site:

$ sudo a2ensite default-ssl

One last thing is you need for apache to point to our our new x.509 certificate we created above. Edit the following file:

$ sudo nano /etc/apache2/sites-enabled/default-ssl.conf

and adjust this section:

SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

to instead point to our new location:

SSLCertificateFile      /etc/apache2/ssl/apache.crt
SSLCertificateKeyFile   /etc/apache2/ssl/apache.key

Exit and save the file. You need to restart the web server daemon for that to take effect:

$ sudo service apache2 restart

At this point you should be able to navigate to https://yoursystemname and get a warning about the certificate not being proper, but it should load. The last piece we need is to set up the squirrelmail virtual directory.

Setup /squirrelmail/ virtual directory:
When you install squirrelmail from apt-get, it puts it into /usr/share/squirrelmail. So, you need to create a mapping to map https://yoursystemname/squirrelmail to be served from /usr/share/squirrelmail. In fact, there are are several specific things that it wants. So, I found this page – which shows you need to add this content:

Alias /squirrelmail /usr/share/squirrelmail/www
<Directory /usr/share/squirrelmail/www>
  Options None
  AllowOverride None
  DirectoryIndex index.php
  Order Allow,Deny
  Allow from all
</Directory>
<Directory /usr/share/squirrelmail/www/*>
  Deny from all
</Directory>
<Directory /usr/share/squirrelmail/www/scripts>
  Allow from all
</Directory>
<Directory /usr/share/squirrelmail/www/images>
  Allow from all
</Directory>
<Directory /usr/share/squirrelmail/www/plugins>
  Allow from all
</Directory>
<Directory /usr/share/squirrelmail/www/src>
  Allow from all
</Directory>
<Directory /usr/share/squirrelmail/www/templates>
  Allow from all
</Directory>
<Directory /usr/share/squirrelmail/www/themes>
  Allow from all
</Directory>
<Directory /usr/share/squirrelmail/www/contrib>
  Order Deny,Allow
  Deny from All
  Allow from 127
  Allow from 10
  Allow from 192
</Directory>
<Directory /usr/share/squirrelmail/www/doc>
  Order Deny,Allow
  Deny from All
  Allow from 127
  Allow from 10
  Allow from 192
</Directory>

to the main configuiration file, /etc/apache2/apache.conf

$ sudo nano /etc/apache2/apache.conf

Now, restart the service one more time:

$ sudo service apache2 restart

At this point, you should be able to navigate to https://yoursystemname/squirrelmail/ and you should see a login prompt:

image

If you have a mailbox defined, go ahead and log in. If you still haven’t pieced how to do that – see the next “Now what?” section.

Putting it all together:
At this point, we have a web service running, we enabled SSL and have it pointing to a self-signed certificate. We then set up an alias for /squirrelmail/ which points to the web files which were installed when we installed that package.

If you have a user defined with a password set, you should be able to log in and see your mailbox.

Now what?
Well first, the concept of adding a new user is kind of complicated, so I made a little script called ~/addmailuser.sh:

#!/bin/bash

user=$1
domain=$2
email=$1@$2
password=$3

logger “Adding new IMAP user: $email”
echo -n “[*] Generating SHA512 password hash for $email… “
result=$(echo -n “$email:”; sudo doveadm pw -s SHA512-CRYPT -p $password)
echo ‘Done.’

echo -n ‘[*] Writing new user/password to Dovecot user database… ‘
echo $result >> /etc/dovecot/passwd
echo ‘Done.’

echo -n “[*] Setting up postfix virtual user $email… “
virtualentry=”$email    $domain/$user/”
echo $virtualentry >> /etc/postfix/virtual
echo ‘Done.’

echo -n ‘[*] Updating postfix to pick up the new mailboxes… ‘
postmap /etc/postfix/virtual
echo ‘Done.’

echo -n “[*] Creating mailbox directory (/var/spool/mail/$domain)… “
mkdir /var/spool/mail/$domain
echo -n ‘ Setting permissions…’
chmod 770 -R /var/spool/mail/$domain
echo -n ‘ Setting owner… ‘
chown mail:mail -R /var/spool/mail/$domain
echo ‘Done.’
logger “IMAP user: $email added.”

Or, just to make it readable, here it is with color-coding:

image

This does the steps necessary to create a new SMTP/IMAP user, and sets permissions. You call it like this:

$ sudo ./addmailuser.sh jdoe example.com MyAwesomePassword

And that’s all you need to do to add a new mailbox!

Now, there is a list of other things you would definitely want to do. For example, but not limited to:

  1. Verify the folder and file permissions for your mailboxes is correct. Make sure that postfix or root are the owners, and the “mail” is the group. Then, verify who is in the “mail” group.
  2. Verify the file permissions on the critical files (like private keys, the passwd files, etc)
  3. Set up a firewall and only allow certain ports. See this post, and the “iptables” section.
  4. Make sure you have a VERY good root and super-user password. See this post for an idea on how to have a very strong password which is easy to remember.
  5. Research more about postfix. I was reading that there are additional steps you should take to make sure spammers don’t use your SMTP server to send out their spam!

But aside from that, this is basically a working e-mail system for multiple domains and multiple accounts – all set up for free.

Bottom Line:
I’m not even sure where to start. When I say that e-mail is a trainwreck, do you see where I’m coming from? It’s not even that any of these products are bad, it’s just that this system has decades of baggage which every mail component needs to account for. This ended up being an enormous undertaking. I’ve been working on this blog post for about a week now, because as I would try to re-implement the things in my notes on a second system (both to better document it, and to test to make sure that’s ALL I needed) – it turns out I did miss things. So, it took some time to get all those bugs worked-out.

So there it is – if you want to host your own mail server, believe it or not, I would still recommend this over Microsoft Exchange. Anyone who’s ever managed MS Exchange will likely tell you that too. For as bad as this is, Exchange is even more cryptic – but worse is there is far less documentation on it. Conversely, what is “great” about this system described above, is there are mountains of documentation, blog posts, forums posts, etc – so finding an answer typically never took more than :20 minutes.

With all of that said, it does kind of come down to your motivations:

  • Saving money: yes, this is all completely free. Given that most setups are $5/user/month, e-mail can start turning into a pocket drain.
  • Security: this is a mixed-bag, for me. On the one hand, you’ll be able to control your own mail server. But, are you really that excellent at locking down a server and your firewall? Also, if your mail server is on-prem and it’s compromised, that will leave your mail server as a gateway to the rest of the servers in your organization. So, it might be best to host off-prem (maybe on a Scaleway server?) That way, if mail is ever compromised, the impact is localized to just that server.
  • Privacy: this is a joke. Mail was never meant to be secure, and still isn’t. Messages are sent and stored in clear-text. Worse, is that Big Mail™ owns the internet. The only way you can send e-mail to major providers is if you pass your mail THROUGH one of the big providers. Given that their terms of service say that can keep any data you pass through them, all of your e-mail is collected and correlated by Them anyhow. Not until people use e-mail certificates, or hosted services like www.protonmail.com (which has end-to-end encryption) will e-mail start to become secure.

So what is the bottom line, will I host my own mail server? In short, yes – so that I have a contact e-mail for the domains I host. Would I use it as my real, primary e-mail? I don’t know, probably not. I’m still on the fence. What about you? Knowing now what it takes to set up a mail server, will you host your own? Leave a comment below!

Posted in Best-practices, Cloud Computing, Computers and Internet, General, Infrastructure, Linux, Open Source, Organization will set you free, Professional Development, Raspberry Pi, Security, Uncategorized
4 comments on “Running your own email server (2015 megapost edition)
  1. […] I’ve been building out my production mail server, I had the need to monitor some files. Sure, I could have multiple windows/tabs open and click back […]

    Like

  2. James says:

    Nice post, though for those folks which doesn’t find it easy OR prefer existing private and secure solutions – I’ll recommend https://www.mailfence.com (in contrast with what author has mentioned) due to their ‘true’ end-to-end encryption and digital signing service, along with a complete emailing-suite (docs, calendar, contacts, groups, polls, …).
    However, it all depends on one preferences and specific requirements – I use mailfence, due their support of IMAP with other existing features.
    Otherwise, GnuPG is also there (the ultimate tool) – but only if you can handle its usability (which almost is close to a nightmare).

    Like

  3. dannijensen says:

    Thx for a super nice guide. I’m going to use this guide for my own 5 private domains.

    Sir would you please consider making a guide on how we can do end-to-end encryption so we can make email secure?? please base it on a free setup like u have shown here.

    These days security seems to be a necessity…if one cares about privacy at least 😉

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Archives
Categories

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 2 other followers

%d bloggers like this: