Getting started with Splunk (MacOS, Linux, and Windows megapost)

I was talking with a colleague the other day about Splunk. This is a very powerful tool for monitoring and searching log files across the organization, and the enterprise version supports alerting too. That is, if certain things are found in any log file, you can receive a notification via e-mail, SMS, etc. I’ve heard of it before, but understood that it was pretty expensive so I never dug into it. Well, he told me there is a free version – so I decided to take a look and see what it’s all about! …that was several days ago.

image

http://www.splunk.com/

So, this splunk>light product is free, but has a limit of 500MB of data per day that it will process. I have no idea if that is a lot or a little. However, in my humble “data center” at my house, I have plenty of systems (Windows servers and workstations, Linux, Mac OS, Raspberry Pi’s, etc). I’ll try to add as many as I can and see how well it handles it (and how quickly I eat through that daily limit)!!

Here is a good page which describes the limitations and differences between splunk>light free, splunk>light, and splunk>enterprise.

What is Splunk?
Consider first that I’m brand-new to it, but Splunk is basically a tool that lets you aggregate and monitor logs files from pretty much any source. That means Windows event logs, “syslogs” from Linux, IIS logs, etc. You can view them all, from multiple systems, via one very powerful web interface. What I saw today in a demo is that it has a pretty intuitive and powerful filtering “language” in the web interface so that you can pull back exactly what you are looking for.

Before we dig into that, we need to install it first. Splunk>light runs on Windows, Linux, and Mac OS.

image

In this blog post, I’ll cover the installation on MacOS, Linux, and on Windows. Once installed, using the web app is pretty much the same on all of them.

Installing Splunk on MacOS X Yosemite:
The new version of MacOS (El Capitan) is due out tomorrow, so hopefully this information won’t become obsolete too soon. To start, you will need to create an account (ugh, really? In the year 2015?). Yes, create an account with a “valid e-mail address”. Then, I downloaded the .dmg image – this is basically like a DVD .iso image.

image

open up the Downloads folder in Finder and double-click the .dmg image to mount it:

image

then, like most MacOS installers – double-click the “Install Splunk” icon to kick off the installer:

image

Like on each of these platforms, after the install, there is this post-installation initialization that needs to happen:

image

and then we are left at this prompt:

image

if I choose “Start and Show Splunk”, that finishes the initialization at a command prompt, and then opens a browser to http://localhost:8000 where the default username/password is admin/changeme:

image

Installing Splunk on Ubuntu Linux:
I spun up a new Ubuntu Server, patched it, and then went to download the .deb file. Since this is an Ubuntu Server, there is no GUI. So then, I went to use secure copy (sc) to upload it from my workstations to the server, but I got an error. Ugh. Then, on the same Splunk download page, what do I see?

image

Cool – wget is a command-line download tool! I can get a link to just download it from the command-line. So, while SSH’ed into this new Ubuntu VM I set up for splunk, I ran this wget command to download it:

$ wget -O splunklight-6.3.0-aa7d4b1ccb80-linux-2.6-amd64.deb http://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=6.3.0&product=splunk_light&filename=splunklight-6.3.0-aa7d4b1ccb80-linux-2.6-amd64.deb&wget=true’

Now, I have this .deb file in my root directory. To “install” this Debian package, I now run:

$ sudo dpkg -i splunklight-6.3.0-aa7d4b1ccb80-linux-2.6-amd64.deb

and that installed with no error, but now what? Fast-forward ahead :15 minutes after scouring the splunk website, and I found this page: http://docs.splunk.com/Documentation/SplunkLight/latest/GettingStarted/StartSplunkLightandlogintoSplunkWeb which didn’t work, but sent me down a path. I finally found from this post on Stackoverflow that splunk gets installed into:

/opt/splunk/bin/

so, now knowing the directory where it’s installed, if go into that directory and run:

$ sudo ./splunk start

that shows me a terms of use and asks me to agree, then finishes setting up the service. It gave me output like this:

Starting splunk server daemon (splunkd)… 
Generating a 1024 bit RSA private key
……………………….++++++
…………………….++++++
writing new private key to ‘privKeySecure.pem’
—–
Signature ok
subject=/CN=sphusplp01/O=SplunkUser
Getting CA Private Key
writing RSA key
Done

Waiting for web server at http://127.0.0.1:8000 to be available…. Done

If you get stuck, we’re here to help. 
Look for answers here:
http://docs.splunk.com

The Splunk web interface is at http://sphusplp01:8000

Now that it’s initialized, if I navigate to that URL in a browser, voila! I see the splunk interface. According to that same page above, the default username/password is admin/changeme.

image

One other thing I found is that Splunk is not set up as a service. So, upon every boot-up, you’d have to go manually run:

$ sudo splunk start

Instead, to install it as a service, run:

$ sudo splunk enable boot-start

Then, you can treat like a normal service:

$ sudo systemctl status splunkd.service
$ sudo systemctl stop splunkd.service
$ sudo systemctl start splunkd.service

Plus, whenever this computer is restarted, it will start the service automatically.

Installing Splunk on Windows:
For Windows, I spun up a new virtual machine using Windows Server 2012 R2. Installing Splunk is pretty much as you might expect. I went to the download page:

http://www.splunk.com/en_us/download/splunk-light.html#

and downloaded the Windows, 64-bit installer. I run the .msi and follow the prompts:

image

On one of the steps, this is an interesting choice:

image

This is probably one of the biggest advantages of Windows Server in general, is the idea of a server “joining a domain”. Because, if you have a domain account, that domain account can be used on ALL computers in that domain. So, if you have an Active Directory domain and want to monitor log data from those servers, this might be a little easier.

I say easier because if you set this up with a domain account, then you can “pull” data from all of the other domain servers. Otherwise, and what you do with the MacOS and Linux installations is wait for other systems to “push” their data via this Splunk Universal Forwarder, which we’ll talk about below.

Then at the end of the installation, you can launch a web browser and connect to the app:

image

and you are shown the login page where the default username/password is admin/changeme:

image

At this point, you should have your main splunk server set up on one of these operating systems. Now, we have to have it point to data sources so that we search/browse that data.

Actually using Splunk:
At this point, I’ll assume you have Splunk installed on one of these 3 platforms. In my case, I actually plan to use Splunk, so I’ll use my Ubuntu server: sphusplp01. Therefore, I’ll start by navigating to:

http://sphusplp01:8000/

and logging in (the default user/password is admin/changeme). When you first login, there are tutorials for each screen – I found those helpful! So, there are basically 3 ways to get splunk to analyze data:

  1. Use local files on the server where splunk is running
  2. Monitor input (WMI, TCP/UDP ports, etc) for incoming data
  3. Receive data that is sent to this server via a “universal forwarder”

Using a static, local file:
This option seems to be for one-time analysis, as you upload a file to process. So, it doesn’t seem that this is for continuous, or live monitoring of a real system. From the main dashboard/landing page, in the top-right:

image

click the “Add Data” button. This is where you see those 3 options:

image

we’ll choose the first one, “upload”:

image

for this, I exported my local Event Viewer contents from a Windows computer and saved it as a CSV file. On the next screens, you can choose from a bunch of known formats:

image

this now is a “source” from which you can search. This means that I can now search from the main screen with something like:

error and source=”test.csv”

which will show me every entry which has the word “error” from any of the entries from that test.csv file I uploaded.

Monitoring local files:
Perhaps an easy thing we can do is add a monitor for the local syslog on this server which is hosting splunk. So, from the main page I would click on “Add Data” in the top right and this time choose the middle option, “monitor”:

image

for this we have a few options:

image

in this case, we can click “Files & Directories” and just point to the location of the syslog, which is /var/log/syslog:

image

Now, back at the main screen this is a new source from which we can search. In this case we can specify the host and/or source with something like in the search textbox on the main page:

error and source=”/var/log/syslog”

which will just show entries in syslog where the word error is found.

Receive data from “universal forwarders”:
The main way you collect data from your other machines is to install this universal forward agent on the source machines.

image

As mentioned above, if you are mostly working with Windows servers in a domain, that works a little more seamlessly. However, in a heterogeneous (and non-Windows) environment, you’ll need to install this agent.

This seems to be slightly different for different operating systems, so I’ll cover each one separately. But first, on our Splunk server, we need to enable this server to receive messages. So, click on the “hamburger menu” in the top left and under “Data”, choose “Receiving”:

image

then click the “new” button and give this a port, like 9997:

image

which results in a listener, listening:

image

This sets up a port to receive data that “universal forwarders” will send. Now, adding forwarders is different for each OS, so I’ll break those out into separate sections, below.

Adding a Windows-based “universal forwarder”:
Luckily, the Windows forwarder is pretty easy because the installer does all of the work! First, download the Windows-based forwarder. Then, on the Windows machine where you want the log files monitored, run that forwarder installer:

image

On this next screen, you could set up your own certs to use – or just left splunk create it’s own self-signed certificates:

image

similar to if you set up Splunk itself on Windows, if you run the forwarder as an Active Directory domain account, that will allow you to do more with other machines in the same domain. In my case though, I’m just going to use a local account:

image

This next screen is where all the magic happens. This computer is supposed to forward logs and events – but WHICH logs and events? On this screen, you just put a checkbox next to what you want to forward to the main Splunk server:

image

image

After going through the documentation, I still don’t get what a deployment server is – so I leave this blank, as it is optional:

image

this is the other key screen – the “receiving indexer” is the “Receiving” port we set up on the main Splunk server. So, point to the name and port number of your main Splunk server. Note that this port is NOT the port that you use in the web browser, this is the “Receiver” port defined up above – by default it is 9997:

image

image

At this point, this Windows machine is sending logs and information to Splunk:

image

Note that I ran this on my other test Windows server “sphwsplp01”. So now, from the main web interface, I can browse and search log information from this external server.

Adding a MacOS X-based “universal forwarder”:
The non-Windows forwarders are little trickier. In the case of the MacOS one – I did NOT get this working. However, here is what I did do before I got jammed-up! First, download the MacOS X-based forwarder.

Once downloaded, open up “Downloads” in Finder and double-click the .dmg file to mount the image:

image

Then, launch the installer and follow the wizard:

image

When done, it sort of leaves you in the lurch:

image

What do you do now? Well, it says you can navigate here: http://docs.splunk.com/Documentation/Splunk but that page doesn’t help much. So, after some research I basically did:

$ cd /Applications/SplunkForwarder/bin
$ sudo ./splunk start

however that gives me a LOT of errors starting with “dyld: Library not loaded”. So, the next think I supposedly do is point to my splunk server – that is, the destination who will receive my logs:

$ sudo ./splunk add forward-server sphusplp01:9997

But that too results in a lot of errors:

dyld: Library not loaded: /Users/eserv/wrangler-2.0/build-home/ember/lib/libmongoc-1.0.0.dylib
  Referenced from: /Applications/SplunkForwarder/bin/splunkd
  Reason: image not found
dyld: Library not loaded: /Users/eserv/wrangler-2.0/build-home/ember/lib/libmongoc-1.0.0.dylib
  Referenced from: /Applications/SplunkForwarder/bin/splunkd
  Reason: image not found
dyld: Library not loaded: /Users/eserv/wrangler-2.0/build-home/ember/lib/libmongoc-1.0.0.dylib
  Referenced from: /Applications/SplunkForwarder/bin/splunkd
  Reason: image not found
Did not find “disabled” setting of “kvstore” stanza in server bundle.
dyld: Library not loaded: /Users/eserv/wrangler-2.0/build-home/ember/lib/libmongoc-1.0.0.dylib
  Referenced from: /Applications/SplunkForwarder/bin/splunkd
  Reason: image not found
Couldn’t complete HTTP request: No error

I spent a fair amount of time researching and didn’t find anything that helped. It’s worth noting that as of this writing, the newest version of OS X came out (El Capitan) yesterday, and I’ve since installed it. So, this could very easily be a problem where the splunk forwarder doesn’t support the very latest OS just yet?

With MacOS X El Capitan – I’m dead in the water.

Adding a Ubuntu Linux-based “universal forwarder”:
For Linux, this too is not as straight-forward as Windows, but I got further than I did than with MacOS!

With that said, I also did NOT get this working either, but below is what I did do.

First, download the Linux-based forwarder. In my case for Ubuntu Linux, I downloaded the 64-bit .deb file. For this case, I clicked on the “Got wget?” section and used this command:

$ cd ~
$ wget -O splunkforwarder-6.3.0-aa7d4b1ccb80-linux-2.6-amd64.deb http://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=6.3.0&product=universalforwarder&filename=splunkforwarder-6.3.0-aa7d4b1ccb80-linux-2.6-amd64.deb&wget=true’

that results in a *.deb file in your home directory. To now install that package, run:

$ sudo dpkg -i ./splunkforwarder-6.3.0-aa7d4b1ccb80-linux-2.6-amd64.deb

Then, navigate to the directory where this gets installed and set up the forwarder:

$ cd /opt/splunkforwarder/bin/
$ sudo ./splunk add forward-server sphusplp01:9997

Then, you’ll also need to run splunk start to accept the license and to start the forwarder:

$ sudo ./splunk start

You will be prompted to accept the license and do some initial setup.  When I was stuck here, I was able to talk to a subject matter expert at work about this. As I understand it, at this point, this server knows to forward everything to my main splunk server, but we haven’t defined anything to monitor! For the Windows-installer above, we just checked some checkboxes – but how do you do the equivalent for Linux?

Go into this directory:

$ cd /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default

then edit inputs.conf:

$ sudo nano ./inputs.conf

and to monitor syslog, I added this to the end of the file:

[monitor:///var/log/syslog]
_TCP_ROUTING = *
index = _internal

Then, restart splunk:

$ sudo systemctl restart splunkd.service

I also tried adding the same to /opt/splunkforwarder/etc/system/local, then restarted splunk and still nothing. I still don’t see this system listed in the web interface. I scoured the documentation, the web, stackoverflow, etc – and everyone says it should be as easy as this. Yet, I simply couldn’t get it working.

With Ubuntu Linux Server – I’m dead in the water.

Bottom line:
Ugg. For crying out loud, I wasn’t expecting to turn into this megapost. However, there ends up being QUITE a bit to this product. As crazy as this sounds, this doesn’t even begin to cover the power and functionality of what Splunk can do – this is just getting is set up.  And with that said, I wasn’t very successful. In the end, all I could get working was:

  • Uploading and reporting on a static/fixed file.
  • Monitoring local files on the splunk server
  • Monitoring stuff from Windows machines

I ultimately couldn’t get the MacOS nor Ubuntu forwarders to work. This seems to be one of the obvious differences between open source and enterprise-type products. Open source tends to have VERY good documentation. In the case of splunk, they are selling the product and selling support. It doesn’t make much business sense to have all of the answers published publicly now does it!!

But yes, I must admit, I am brand new to splunk, so I likely missed something. However, in my defense, I am a new user and I found the documentation to be quite terrible. What started out as a whimsical ideal has turned into a multi-day event. I probably have about 15 hours into playing with this so far – and this blog is what I wrote down as I was exploring.

If you are familiar with splunk and have some ideas why I failed, above, please leave a comment below. Otherwise, I’ll continue to dabble. If I do get MacOS or Linux working, I’ll either update this post or write a follow-up.

Meanwhile, if you were curious about what it takes to set up splunk>light, hopefully this blog posts helps!

Posted in Computers and Internet, General, Infrastructure, Linux, Organization will set you free, Professional Development, Uncategorized
4 comments on “Getting started with Splunk (MacOS, Linux, and Windows megapost)
  1. Greg Beddow says:

    Rob, Thanks for the article. I’m also new to Splunk, ran across the same problem under OS X El Capitan, and came upon this:

    https://answers.splunk.com/answers/307112/install-failure-in-os-x-1011-beta.html

    The instructions there were almost correct – something like this worked for me:

    $ mkdir -p /Users/eserv/wrangler-2.0/build-home/6.3.0
    $ cd /Users/eserv/wrangler-2.0/build-home/6.3.0
    $ ln -s /Applications/splunk/lib lib
    $ mkdir -p /Users/eserv/wrangler-2.0/build-home/ember
    $ cd /Users/eserv/wrangler-2.0/build-home/ember
    $ ln -s /Applications/splunk/lib lib

    Then reinstalled from the .dmg and restarted Splunk:

    $ /Applications/Splunk/bin/splunk start

    and the web interface came up at:

    http://127.0.0.1:8000

    Like

  2. Robb Bittner says:

    Rob, Great article and great feedback for us. Really appreciate you taking the time to detail your experience getting started with Splunk Light and the issues you ran into. We will continue to improve the docs and streamline the getting started process.

    One thing I would add is that the Splunk Community is a great place to get information about using and running Splunk:
    http://www.splunk.com/en_us/community.html

    Happy Splunking!

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Archives
Categories

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 5 other followers

%d bloggers like this: