How to install Logstash on Windows Server 2012 with Kibana in IIS.

This post is currently outdated, please have a look here to see a up to date version:
https://community.ulyaoth.net/threads/how-to-install-logstash-on-a-windows-server-with-kibana-in-iis.17/
This guide will be updated as soon as possible.

In this guide I will show that it is also possible to run Logstash on a Windows Server 2012 machine and use IIS as web server. This guide probably requires some improvements and optimizations but it should give you a good example of how to set everything up.

Please, be aware that you will probably have to configure Kibana in a different way then I did to make everything look shiny, and you will probably have to use a different kind of logstash configuration to make things show as you would like. I am also aware that Logstash provides all-in-one pages that have ElasticSearch and Kibana built in, however I still feel setting things up separately is more appropriate.

The config below is just meant to be an example to show that everything works just as fine on Windows as it does on Linux.

If you are interested in Linux then please have a look at my other guide at:

Now lets start with the guide!

Step 1: Download Logstash, Kibana and ElasticSearch.
Simpely go to “http://www.elasticsearch.org/overview/elkdownloads/

Logstash: https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.zip
Kibana: https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.zip
Elasticsearch: https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.zip

Step 2: Extract all packages
I created myself a folder called “basefarm” in “c:\basefarm\” and extracted all folders there to make it easier.

So, for me it looks like this now:
c:\basefarm\elasticsearch
c:\basefarm\kibana
c:\basefarm\logstash

Step 3: Download the JDK version of Java and install it.
Go to the Java website: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
Accept the license and then download: “Windows x64 (jdk-8u5-windows-x64.exe)” package.
Now install it!

Step 4: Add the JAVA_HOME variable to the server
Now right click on “This PC” and choose “Properties” on the right bottom site next to your computer and full computer name click on Change settings.
On the window that opens go to the Advanced tab and click on “Environment Variables”.
at the bottom box called “System Variables” click on “new” and add the following:
Variable Name: JAVA_HOME
Variable value: C:\Program Files\Java\jdk1.8.0_05

It should look like this:

Step 5: Download the required configuration files
Logstash.conf: https://github.com/sbagmeijer/ulyaoth/blob/master/guides/logstash/windows/logstash.conf

Place this file in:
C:\basefarm\logstash\bin

ulyaoth.json:
https://raw.githubusercontent.com/sbagmeijer/ulyaoth/master/guides/logstash/kibana/dashboard/ulyaoth.json

Place this file in:
C:\basefarm\kibana\app\dashboards

rename “ulyaoth.json” to “basefarm.json” so you end up with “C:\basefarm\kibana\app\dashboards\basefarm.json”.

Step 6: Configure Kibana & Logstash
Open the file: C:\basefarm\kibana\config.js

and change the following line:
default_route : ‘/dashboard/file/default.json’,

to:
default_route : ‘/dashboard/file/basefarm.json’,

Now open the file: C:\basefarm\kibana\app\dashboards\basefarm.json

and change the following line:
“title”: “Ulyaoth: Logstash Search”,

to:
“title”: “Basefarm: Logstash Search”,

Step 7: Install IIS
Go to “Server Manager” and choose “Add Roles and Features Wizard” from the list here choose “Web Server (IIS)” now go further and let it install.

Step 8: Open IIS Manager and stop the “Default Web Site”
Just press the stop button like you see below in the picture:

Step 9: Create a new website for Kibana as shown below
Right click on “sites” in the left part of IIS Manager and click “Add Website”.

Fill it in something like this:

It should automatically start.

Step 10: Start Elasticsearch and put it on auto-start
Open a console and go to “c:\basefarm\elasticsearch\bin\”
now type the following command:
service install

You should see something like:

Now type the following:
service manager

You should see the elasticsearch service manager:

You have to change on the tab the “Startup type” from Manual to Automatic and then press “Apply”. This should make Elasticsearch start automatically on server boot.

This window contains some more options such as how much memory Elasticsearch will use. You can find this under the “Java” tab. I would suggest to make this fitfor your server if you have a server that will handle a huge amount of logs. I would increase the “Maximum Memory Pool: 1024” at least to a higher amount.

Before you close the window make sure to press “Start” so it actually will run right now 🙂

This is everything to start ElasticSearch automatically on boot. To test that it is working, open a browser and go to this url: http://127.0.0.1:9200/

If you see a json string something like what you see below in the picture then it means it is running:

Step 11: Start Logstash & Autostart it
For this step we need another small program to create a proper Windows service, so please go ahead and download “NSSM” (the Non-Sucking Service Manager) from: http://nssm.cc/
http://nssm.cc/release/nssm-2.23.zip

Once you have the zip file simply unzip it and copy the file from the unzipped folder you now have: “nssm-2.23\win64” (nssm.exe) to “C:\basefarm\logstash\bin” so it should result in you having “C:\basefarm\logstash\bin\nssm.exe”.

I know you technically do not have to copy this file but just to keep things clean and to have this available for any future use you never know. 🙂

Now open a Command Prompt and type:
cd C:\basefarm\logstash\bin

And then type the following:
nssm install logstash

You will now see a GUI to create a server fill in the following:
Path: C:\basefarm\logstash\bin\logstash.bat
Startup directory: C:\basefarm\logstash\bin
Arguments: agent -f C:/basefarm/logstash/bin/logstash.conf

It should look like this:

If all looks okay double check on the “Details” tab that “Startup Type” is set to “Automatic” and then press “Install service”. This should be all for Logstash to automatically start on server boot.

If you wish to adjust the memory Logstash does use then simpely open the file “C:\basefarm\logstash\bin\logstash.bat” and the change the following two lines accordingly to the amount of memory you wish it to use:
[code]
set LS_MIN_MEM=256m
set LS_MAX_MEM=1g
[/code]

Step 12: Edit your host file (optional)
This step I only do because I run everything on a test server with no internet connection.

open: C:\Windows\System32\drivers\etc\hosts

Now add:
127.0.0.1 loghost.basefarm.com

And save the file.

Now reboot your server so you can test that everything is automatically coming online.

This is all you should have to do once the server is back online you have logstash up and running so just go to:
http://loghost.basefarm.com/

And you should see:

As you can see, your Kibana IIS logs are shipped now to the Logstash instance.

Just remember, if you run this website over the internet you probably need to make sure port 9200 is accessible but I would restrict it to internal use only so Kibana can reach it but not the outside world.

If you want to ship logs from another server to your loghost server I would suggest to have a look into a program called “nxlog” (http://nxlog-ce.sourceforge.net/) this is a fairly simple way of shipping logs to Lgstash and works perfect on Wndows.

If you have any suggestions to improve this guide then please feel free to or update the configs on GitHub or to provide me the information so I can update the guide and help others!

I also would like to thank “Milo Bofacher” for pointing to “nssm” and “nxlog”!

How to install MongoDB on Windows Server 2012 with a replication set.

This post is currently outdated, please have a look here to see a up to date version:
https://community.ulyaoth.net/threads/how-to-install-mongodb-on-windows-with-a-replication-set.18/
This guide will be updated as soon as possible.

In this guide I will show some simple steps of how to set up a MongoDB installation in a replication on Windows Server 2012.

For this setup, I used the following three servers that have Windows 2012 Standard edition installed.

bf-mongodb01: 192.168.1.42 / 4gb ram / 2 cpu
bf-mongodb02: 192.168.1.43 / 4gb ram / 2 cpu
bf-mongodb03: 192.168.1.44 / 4gb ram / 2 cpu

Workgroup: bf-mongodb

They all have to be in the same workgroup or in the same domain. If they are not then you have to add all the servers in your hosts file so mongodb knows how to connect to them.

Also, for this example I installed everything as “Administrator”. Depending on how you are going to use it I would suggest to make a non admin user to run all this.

So lets start!

Step 1: Download MongoDB (on all three servers)
MongoDB provides Windows installer packages, so simply download their msi file from their website http://www.mongodb.org/.
https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-2.6.3-signed.msi

Even if it does say ‘Windows 2008’ it does work perfect on Windows 2012!

Step 2: Install MongoDB
Just follow the pictures below to install MongoDB. You have to do this on all three servers.


Just press ‘next’.


Read the license and then tick the box you accept the license and press ‘next’.


Press you wish to install the “Complete” version.


Just press ‘Install’ to start the installation.


You should now have finished the installation so simpely press ‘Finish’.

Remember, you have to do this on all three of your servers.

Step 3: Create a database and log directory (on all three servers)
Create the following directories:
C:\basefarm\mongodb\data\db
C:\basefarm\mongodb\data\log

Step 4: Fix the Windows firewall (on all three servers).
Go to your “Control Panel” and then click on “Network and Internet”. Once there, click on “Network and Sharing Center” and then at the right side at the bottom click on “Windows Firewall”.

You should now see your Windows firewall like this:

On this window click on “Allow an app or feature trough Windows Firewall” your window will change and click on this window on “Allow another app..” and fill everything in as shown below.

If everything looks as above press on “OK” to close the firewall configuration window.

Step 5: Create a MongoDB config file (on all three servers)
Open a notepad and add the following information:
logpath=C:\basefarm\mongodb\data\log\mongod.log
dbpath=C:\basefarm\mongodb\data\db
bind_ip=0.0.0.0
replSet=bf

Once added, save the file as “mongod.cfg” in the directory:
“C:\Program Files\MongoDB 2.6 Standard\”
You should end up with
“C:\Program Files\MongoDB 2.6 Standard\mongod.cfg”

Step 6: Create a service that will automatically start MongoDB (on all three servers)
Open a Command Prompt and type the following:
sc.exe create MongoDB binPath= "\"C:\Program Files\MongoDB 2.6 Standard\bin\mongod.exe\" --service --config=\"C:\Program Files\MongoDB 2.6 Standard\mongod.cfg\"" DisplayName= "MongoDB 2.6 Standard" start= "auto"

This should create a service for MongoDB. If you did it correct, it should look like this:

Step 7: Start MongoDB (on all three servers)
Just restart the server and MongoDB should automatically start, so it is a good test that the previous commands worked.

Step 8: Go into the MongoDB shell (only on server one)
Go to “C:\Program Files\MongoDB 2.6 Standard\bin” and double click on “mongo”. A terminal window should open that looks like this:

Step 9: Create the replica set in the mongo shell. (only on server one)
While being in the mongo shell type the following commands:
rs.initiate()
rs.add("bf-mongodb02:27017")
rs.add("bf-mongodb03:27017")
cfg = rs.conf()
cfg.members[0].priority = 100
cfg.members[1].priority = 50
cfg.members[2].priority = 50
rs.reconfig(cfg)

Step 10: Test if your configuration is working. (only on server one)
in the mongo shell still type:
rs.status()

You should see something like this if everything is correct:

Congratulations, you now have successfully installed MongoDB on Windows and you have set it up in a replication :)! Now, let’s test that the replication works by creating a database on the master (mongodb01).

Step 11: Create a test collection with some data on the master (only on server one)
In the mongo shell type the following:
use basefarm
bf = { name : "basefarmblog" }
db.Data.insert( bf )
show dbs
show collections
db.Data.find()

You should see something like this:

As you can see “show dbs” did show you have a database Basefarm, “show collections” shows you have the collection Data and “db.Data.find()” shows that the Data collections contains the information “basefarmblog”.

If everything did work as intended, all this should have been replicated to your servers mongodb02 and mongodb03, so let’s test it!

Step 12: Check if the slaves have data (on server two or three)
Go to your server two or three and open a mongo shell by double clicking on the “mongo” file and run the following commands:
show dbs
rs.slaveOk()
use basefarm
show collections
db.Data.find()

If everything worked you should see the following:

The command “rs.slaveOk()” I used because this will you allow to read from the slave. By default this is not enabled.

Well, this was it. Everything worked and you can now use your MongoDB replica set for anything you like!

As always, if you have any improvements or see any mistakes, please let me know. I am always open to hear your ideas or to learn from you!