How to install Logstash on Windows Server 2012 with Kibana in IIS.

This post is currently outdated, please have a look here to see a up to date version:
https://community.ulyaoth.net/threads/how-to-install-logstash-on-a-windows-server-with-kibana-in-iis.17/
This guide will be updated as soon as possible.

In this guide I will show that it is also possible to run Logstash on a Windows Server 2012 machine and use IIS as web server. This guide probably requires some improvements and optimizations but it should give you a good example of how to set everything up.

Please, be aware that you will probably have to configure Kibana in a different way then I did to make everything look shiny, and you will probably have to use a different kind of logstash configuration to make things show as you would like. I am also aware that Logstash provides all-in-one pages that have ElasticSearch and Kibana built in, however I still feel setting things up separately is more appropriate.

The config below is just meant to be an example to show that everything works just as fine on Windows as it does on Linux.

If you are interested in Linux then please have a look at my other guide at:
http://blog.basefarm.com/blog/how-to-install-logstash-with-kibana-interface-on-rhel/

Now lets start with the guide!

Step 1: Download Logstash, Kibana and ElasticSearch.
Simpely go to “http://www.elasticsearch.org/overview/elkdownloads/

Logstash: https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.zip
Kibana: https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.zip
Elasticsearch: https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.zip

Step 2: Extract all packages
I created myself a folder called “basefarm” in “c:\basefarm\” and extracted all folders there to make it easier.

So, for me it looks like this now:
c:\basefarm\elasticsearch
c:\basefarm\kibana
c:\basefarm\logstash

Step 3: Download the JDK version of Java and install it.
Go to the Java website: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
Accept the license and then download: “Windows x64 (jdk-8u5-windows-x64.exe)” package.
Now install it!

Step 4: Add the JAVA_HOME variable to the server
Now right click on “This PC” and choose “Properties” on the right bottom site next to your computer and full computer name click on Change settings.
On the window that opens go to the Advanced tab and click on “Environment Variables”.
at the bottom box called “System Variables” click on “new” and add the following:
Variable Name: JAVA_HOME
Variable value: C:\Program Files\Java\jdk1.8.0_05

It should look like this:

Step 5: Download the required configuration files
Logstash.conf: https://github.com/sbagmeijer/ulyaoth/blob/master/guides/logstash/windows/logstash.conf

Place this file in:
C:\basefarm\logstash\bin

ulyaoth.json:
https://raw.githubusercontent.com/sbagmeijer/ulyaoth/master/guides/logstash/kibana/dashboard/ulyaoth.json

Place this file in:
C:\basefarm\kibana\app\dashboards

rename “ulyaoth.json” to “basefarm.json” so you end up with “C:\basefarm\kibana\app\dashboards\basefarm.json”.

Step 6: Configure Kibana & Logstash
Open the file: C:\basefarm\kibana\config.js

and change the following line:
default_route : ‘/dashboard/file/default.json’,

to:
default_route : ‘/dashboard/file/basefarm.json’,

Now open the file: C:\basefarm\kibana\app\dashboards\basefarm.json

and change the following line:
“title”: “Ulyaoth: Logstash Search”,

to:
“title”: “Basefarm: Logstash Search”,

Step 7: Install IIS
Go to “Server Manager” and choose “Add Roles and Features Wizard” from the list here choose “Web Server (IIS)” now go further and let it install.

Step 8: Open IIS Manager and stop the “Default Web Site”
Just press the stop button like you see below in the picture:

Step 9: Create a new website for Kibana as shown below
Right click on “sites” in the left part of IIS Manager and click “Add Website”.

Fill it in something like this:

It should automatically start.

Step 10: Start Elasticsearch and put it on auto-start
Open a console and go to “c:\basefarm\elasticsearch\bin\”
now type the following command:
service install

You should see something like:

Now type the following:
service manager

You should see the elasticsearch service manager:

You have to change on the tab the “Startup type” from Manual to Automatic and then press “Apply”. This should make Elasticsearch start automatically on server boot.

This window contains some more options such as how much memory Elasticsearch will use. You can find this under the “Java” tab. I would suggest to make this fitfor your server if you have a server that will handle a huge amount of logs. I would increase the “Maximum Memory Pool: 1024” at least to a higher amount.

Before you close the window make sure to press “Start” so it actually will run right now 🙂

This is everything to start ElasticSearch automatically on boot. To test that it is working, open a browser and go to this url: http://127.0.0.1:9200/

If you see a json string something like what you see below in the picture then it means it is running:

Step 11: Start Logstash & Autostart it
For this step we need another small program to create a proper Windows service, so please go ahead and download “NSSM” (the Non-Sucking Service Manager) from: http://nssm.cc/
http://nssm.cc/release/nssm-2.23.zip

Once you have the zip file simply unzip it and copy the file from the unzipped folder you now have: “nssm-2.23\win64” (nssm.exe) to “C:\basefarm\logstash\bin” so it should result in you having “C:\basefarm\logstash\bin\nssm.exe”.

I know you technically do not have to copy this file but just to keep things clean and to have this available for any future use you never know. 🙂

Now open a Command Prompt and type:
cd C:\basefarm\logstash\bin

And then type the following:
nssm install logstash

You will now see a GUI to create a server fill in the following:
Path: C:\basefarm\logstash\bin\logstash.bat
Startup directory: C:\basefarm\logstash\bin
Arguments: agent -f C:/basefarm/logstash/bin/logstash.conf

It should look like this:

If all looks okay double check on the “Details” tab that “Startup Type” is set to “Automatic” and then press “Install service”. This should be all for Logstash to automatically start on server boot.

If you wish to adjust the memory Logstash does use then simpely open the file “C:\basefarm\logstash\bin\logstash.bat” and the change the following two lines accordingly to the amount of memory you wish it to use:
[code]
set LS_MIN_MEM=256m
set LS_MAX_MEM=1g
[/code]

Step 12: Edit your host file (optional)
This step I only do because I run everything on a test server with no internet connection.

open: C:\Windows\System32\drivers\etc\hosts

Now add:
127.0.0.1 loghost.basefarm.com

And save the file.

Now reboot your server so you can test that everything is automatically coming online.

This is all you should have to do once the server is back online you have logstash up and running so just go to:
http://loghost.basefarm.com/

And you should see:

As you can see, your Kibana IIS logs are shipped now to the Logstash instance.

Just remember, if you run this website over the internet you probably need to make sure port 9200 is accessible but I would restrict it to internal use only so Kibana can reach it but not the outside world.

If you want to ship logs from another server to your loghost server I would suggest to have a look into a program called “nxlog” (http://nxlog-ce.sourceforge.net/) this is a fairly simple way of shipping logs to Lgstash and works perfect on Wndows.

If you have any suggestions to improve this guide then please feel free to or update the configs on GitHub or to provide me the information so I can update the guide and help others!

I also would like to thank “Milo Bofacher” for pointing to “nssm” and “nxlog”!

About Sjir Bagmeijer

Works as a Linux Technical Account Manager at Basefarm.
This entry was posted in Technical guides and tagged , , , , , , , . Bookmark the permalink.

28 Responses to How to install Logstash on Windows Server 2012 with Kibana in IIS.

  1. Alex Parris says:

    Great guide!

    Managed to get this working without any real knowledge about the products as part of a project for work.

    Alex

    • Great to hear! If you have any improvements or suggestions please let me know! 🙂

      • Alex Parris says:

        It’s all pretty alien to me at the moment and I managed to follow the guide with no real issues so can’t say there are any improvements to be made 🙂

  2. Marion says:

    Hi !

    Great guide thanks !

    One question though: I have the website up and running on the machine it runs (VM) and I would like to be able to access it from another machine in the corporate domain, but I can’t manage to do that..

    Any clue?

    Thanks!

  3. Hello Marion,

    Glad to hear you got it to work!

    In order to connect from a vm to the corporate domain normally you have to register the vm into the domain also.

    Also you need to make sure that you or use a bridged network so you get a ip from your networks dhcp server or you assign a ip from your network manually.
    Alternative you can add a second network card to your vm that you will use to connect to your domains network.

    I hope that helps you.

  4. karthick says:

    HI ,

    CAN YOU GUIDE ME . AM GETTING THIS ERROR .

    Error Could not load dashboards/logger.json. Please make sure it exists .

    Thanks in advance

    • Normally this means something is from with your Kibana configuration file, but this error is difficult to trouble shoot without actually knowing how everything is setup I am afraid.

  5. raj says:

    Hi,

    I installed logstash central server on ubuntu and I would like to install logstash shipper/agent on windows 2012, can you please let me know how to install agent on 2012 ? What changes I should make in above steps to make it work as an agent?

    Thanks,

    Raj

    • If you want to make logstash a simple agent then you only have to follow the installation steps of Java and Logstash, and then the logstash config file should simply input the log files you wish to ship.

      Just be aware that this way you have to run Java on all your servers to run logstash.

  6. Vagif Abilov says:

    Thanks for a great coverage of Logstash/Kibana setup. But you recommended to use nxlog to ship logs from different machines. Why do you need a different shipper? Can’t you just use Logstash as shipper, or there are some disadvantages with it?

    • I personally would not use Logstash on all servers because it would mean I need to run Java on every server.

      However nothing stops you from using Logstash as a shipper it will works just as fine and you can even put redis in-between also, the nxlog was a example as it is a fairly easy way to ship logs from a windows machine but there are also other programs you can use as long as it can ship to a network address.

      So I think there are so many options that it just comes down on personal preferences.

      • Alex Parris says:

        I found nxlog very simple to install and setup. I now have 25 different web logs shipping from 1 server to a server with logstash/elasticsearch using nxlog 🙂

        • kyle says:

          Can you provide some info on your nxlog configuration? I am trying to get my iis logs to go to elasticsearch and I’m running into issues. I’ve read that certain versions have difficulty sending multiple log files to elasticsearch, but it sounds like you’re not having any issues. Any help would be greatly appreciated!

          • Alex Parris says:

            Hi.

            Below is a sample of the config that is running on various web servers. This is pushing 2 sets of IIS logs from the same server to our central server.

            define ROOT C:\Program Files (x86)\nxlog

            # Standard config for logging etc
            Moduledir %ROOT%\modules
            CacheDir %ROOT%\data
            Pidfile %ROOT%\data\nxlog.pid
            SpoolDir %ROOT%\data
            LogFile %ROOT%\data\nxlog.log

            # Load the json extension

            Module xm_json

            # Select the input folder where logs will be scanned

            Module im_file
            File “D:\LOGFILES\W3SVC2\ex*.log”
            ReadFromLast True
            SavePos True

            # Drop comments from the log file
            Exec if $raw_event =~ /^#/ drop();

            # Select the input folder where logs will be scanned

            Module im_file
            File “D:\LOGFILES\W3SVC3\ex*.log”
            ReadFromLast True
            SavePos True
            # Send the read log lines out to nxlog server

            #Send to central nxlog listener on tcp port 3515, change host address
            Module om_tcp
            Host 10.*.*.*
            Port 3515
            OutputType LineBased

            # Build the route from nxlog on Windows to nxlog on server

            Path w3c2 => out-3515

            Path w3c3 => out-3515

  7. bman says:

    and change the following line:
    default_route : ‘/dashboard/file/default.json’,

    to:
    default_route : ‘/dashboard/file/basefarm.json’,

    no such path exists

  8. Prakash says:

    Hi

    We are trying to install ELK stack on some Python log files. We have install all the ELL tools by following the steps given above. But when, we are trying to execute http://loghost.basefarm.com/ from browser, it is showing the below error message :

    Connection Failed
    Possibility #1: Your elasticsearch server is down or unreachable
    This can be caused by a network outage, or a failure of the Elasticsearch process. If you have recently run a query that required a terms facet to be executed it is possible the process has run out of memory and stopped. Be sure to check your Elasticsearch logs for any sign of memory pressure.
    Possibility #2: You are running Elasticsearch 1.4 or higher
    Elasticsearch 1.4 ships with a security setting that prevents Kibana from connecting. You will need to set http.cors.allow-origin in your elasticsearch.yml to the correct protocol, hostname, and port (if not 80) that your access Kibana from. Note that if you are running Kibana in a sub-url, you should exclude the sub-url path and only include the protocol, hostname and port. For example, http://mycompany.com:8080, not http://mycompany.com:8080/kibana.
    Click back, or the home button, when you have resolved the connection issue

    I have added below line in
    http.cors.allow-origin: “/http?:\/\/localhost(:[0-9]+)?/”
    in elasticsearch.yml but still the problem is not solved.

    Please guide me how to solve this issue.

    Thanks in advance,

  9. vanda says:

    hello my name is vanda, I am a university student studying
    > information systems in Brazil, I am currently working on a research
    > project to implement a treatment tool and storage of logs and would
    > like to learn more about logstash, Kibana and ElasticSearch, the
    > advantages,I have to take into consideration to deploy on windows? researched ELK astack and found it excellent, but I have no knowledge of how to start making thank’s.

  10. Kalyan says:

    Upgrade Required Your version of Elasticsearch is too old. Kibana requires Elasticsearch 0.90.9 or above.

    I am getting above exceptions.
    Below are my versions:
    logstash-1.4.2
    kibana-3.1.2
    elasticsearch-1.4.0

  11. Sai Siddhartha Kambaiyyagari says:

    For not basefarm.json not loaded issue, you refer the below link,
    https://github.com/elasticsearch/kibana/issues/107

  12. Network Error (dns_unresolved_hostname)
    Your requested host “loghost.basefarm.com could not be resolved by DNS.

  13. Kumar says:

    Hello,

    Thanks for the elaborate guide with screenshots.

    I have installed this on Windows 8.1 with IIS installed and the kibana UI comes up just fine. However, it is not displaying any data.

    My analysis:
    1. I have the IIS logs in the same place as per the logstash.conf file & there is data.
    2. The logstash service is also running
    3. Accessing the URL http://127.0.0.1:9200/ displays a valid response:

    {
    status: 200,
    name: “Gladiator”,
    version: {
    number: “1.2.1”,
    build_hash: “6c95b759f9e7ef0f8e17f77d850da43ce8a4b364”,
    build_timestamp: “2014-06-03T15:02:52Z”,
    build_snapshot: false,
    lucene_version: “4.8”
    },
    tagline: “You Know, for Search”
    }

    Can you please help what I might be missing?

  14. Anudeep says:

    Hi I am getting below Error. Please help

    Could not load dashboards/basefarm.json. Please make sure it exists .

    Thanks in Advance

  15. Atul Sirpal says:

    Hi,

    I am getting blank Kibana dashboard after following the steps you mentioned.
    I found
    1.ElasticSearch service is running fine as I can see http://127.0.0.1:9200/ as you wrote.
    2.LogStash Service is also running.

    Can you help me in this?

  16. pankaj says:

    I am trying to implement this for monitoring my VMware Environment so what changes I need to make to work it for VMware Environment

  17. HopesDreams says:

    Thank you so much for the excellent details shown this article.

    I have a question regarding shipping logs from windows servers to the logstash server which is installed on windows; I have installed nxlog on the server I want to ship logs from and I honestly don’t know which details I need to put such as port details of logstash server & certificate details. I am really stuck on this point, your assistance will be appreciated.

    Thank you in advance.

  18. Jagdish says:

    i have done installation in cloud VM, in VM it is running fine, but when i am trying out of network it is showing below error,

    Error Could not contact Elasticsearch at http://104.215.190.120:9200. Please ensure that Elasticsearch is reachable from your system.

    how to resolve this issue, please help me to resolve.

  19. Naveen says:

    hi Sjir Bagmeijer ,

    Thanks for post after lot of googling came to your blog . Its so interesting and simple but I need updated version of this document url

    currently am using kibana and elasticsearch 5.1 version am not using any logstash just logging into a txt file and converting it into json and reading it from elastic and viewing on kibana dashboard.

    -Naveen