How to install Logstash on Windows Server 2012 with Kibana in IIS.

This post is currently outdated, please have a look here to see a up to date version: https://ulyaoth.com/tutorials/how-to-install-elastic-stack-5-4-on-windows-server-2016/

This post is a repost from Sjir Bagmeijer’s personal website, you can find a more up to date post on his website:
https://ulyaoth.com/tutorials/how-to-install-elastic-stack-5-4-on-windows-server-2016/

In this guide I will show that it is also possible to run Logstash on a Windows Server 2012 machine and use IIS as web server. This guide probably requires some improvements and optimizations but it should give you a good example of how to set everything up.

Please, be aware that you will probably have to configure Kibana in a different way then I did to make everything look shiny, and you will probably have to use a different kind of logstash configuration to make things show as you would like. I am also aware that Logstash provides all-in-one pages that have ElasticSearch and Kibana built in, however I still feel setting things up separately is more appropriate.

The config below is just meant to be an example to show that everything works just as fine on Windows as it does on Linux.

If you are interested in Linux then please have a look at my other guide at:

Now lets start with the guide!

Step 1: Download Logstash, Kibana and ElasticSearch.
Simpely go to “http://www.elasticsearch.org/overview/elkdownloads/

Logstash: https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.zip
Kibana: https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.zip
Elasticsearch: https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.zip

Step 2: Extract all packages
I created myself a folder called “basefarm” in “c:\basefarm\” and extracted all folders there to make it easier.

So, for me it looks like this now:
c:\basefarm\elasticsearch
c:\basefarm\kibana
c:\basefarm\logstash

Step 3: Download the JDK version of Java and install it.
Go to the Java website: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
Accept the license and then download: “Windows x64 (jdk-8u5-windows-x64.exe)” package.
Now install it!

Step 4: Add the JAVA_HOME variable to the server
Now right click on “This PC” and choose “Properties” on the right bottom site next to your computer and full computer name click on Change settings.
On the window that opens go to the Advanced tab and click on “Environment Variables”.
at the bottom box called “System Variables” click on “new” and add the following:
Variable Name: JAVA_HOME
Variable value: C:\Program Files\Java\jdk1.8.0_05

It should look like this:

Step 5: Download the required configuration files
Logstash.conf: https://github.com/sbagmeijer/ulyaoth/blob/master/guides/logstash/windows/logstash.conf

Place this file in:
C:\basefarm\logstash\bin

ulyaoth.json:
https://raw.githubusercontent.com/sbagmeijer/ulyaoth/master/guides/logstash/kibana/dashboard/ulyaoth.json

Place this file in:
C:\basefarm\kibana\app\dashboards

rename “ulyaoth.json” to “basefarm.json” so you end up with “C:\basefarm\kibana\app\dashboards\basefarm.json”.

Step 6: Configure Kibana & Logstash
Open the file: C:\basefarm\kibana\config.js

and change the following line:
default_route : ‘/dashboard/file/default.json’,

to:
default_route : ‘/dashboard/file/basefarm.json’,

Now open the file: C:\basefarm\kibana\app\dashboards\basefarm.json

and change the following line:
“title”: “Ulyaoth: Logstash Search”,

to:
“title”: “Basefarm: Logstash Search”,

Step 7: Install IIS
Go to “Server Manager” and choose “Add Roles and Features Wizard” from the list here choose “Web Server (IIS)” now go further and let it install.

Step 8: Open IIS Manager and stop the “Default Web Site”
Just press the stop button like you see below in the picture:

Step 9: Create a new website for Kibana as shown below
Right click on “sites” in the left part of IIS Manager and click “Add Website”.

Fill it in something like this:

It should automatically start.

Step 10: Start Elasticsearch and put it on auto-start
Open a console and go to “c:\basefarm\elasticsearch\bin\”
now type the following command:
service install

You should see something like:

Now type the following:
service manager

You should see the elasticsearch service manager:

You have to change on the tab the “Startup type” from Manual to Automatic and then press “Apply”. This should make Elasticsearch start automatically on server boot.

This window contains some more options such as how much memory Elasticsearch will use. You can find this under the “Java” tab. I would suggest to make this fitfor your server if you have a server that will handle a huge amount of logs. I would increase the “Maximum Memory Pool: 1024” at least to a higher amount.

Before you close the window make sure to press “Start” so it actually will run right now 🙂

This is everything to start ElasticSearch automatically on boot. To test that it is working, open a browser and go to this url: http://127.0.0.1:9200/

If you see a json string something like what you see below in the picture then it means it is running:

Step 11: Start Logstash & Autostart it
For this step we need another small program to create a proper Windows service, so please go ahead and download “NSSM” (the Non-Sucking Service Manager) from: http://nssm.cc/
http://nssm.cc/release/nssm-2.23.zip

Once you have the zip file simply unzip it and copy the file from the unzipped folder you now have: “nssm-2.23\win64” (nssm.exe) to “C:\basefarm\logstash\bin” so it should result in you having “C:\basefarm\logstash\bin\nssm.exe”.

I know you technically do not have to copy this file but just to keep things clean and to have this available for any future use you never know. 🙂

Now open a Command Prompt and type:
cd C:\basefarm\logstash\bin

And then type the following:
nssm install logstash

You will now see a GUI to create a server fill in the following:
Path: C:\basefarm\logstash\bin\logstash.bat
Startup directory: C:\basefarm\logstash\bin
Arguments: agent -f C:/basefarm/logstash/bin/logstash.conf

It should look like this:

If all looks okay double check on the “Details” tab that “Startup Type” is set to “Automatic” and then press “Install service”. This should be all for Logstash to automatically start on server boot.

If you wish to adjust the memory Logstash does use then simpely open the file “C:\basefarm\logstash\bin\logstash.bat” and the change the following two lines accordingly to the amount of memory you wish it to use:
[code]
set LS_MIN_MEM=256m
set LS_MAX_MEM=1g
[/code]

Step 12: Edit your host file (optional)
This step I only do because I run everything on a test server with no internet connection.

open: C:\Windows\System32\drivers\etc\hosts

Now add:
127.0.0.1 loghost.basefarm.com

And save the file.

Now reboot your server so you can test that everything is automatically coming online.

This is all you should have to do once the server is back online you have logstash up and running so just go to:
http://loghost.basefarm.com/

And you should see:

As you can see, your Kibana IIS logs are shipped now to the Logstash instance.

Just remember, if you run this website over the internet you probably need to make sure port 9200 is accessible but I would restrict it to internal use only so Kibana can reach it but not the outside world.

If you want to ship logs from another server to your loghost server I would suggest to have a look into a program called “nxlog” (http://nxlog-ce.sourceforge.net/) this is a fairly simple way of shipping logs to Lgstash and works perfect on Wndows.

If you have any suggestions to improve this guide then please feel free to or update the configs on GitHub or to provide me the information so I can update the guide and help others!

I also would like to thank “Milo Bofacher” for pointing to “nssm” and “nxlog”!

Resizing mgmt_tablespace in Grid Control 12c and reclaiming space

We noticed the MGMT_TABLESPACE in our 12c Grid Control production database was very big. Almost 380gb of data was stored in this tablespace.

I did not think anything was wrong because of all the targets registered in this Grid Control instance and the frequent usage of this Grid Control.

Until someone looked into this and found out it was a bug in 12c…

Note 1502370.1 describes this bug and also the solution for this. I had to install a patch and truncate the em_job_metrics table. After applying the patch and truncating the table, there was only around 10gb of data left in the MGMT_TABLESPACE.

Because I already experienced something like this five years ago in a 10g OEM database , I knew that it is not easy to reclaim the space in this tablespace. And we had a lot to reclaim, only 10gb data in this tablespace of 380gb…

I decided to check my Oracle support, maybe someone else already had the same problem. I found bug 17461366, making it impossible to reorg the mgmt_tablespace because of aq objects. Same problem I faced five years ago. Because I really wanted to reclaim the free space (370gb), I decided to follow my own notes. Although these steps ware performed on a 10g OEM environment and now I was working with a 12c Grid Control environment.

Just like 5 years ago, I could think of two ways to reclaim the space:

1. export mgmt_tablespace, drop the tablespace and import it again

2. export sysman, drop repository and, run scripts from the oms_home and import sysman again

Export MGMT_TABLESPACE

Unfortunately, dropping the mgmt_tablespace was a mission impossible. The export succeeded without errors, but I was not able to drop the tablespace. After dropping some related objects that were showing in the errors I received, I decided this was not going to work. I restored the database and executed option 2.

Export Sysman user(s)

I decided to follow note 388090.1. This note describes a platform migration for a 10g Grid Control environment, but I could not find any document about a 12c Grid Control environment.

Just to be sure, I also exported (expdp) the other sysman users (sysman_apm, sysman_mds, sysman_opss and sysman). Also set job_queue_processes to zero

The next step was to drop the current repository using the repmanger utility ($OMS_HOME/sysman/admin/emdrep/bin)

The repmanager also did not drop the mgmt_tablespace, I had to drop the tablespace myself. But after deleting the repository I was able to drop the tablespace. The drop repository did drop all the sysman users and other tablespaces

According to the note, before the impdp, I had to run some scripts from the $OMS_HOME/sysman/admin/emdrep/sql/core/latest/admin directory.

These script should create the sysman users and tablespaces. The following sql’s were executed :

– admin_create_tablespaces.sql

– admin_create_repos_user.sql

– admin_pre_import.sql

– admin_sys_procs.sql

– admin_profiles.sql

– admin_grants_repos_user.sql

– admin_grants_view_user.sql

Now, you should be ready to inport the sysman scheme again (impdp). But the import showed a lot of errors. Mean reason, the sysman_ro user was not available and the mgmt_ad4j_ts tablespace was not created. Again I decided to drop the sysman user again and start over again. But this time, instead of running the scripts before the import I created a whole new repository using Repmanager.

The new repository did have the sysman_ro user and also the missing tablespace.

At this time I started the import again, using the ‘table_exist_action=replace’ option. The repository creation already created the sysman objects, so I wanted the impdp to replace the already created tables with the tables from the dump. I noticed one table creation error in the logfile, the em_job_type_creds_info table was not created by impdp. I had to create this one after the impdp.

Also the other sysman users were not created by the repmanager. I had to create and import these users (sysman_.mds, sysman_apm and sysman_opss).

After the import I returned to the note again, to check for post import steps.

Again in $OMS_HOME/sysman/admin/emdrep/sql/core/latest/admin

– admin_recompile_invalid.sql

– admin_create_synonyms.sql

– admin_post_import.sql

Reset the job_queue_processes to its original  value and submit the EM dbms jobs

– admin_submit_dbms_jobs.sql

After compiling, there were still problems with aq objects.

I dropped the aq objects (exec dbms_aqadm.drop_queue_table) and created (dbms_aqadm.create_queue_table) them again. This solved the problems and also cleared the invalid objects

At this point, there were no invalid objects anymore, so the oms could be started again to see what happens.

Oms started oke and I was able to login 12c Grid Control.

With 12c Grid Control running, I noticed the oms was bouncing every 12 minutes by itself.

I checked the emctl.msg file in /gc_inst/em/EMGC_OMS1/sysman/log for errors and found the following error:

HealthMonitor Nov 15, 2013 12:57:48 PM PbsAdminMsgListener error: PbsAdminMsgListener thread timed out.
Critical error err=3 detected in module PbsAdminMsgListener
OMS will be restarted. A full thread dump will be generated in the log file

It seemed the em_cntr_queue was missing? I checked and found out some other queues were also missing (not in the dump?) I recreated the missing queues (check the queues in an OTA Grid environment) and that solved the restart error. Now oms was not restarting by itself anymore.

Reclaimable free space

After these steps (took me two days…) I reclaimed 370gb. The mgmt_tablespace is now 10gb (instead of the 380gb before). The total size of the entire database was shrinked from more than 400gb to 22gb!!