Posts

Patch Tuesday February 2016

Yet another patch Tuesday has come upon us.
Microsoft released 13 updates, some of which fix critical issues, to address vulnerabilities in their product line. Adobe on the other hand has released patches which address 22 vulnerabilities for their Adobe Flash and Adobe Acrobat/Reader products.
Oracle also pushed out a new update – Java SE 8, Update 73.

Microsoft
Adobe

SandWorm

On Tuesday, October 14, 2014, iSIGHT Partners – in close collaboration with Microsoft – announced the discovery of a zero-day vulnerability impacting all supported versions of Microsoft Windows and Windows Server 2008 and 2012.

Microsoft is making a patch for this vulnerability available as part of patch updates on the 14th – CVE-2014-4114.

Exploitation of this vulnerability was discovered in the wild in connection with a cyber-espionage campaign that iSIGHT Partners attributes to Russia.

This is making the rounds in the news, which isn’t surprising given the potential source as well as targets, but should you as an end user be worried over this? Probably not – in most cases. The vulnerability isn’t released in the wild, which means that you’d need to be the target for a very specific group of people to be hit by this. You should however of course still tread with caution until tomorrow’s Windows Update which will fix this vulnerability.

More information:
http://www.isightpartners.com/2014/10/cve-2014-4114/

How to install Logstash on Windows Server 2012 with Kibana in IIS.

This post is currently outdated, please have a look here to see a up to date version:
https://community.ulyaoth.net/threads/how-to-install-logstash-on-a-windows-server-with-kibana-in-iis.17/
This guide will be updated as soon as possible.

In this guide I will show that it is also possible to run Logstash on a Windows Server 2012 machine and use IIS as web server. This guide probably requires some improvements and optimizations but it should give you a good example of how to set everything up.

Please, be aware that you will probably have to configure Kibana in a different way then I did to make everything look shiny, and you will probably have to use a different kind of logstash configuration to make things show as you would like. I am also aware that Logstash provides all-in-one pages that have ElasticSearch and Kibana built in, however I still feel setting things up separately is more appropriate.

The config below is just meant to be an example to show that everything works just as fine on Windows as it does on Linux.

If you are interested in Linux then please have a look at my other guide at:

Now lets start with the guide!

Step 1: Download Logstash, Kibana and ElasticSearch.
Simpely go to “http://www.elasticsearch.org/overview/elkdownloads/

Logstash: https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.zip
Kibana: https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.zip
Elasticsearch: https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.zip

Step 2: Extract all packages
I created myself a folder called “basefarm” in “c:\basefarm\” and extracted all folders there to make it easier.

So, for me it looks like this now:
c:\basefarm\elasticsearch
c:\basefarm\kibana
c:\basefarm\logstash

Step 3: Download the JDK version of Java and install it.
Go to the Java website: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
Accept the license and then download: “Windows x64 (jdk-8u5-windows-x64.exe)” package.
Now install it!

Step 4: Add the JAVA_HOME variable to the server
Now right click on “This PC” and choose “Properties” on the right bottom site next to your computer and full computer name click on Change settings.
On the window that opens go to the Advanced tab and click on “Environment Variables”.
at the bottom box called “System Variables” click on “new” and add the following:
Variable Name: JAVA_HOME
Variable value: C:\Program Files\Java\jdk1.8.0_05

It should look like this:

Step 5: Download the required configuration files
Logstash.conf: https://github.com/sbagmeijer/ulyaoth/blob/master/guides/logstash/windows/logstash.conf

Place this file in:
C:\basefarm\logstash\bin

ulyaoth.json:
https://raw.githubusercontent.com/sbagmeijer/ulyaoth/master/guides/logstash/kibana/dashboard/ulyaoth.json

Place this file in:
C:\basefarm\kibana\app\dashboards

rename “ulyaoth.json” to “basefarm.json” so you end up with “C:\basefarm\kibana\app\dashboards\basefarm.json”.

Step 6: Configure Kibana & Logstash
Open the file: C:\basefarm\kibana\config.js

and change the following line:
default_route : ‘/dashboard/file/default.json’,

to:
default_route : ‘/dashboard/file/basefarm.json’,

Now open the file: C:\basefarm\kibana\app\dashboards\basefarm.json

and change the following line:
“title”: “Ulyaoth: Logstash Search”,

to:
“title”: “Basefarm: Logstash Search”,

Step 7: Install IIS
Go to “Server Manager” and choose “Add Roles and Features Wizard” from the list here choose “Web Server (IIS)” now go further and let it install.

Step 8: Open IIS Manager and stop the “Default Web Site”
Just press the stop button like you see below in the picture:

Step 9: Create a new website for Kibana as shown below
Right click on “sites” in the left part of IIS Manager and click “Add Website”.

Fill it in something like this:

It should automatically start.

Step 10: Start Elasticsearch and put it on auto-start
Open a console and go to “c:\basefarm\elasticsearch\bin\”
now type the following command:
service install

You should see something like:

Now type the following:
service manager

You should see the elasticsearch service manager:

You have to change on the tab the “Startup type” from Manual to Automatic and then press “Apply”. This should make Elasticsearch start automatically on server boot.

This window contains some more options such as how much memory Elasticsearch will use. You can find this under the “Java” tab. I would suggest to make this fitfor your server if you have a server that will handle a huge amount of logs. I would increase the “Maximum Memory Pool: 1024” at least to a higher amount.

Before you close the window make sure to press “Start” so it actually will run right now 🙂

This is everything to start ElasticSearch automatically on boot. To test that it is working, open a browser and go to this url: http://127.0.0.1:9200/

If you see a json string something like what you see below in the picture then it means it is running:

Step 11: Start Logstash & Autostart it
For this step we need another small program to create a proper Windows service, so please go ahead and download “NSSM” (the Non-Sucking Service Manager) from: http://nssm.cc/
http://nssm.cc/release/nssm-2.23.zip

Once you have the zip file simply unzip it and copy the file from the unzipped folder you now have: “nssm-2.23\win64” (nssm.exe) to “C:\basefarm\logstash\bin” so it should result in you having “C:\basefarm\logstash\bin\nssm.exe”.

I know you technically do not have to copy this file but just to keep things clean and to have this available for any future use you never know. 🙂

Now open a Command Prompt and type:
cd C:\basefarm\logstash\bin

And then type the following:
nssm install logstash

You will now see a GUI to create a server fill in the following:
Path: C:\basefarm\logstash\bin\logstash.bat
Startup directory: C:\basefarm\logstash\bin
Arguments: agent -f C:/basefarm/logstash/bin/logstash.conf

It should look like this:

If all looks okay double check on the “Details” tab that “Startup Type” is set to “Automatic” and then press “Install service”. This should be all for Logstash to automatically start on server boot.

If you wish to adjust the memory Logstash does use then simpely open the file “C:\basefarm\logstash\bin\logstash.bat” and the change the following two lines accordingly to the amount of memory you wish it to use:
[code]
set LS_MIN_MEM=256m
set LS_MAX_MEM=1g
[/code]

Step 12: Edit your host file (optional)
This step I only do because I run everything on a test server with no internet connection.

open: C:\Windows\System32\drivers\etc\hosts

Now add:
127.0.0.1 loghost.basefarm.com

And save the file.

Now reboot your server so you can test that everything is automatically coming online.

This is all you should have to do once the server is back online you have logstash up and running so just go to:
http://loghost.basefarm.com/

And you should see:

As you can see, your Kibana IIS logs are shipped now to the Logstash instance.

Just remember, if you run this website over the internet you probably need to make sure port 9200 is accessible but I would restrict it to internal use only so Kibana can reach it but not the outside world.

If you want to ship logs from another server to your loghost server I would suggest to have a look into a program called “nxlog” (http://nxlog-ce.sourceforge.net/) this is a fairly simple way of shipping logs to Lgstash and works perfect on Wndows.

If you have any suggestions to improve this guide then please feel free to or update the configs on GitHub or to provide me the information so I can update the guide and help others!

I also would like to thank “Milo Bofacher” for pointing to “nssm” and “nxlog”!

How to install MongoDB on Windows Server 2012 with a replication set.

This post is currently outdated, please have a look here to see a up to date version:
https://community.ulyaoth.net/threads/how-to-install-mongodb-on-windows-with-a-replication-set.18/
This guide will be updated as soon as possible.

In this guide I will show some simple steps of how to set up a MongoDB installation in a replication on Windows Server 2012.

For this setup, I used the following three servers that have Windows 2012 Standard edition installed.

bf-mongodb01: 192.168.1.42 / 4gb ram / 2 cpu
bf-mongodb02: 192.168.1.43 / 4gb ram / 2 cpu
bf-mongodb03: 192.168.1.44 / 4gb ram / 2 cpu

Workgroup: bf-mongodb

They all have to be in the same workgroup or in the same domain. If they are not then you have to add all the servers in your hosts file so mongodb knows how to connect to them.

Also, for this example I installed everything as “Administrator”. Depending on how you are going to use it I would suggest to make a non admin user to run all this.

So lets start!

Step 1: Download MongoDB (on all three servers)
MongoDB provides Windows installer packages, so simply download their msi file from their website http://www.mongodb.org/.
https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-2.6.3-signed.msi

Even if it does say ‘Windows 2008’ it does work perfect on Windows 2012!

Step 2: Install MongoDB
Just follow the pictures below to install MongoDB. You have to do this on all three servers.


Just press ‘next’.


Read the license and then tick the box you accept the license and press ‘next’.


Press you wish to install the “Complete” version.


Just press ‘Install’ to start the installation.


You should now have finished the installation so simpely press ‘Finish’.

Remember, you have to do this on all three of your servers.

Step 3: Create a database and log directory (on all three servers)
Create the following directories:
C:\basefarm\mongodb\data\db
C:\basefarm\mongodb\data\log

Step 4: Fix the Windows firewall (on all three servers).
Go to your “Control Panel” and then click on “Network and Internet”. Once there, click on “Network and Sharing Center” and then at the right side at the bottom click on “Windows Firewall”.

You should now see your Windows firewall like this:

On this window click on “Allow an app or feature trough Windows Firewall” your window will change and click on this window on “Allow another app..” and fill everything in as shown below.

If everything looks as above press on “OK” to close the firewall configuration window.

Step 5: Create a MongoDB config file (on all three servers)
Open a notepad and add the following information:
logpath=C:\basefarm\mongodb\data\log\mongod.log
dbpath=C:\basefarm\mongodb\data\db
bind_ip=0.0.0.0
replSet=bf

Once added, save the file as “mongod.cfg” in the directory:
“C:\Program Files\MongoDB 2.6 Standard\”
You should end up with
“C:\Program Files\MongoDB 2.6 Standard\mongod.cfg”

Step 6: Create a service that will automatically start MongoDB (on all three servers)
Open a Command Prompt and type the following:
sc.exe create MongoDB binPath= "\"C:\Program Files\MongoDB 2.6 Standard\bin\mongod.exe\" --service --config=\"C:\Program Files\MongoDB 2.6 Standard\mongod.cfg\"" DisplayName= "MongoDB 2.6 Standard" start= "auto"

This should create a service for MongoDB. If you did it correct, it should look like this:

Step 7: Start MongoDB (on all three servers)
Just restart the server and MongoDB should automatically start, so it is a good test that the previous commands worked.

Step 8: Go into the MongoDB shell (only on server one)
Go to “C:\Program Files\MongoDB 2.6 Standard\bin” and double click on “mongo”. A terminal window should open that looks like this:

Step 9: Create the replica set in the mongo shell. (only on server one)
While being in the mongo shell type the following commands:
rs.initiate()
rs.add("bf-mongodb02:27017")
rs.add("bf-mongodb03:27017")
cfg = rs.conf()
cfg.members[0].priority = 100
cfg.members[1].priority = 50
cfg.members[2].priority = 50
rs.reconfig(cfg)

Step 10: Test if your configuration is working. (only on server one)
in the mongo shell still type:
rs.status()

You should see something like this if everything is correct:

Congratulations, you now have successfully installed MongoDB on Windows and you have set it up in a replication :)! Now, let’s test that the replication works by creating a database on the master (mongodb01).

Step 11: Create a test collection with some data on the master (only on server one)
In the mongo shell type the following:
use basefarm
bf = { name : "basefarmblog" }
db.Data.insert( bf )
show dbs
show collections
db.Data.find()

You should see something like this:

As you can see “show dbs” did show you have a database Basefarm, “show collections” shows you have the collection Data and “db.Data.find()” shows that the Data collections contains the information “basefarmblog”.

If everything did work as intended, all this should have been replicated to your servers mongodb02 and mongodb03, so let’s test it!

Step 12: Check if the slaves have data (on server two or three)
Go to your server two or three and open a mongo shell by double clicking on the “mongo” file and run the following commands:
show dbs
rs.slaveOk()
use basefarm
show collections
db.Data.find()

If everything worked you should see the following:

The command “rs.slaveOk()” I used because this will you allow to read from the slave. By default this is not enabled.

Well, this was it. Everything worked and you can now use your MongoDB replica set for anything you like!

As always, if you have any improvements or see any mistakes, please let me know. I am always open to hear your ideas or to learn from you!

Quick way to name your NICs in Windows Servers

If you, like me, manage many servers, it’s essential to name network adapters in a way that makes it easy to troubleshoot issues when they arise.

In complex networks with thousands of servers and all servers connected using multiple paths a consistent naming standard is very important!

PowerShell and the cmdlets available in Windows Server makes naming adapters a breeze. The servers we usualy deploy have built in four (4) port network adapters. We like to name the Windows NICs the same as is the default in Linux; eth0, eth1, etc.

In the following example we name the adapters eth0, eth1, eth2 and eth3 in Windows. The NIC with the lowest MAC address gets the name eth0 etc. (If you prefer to to start naming adapters from eth1 change the variable $NICs to 0):


$NICs = -1
Get-NetAdapter Etherne* | Sort-Object MacAddress | % { Rename-NetAdapter -InterfaceAlias $_.InterfaceAlias -NewName eth$NICs }

PowerShell really makes life easy 😉

Windows Server 2012 is coming!

A week from today Microsoft releases Windows Server 2012. For ordinary computer users this release may not mean a lot, but for us working with running large server systems it will be a game changer.

Fundamental parts of the Windows Server operating system have been changed. Some changes are visible such as the lack of a graphical user interface on a standard server. Other changes are less visible; new storage options, filesystems etc.

A very big change for operations is that PowerShell really have moved into the core of managing Windows. This will allow us to automate more than before, with ease!

I won’t go into all the details here but if you want to be part of the launch event for Windows Server 2012, setup a reminder here.

Configuring Windows Server 2008 R2 Features

At Basefarm we frequently need to ensure that many Windows servers are identical in terms of the roles and features they have installed. Adding features can be done in a number of ways. Mostly the graphical userinterface (Server Manager) is used. Or for large operations System Center or similar. I will show you how this can be done more easily using the command line. This method doesn’t require anything beyond Windows Server 2008 R2 (or later) and PowerShell.

The Server Manager module

The Server Manager module (introduced with Windows Server 2008 R2) has three very useful commands, they are:

  • Add-WindowsFeature
  • Get-WindowsFeature
  • Remove-WindowsFeature

Using these is simple. Start a PowerShell session with administrative privileges (Run As…) . Then check that the Servermanager module is available in your server:

PS C:\> Get-Module -ListAvailable

Get-Module -ListAvailable

This shows that the Server Manager module is available on our server but that it is not yet loaded into the PowerShell session. To load it (and make its commands available):

PS C:\> Import-Module Servermanager

Now the commands of the Server Manager module are available to you. Check which commands are exposed by the module:

PS C:\> Get-Command -Module Servermanager

Ok, we’re all set. Let’s use these commands!

HOWTO: Document what is installed

To see what is installed in a server use:

PS C:\> Get-WindowsFeature

Get-WindowsFeature

ooops, that’s a lot of text flying by on the screen! As you probably can guess only lines with [X] are installed. So we need to filter the list to only show what is actually installed, try this instead:

PS C:\> Get-WindowsFeature | ? { $_.Installed }

Get-WindowsFeature-installed

A nice clean list showing which features are installed on the server ;-), perfect for documenting your server(s)

HOWTO: Clone installed features to another server

As shown above it’s easy to list what is installed. But just having this list on the screen doesn’t make much sense, we need to be able to store this in a structured way so that we can use the list on another server to install the same features. PowerShell makes this very simple. We use the Export-CliXml cmdlet to save the information in a structured XML file:

PS C:\> Get-WindowsFeature | ? { $_.Installed } | Export-Clixml .\features.xml

The output from the Get-WindowsFeature cmdlet is saved in a structured way in the XML file features.xml. This file can now shared to other servers and used as input for the Add-WindowsFeature cmdlet!

HOWTO: Add features from another server (using XML file)

Start PowerShell with administrative privileges.  Now try this:

PS C:\> Import-Module Servermanager
PS C:\> Import-Clixml .\features.xml

Now you have the same list of installed features on the new server. But… this is simply a list in memory and on screen. The features haven’t been added yet. In order to do that we need to pipe the information into the Add-WindowsFeature cmdlet.

Before I show you how to do that there is one important thing I need to explain. When we exported the list of installed features we included all features that were marked as installed. As you saw in the output this resulted in a tree like structure where “[X] Web Server (IIS)” was on the top followed by “[X] Web Server” and so on.

That looks fine but if we use this as input for the Add-WindowsFeature cmdlet we will end up with more than we asked for. The reason is that when the top level feature such as “Web Server (IIS)” is choosen everyting underneath it will also be installed. And in order to keep our servers a lean as possible we do not want this! We need to go back and filter the output of Get-WindowsFeatre a little more. Try this instead of what I showed you earlier:

PS C:\> Get-WindowsFeature | ? {$_.Installed -AND $_.SubFeatures.Count -eq 0 }

Now the output will only contain information from the bottom-up so to speak. This works fine as input for the next server we want to make identical. Save the new list to a file:

PS C:\> Get-WindowsFeature | ? {$_.Installed -AND $_.SubFeatures.Count -eq 0 } | Export-Clixml .\features.xml

Now we can finally install these features in the new server:

PS C:\> Import-Clixml .\features.xml | Add-WindowsFeature

Est Voilá! The two servers now have the same Windows features installed.

As always with PowerShell, if your environment enables PowerShell remoting these commands could be executed on any number of servers from a single commandline. A Power(full)Shell that is!

Summary

This became a longer post than I intended simply because I wanted to explain the details about filtering the export. Here’s a Quick summary of the commands you use to export what is installed:

PS C:\> Import-Module Servermanager
PS C:\> Get-WindowsFeature | ? {$_.Installed -AND $_.SubFeatures.Count -eq 0 } | Export-Clixml .\filename.xml

Copy the file ‘filename.xml’ to a network share or other location where the next server can reach it, then do this on the other server:

PS C:\> Import-Module Servermanager
PS C:\> Import-Clixml .\filename.xml | Add-WindowsFeature

All features are installed on the new server without having to click-around in the graphical server manager! To verify what is installed quickly use:

PS C:\> Get-WindowsFeature | ? { $_.Installed }

I hope I have showed you that PowerShell is much better than giving your arms RSI using the mouse to handle feature installations!

Cannot add new node to windows hyper-v cluster–SCSI disk validation error

We make extensive use of Hyper-V within Basefarm and I recently encountered a strange problem when doing maintenance on a 5 node windows cluster running the hyper-v role. For reasons I won’t bore you with here (pre-production testing basically), I had been evicting and adding nodes to my host level windows cluster, but when trying to add a node back I encountered errors in the validation tests. This was strange as the node had actually previously been in the cluster and nothing had been changed on it whatsoever, so I already knew that it had previously passed the validation test and been successfully running as a cluster node!

The cluster validation reported an error of this format in the storage tests named

Validate SCSI device Vital Product Data (VPD)

cluster_validate_error

It looks like the above picture when you view the validation report.

The actual errors returned were like this:

Failed to get SCSI page 83h VPD descriptors for cluster disk 1 from node <nodename> status 2

(I’ve removed the node name here obviously but it does say specifically which one in the report):

Fortunately before too long I found that this was due to a bug in hyper-v validation for which a hotfix is available here

http://support.microsoft.com/kb/2531907

Downloading and installing this on all the nodes and potential new nodes resolves the error.

This goes to show the value of pre-production testing as the aim of this cluster is to provide a dynamically expanding virtualisation service for one of our largest customers. If it came to a production situation where I needed to add a node quickly, this is not the type of scenario I would be wanting to troubleshoot live!

How to resolve orphaned file ownership in windows 2008

Here’s something I was looking at this morning, which is not an uncommon problem I think. I was doing some disk cleaning for a customer and some extremely large files that we needed to remove were locked and inaccessible even by a local administrator on the machine. There are multiple reasons why this can occur, but in this case it was because the original file was created by a domain user account which had subsequently been disabled.

This meant that if you tried to delete the file, or take ownership of it using standard windows controls you would be refused access.

In this case I resolved using the takeown command line tool that comes directly in windows

http://technet.microsoft.com/en-us/library/cc753024(WS.10).aspx

It was actually a 3 stage process.

1. Run takeown with the /F switch to grant ownership to the current user (if you want you can grant ownership to a different user) but I simply logged on as a local administrator and took ownership using this login.

2. Following this you have ownership but you still can’t delete or move the file. You need to go into the security tab and explicitly grant access to a login (can include yourself) to give the new user full control

image

3. Once this is complete you can simply delete the file using the account you have granted permission to. (Before using takeown you are unable to access this security tab.)

It’s probably more sensible to grant the permission to an administrative group in the longer term, if you’re not just simply deleting the files. It’s worth noting as well that takeown can do the same functions to groups of files or directories themselves, all of which can experience the same problem.