Year 2011

Now it’s just some few days left of this year and if we look back, it’s really been an eventful year. Here are some highlights from 2011:

  • 2011 was the year of expansion

Today we are approximately 260 employees and we’re still recruiting. Want to work with us?

  • 2011 was the year of acquisitions

This year we acquired Webdeal in Norway and Bluedome in Holland. This will strengthen our position in Northern Europe and has increased our Basefarm family with almost 40 new employees (only the acquisitions included).

  • 2011 was the year of new products

We launched hybrid hosting to our service offering. Hybrid hosting makes it possible to combine a traditional operating platform with modern public cloud services. Learn more about our service hybrid hosting.

  • 2011 was the year of exploring new market segments

We’ve had a breakthrough in the bank and finance market in Norway and within public sector in Sweden.

  • 2011 was the year of investing for the future

We are building a new colocation center in Norway

  • 2011 was the year of certifications

Basefarm was the first hosting provider in Norway and Sweden to be PCI DSS certificated. In Holland we have also been certificated according to ISO 27001.

  • 2011 was the year of customer satisfaction

We’ve got several new customers in 2011. To mention some of them: Avito (Europe’s largest website), Schipol, Viasat and Kirkerådet (a council for the Norwegian Church). In addition, we have renewed confidence to several customers.

We look forward to a new exciting year in 2012. This year’s christmas present goes to unicef (see christmas card below). We want to thank all our friends for a great year in 2011 by wishing Merry christmas and happy new year!

 

En konferens i molnet…

I torsdags den 1 december var (ännu) en dag i molnets tecken då 2011 års Cloud Conference gick av stapeln i Kista Entré. Personligen uppskattade jag detta format mycket. Spännande keynotes varvas med pauser där deltagare och utställare byter erfarenheter och skapar relationer.

Ser vi till de föredrag som hölls så är det primärt 2 st som lämnat intryck så här några dagar efter.

1. Jan-Erik Gustavsson (CTO) på Ericsson Global Service Center i Indien delgav sin syn på molnet. Förutom att de har en process med att byta merparten av sin IT från traditionell Windows/Office till Ubuntu/OpenOffice med en takt av c:a 500 – 1000 användare i månaden så är de enligt egen utsago världens största NOC. Med 5000+ anställda så blir det ibland så att man har 100+ personer på “bänken” mellan olika projekt. För att lösa detta sjösätter man olika projekt för crowd computing.

2. Carl-Magnus Hallberg (SVP Global IT Services) Nasdaq OMX berättade om alla fördelar med molnet, men att det för deras del varit besparingar som driver. Virtualisering och automatisering är det som gör att de kan lägga mer tid på innovationer/utveckling och mindre tid på förvaltningen (ett argument som även Per Åström på TV4 delger). Med drift av flera börser så omfattas man av kravet att samtliga transaktioner (många) ska lagras 10 år tillbaka i tiden. Det skapar svindlande volymer av data som måste kunna plockas fram.

Annars verkar de flesta ganska eniga om att begreppet “digital natives digital immigrants” börjar slå igenom på allvar. Dagens utvecklare har en hög mognadsgrad och väldigt sofistikerad IT-miljö “hemma” för sin utveckling. Närmast obegränsade dataresurser via t. ex. Amazon, bygger in funktioner från t. ex. Disqus och versionshanterar från Github etc. Nätet fullständigt exploderar just nu med SaaS-tjänster och den interna IT-avdelningen och CIO/CTO upplevs av en del användare som “bromskloss”.

Reflections from Internetdagarna 2011

As we previously have mentioned on basefarm.se, we have been an exhibitor at the swedish digital event Internetdagarna 2011 in Stockholm. It was our first visit to Internetdagarna, but probably not the last! We had many nice discussions in our booth and met several interesting contacts. Moreover, our sales team could be seen in some pictures at Binero’s blog on their photo tour, and they where also seen on the pictures from IT24.

Our sales team would like to share 2 stories from Internetdagarna:

1. Internet is becoming an increasingly important part of the community

Today, everything from banking, insurance, social interaction and media (newspapers, television, news) as well as archives of personal creations on the Internet. But who really watches the Internet? Under the theme “United Nations and The Governance of the Internet” was that the topic of one of Monday’s themes. Should there be a UN for the Internet? Should NATO guard internet also and not only countries? To illustrate this, a history was told that NATO is threated by “hacker groups” because they were investigating and were asked to stop doing so. Read more about this.

2. Amazon is not only a bookstore or a rainforest

Amazon Web Services had sent their technology evangelist Matt Wood (@mza on Twitter) who spoked about the development of “pay-as-you-go” for infrastructure. We really liked this part especially when we (even though we are big in application management) sees a player like AWS as a natural part of the delivery. Can you rent out 30 472 cores for a few hours, there is demonstrably capacity that impress on tech geeks like us.

As a visitor, it was a lot of exiting events. As an exhibitor, we had however liked a bit more and longer pauses between the various speakers. It had given more time to talk to many of the exciting visitors who were there. Now it was a little speed-dating over the breaks.

How to succeed in the IT industry

Yesterday, two of our employees had a career discussion with the students at KYH in Stockholm. They shared their career-related experience in the IT industry, shared their lessons learned from recent career choices and how it’s like to work at Basefarm. We at Basefarm are currently looking for new staff and are always interested in getting in touch with new talent who are passionate about technology as much as we are. We thought that you who are students or young professionals also can benefit of what we discussed:

Tips for you who seeks a career in the IT industry:

  • Have a 3-5 year plan
  • Some adversities are inevitable
  • Working with support is good to learn, even if it’s not your long-term goal in your career
  • Variety is the spice of life, you might need to move/change work to develop
  • You need to find out what you want to do
  • Large companies controls you more, but you can use them to many benefits, both financial and personal
  • Small companies can often develop you more if you are prepared to get involved – Basefarm example
  • Money is not everything, but it helps

Thanks to everyone at KYH who attened our career discussion yesterday and thank’s for all the rewarding conversations afterwards. We hope to see you in the future!

Would you also like us to visit your school or are you interested in a future at Basefarm? Contact us at rekrytering-se@basefarm.se

 

MongoDB 1.8 + 0.2 = 2.0

För ett tag sedan släpptes version 2.0 av denna populära “NoSQL” databas. Version 2.0.0 utlovar 25 % mindre index och 25 % bättre prestanda.

Precis som med CouchDB och andra NoSQL alternativ så frångår MongoDB den traditionella relationsdatabasmodellen till förmån för en mer distribuerad plattform anpassad efter dagens webbapplikationer.

“By reducing transactional semantics, we could still solve an interesting set of problems, but we could also scale” 10gen CEO Dwight Merriman

I takt med att databaser växer sig allt större och mer komplexa riskerar de att bli en flaskhals för populära webbsajter. Detta leder då klart till att utvecklare letar efter nya tekniker för att supporta dem i sin strävan efter att bygga den ultimata sajten.

Läs hela pressreleasen här: http://blog.mongodb.org/post/10126837729/mongodb-2-0-released

pdfforge toolbar causes critical errors in Microsoft Office

Recently one of our customers (who has administrative access and is responsible for their own applications) chose to install the product pdfforge on their server. This server acts as a terminal server, providing Microsoft Office services for multiple users who access it remotely.

Everything had been working fine on this server when one Sunday morning it started throwing errors like this below and the whole of Microsoft Office stopped working.

Activation context generation failed for “C:\Program Files (x86)\ReadSoft\INVOICES\Bin\eiveri.exe.Manifest”.Error in manifest or policy file “” on line . A component version required by the application conflicts with another component version already active. Conflicting components are:. Component 1: C:\Windows\WinSxS\manifests\x86_Microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30304.0 _none_d9c474bda3593bfa.manifest. Component 2: C:\Windows\WinSxS\manifests\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729. 4148_none_5090ab56bcba71c2.manifest.

We soon tracked down the start time of the office to be matching with the install of the pdfforge toolbar v4.7, so without much detailed troubleshooting we suspected this change to be the cause of the problem.

Unfortunately despite uninstalling the offending component and trying other actions like repairing MS Office, we were stuck with the same error.

Now it’s my personal opinion that troubleshooting office shared assemblies is about as much fun as a poke in the eye with a sharp stick, and moreover in my experience you can waste inordinate numbers of hours , if not days, with little to no success. The same often counts for troubleshooting corrupted SSMS installs that are having .NET assembly problems. I’m not saying you can’t do it, I have done it myself, I’d just like to be able to reclaim those days and have devoted them to something worthwhile!

So what I really wanted was a quicker solution, especially after I read numerous reports of this software being considered as spyware / malware or just generally invasive and horrible. I’m not making a judgment on any of these specific points, I’m just saying that if you Google about the toolbar you will find innumerable reports of people complaining that it trashed their systems. Your mileage may vary of course, but I didn’t like what I read at all.

It would appear that pdfforge itself is well respected but that the toolbar contains the more problematic issues. Looking back through our logs we could actually see that the toolbar v4.6 had run without issue for several months. Only the upgrade to v4.7 seemed to cause the issue. What was also worrying was that no-one authorised the v4.7 upgrade, it just installed itself in the background. You can read why in this excellent article:

http://jivebay.com/2010/06/23/pdfforges-pdfcreator-sneaks-in-pdfforge-toolbar/

Anyway, we took the pragmatic approach and ran a system state restore on the server in question to a time before the toolbar upgraded itself. We then advised the customer in question to consider whether they might want to remove this software completely, or at least ensure that it couldn’t download and install upgrades without user interaction.

SQL Server 2008 R2 setup fails due to invalid credentials

A colleague was trying to install a SQL Server 2008 R2 standalone instance today and he kept hitting a weird error when passing the credentials for the service accounts. He was running the setup as a local administrator on the server in question and he was trying to add a domain user as the service account.

The server in question was joined to the appropriate domain already and it had been checked that all the appropriate firewall ports were open. We knew that the user account he was using could query the domain structure as it was able to browse the domain to select the user account from the setup screen.

The errors we got for both accounts were (the text is truncated on the form)

The credentials you provided for the SQL Server Agent service are invalid. To continue…

The specified credentials for the SQL Server service are not valid. To continue…

We took a methodical troubleshooting approach and tested some other domain accounts which we knew were valid and were running other SQL instances elsewhere. These failed as well, meaning that we must have been encountering either some unexpected behaviour within this setup session, or we were being blocked from talking to the domain controllers in some fashion. We again checked the firewalls and they were confirmed as OK.

Then we went into the set logs. The summary log just showed that we had cancelled the installation. For example you see stacks like this which tells you nothing really

2011-09-12 15:59:21 Slp: Exception type: Microsoft.SqlServer.Chainer.Infrastructure.CancelException
2011-09-12 15:59:21 Slp:     Message:
2011-09-12 15:59:21 Slp:         User has cancelled.

However if you look in the detail.txt log file, contained one directory lower, you can scroll from the bottom up and find the actual cause of the problem in the page validation routines. It looks like this (I’ve removed the timestamps for better readability and also blanked all the identifying information obviously)

SQLEngine: –InputValidator: Engine : Attempting to get account sid for account DOMAIN\account
Slp: Sco: Attempting to get account sid for user account DOMAIN\account
Slp: Sco: Attempting to get sid for user account DOMAIN\account
Slp: Sco: GetSidForAccount normalized accountName DOMAIN\account parameter to DOMAIN\account
Slp: Sco: Attempting to get account from sid S-1-5-21-999999999-999999999-999999999-9999
Slp: Sco: Attempting to get account sid for user account DOMAIN\account
Slp: Sco: Attempting to get sid for user account DOMAIN\account
Slp: Sco: GetSidForAccount normalized accountName DOMAIN\account parameter to DOMAIN\account
Slp: Sco: Attempting to get account sid for user account DOMAIN\account
Slp: Sco: Attempting to get sid for user account DOMAIN\account
Slp: Sco: GetSidForAccount normalized accountName DOMAIN\account parameter to DOMAIN\account
Slp: Sco: Attempting to get account sid for user account DOMAIN
Slp: Sco: Attempting to get sid for user account DOMAIN
Slp: Sco: GetSidForAccount normalized accountName DOMAIN parameter to DOMAIN
SQLEngine: –InputValidator: Engine : Service Acccount Specified, Validating Password
Slp: Sco: Attempting to get account sid for user account DOMAIN\account
Slp: Sco: Attempting to get sid for user account DOMAIN\account
Slp: Sco: GetSidForAccount normalized accountName DOMAIN\account parameter to DOMAIN\account
Slp: Sco: Attempting to validate credentials for user account DOMAIN\account
Slp: Sco: Attempting to get account sid for user account DOMAIN\account
Slp: Sco: Attempting to get sid for user account DOMAIN\account
Slp: Sco: GetSidForAccount normalized accountName DOMAIN\account parameter to DOMAIN\account
Slp: Sco: Attempting to get account sid for user account DOMAIN
Slp: Sco: Attempting to get sid for user account DOMAIN
Slp: Sco: GetSidForAccount normalized accountName DOMAIN parameter to DOMAIN
Slp: Sco: Attempting to see if user DOMAIN\account exists
Slp: Sco.User.OpenRoot – Attempting to get root DirectoryEntry for domain/computer ‘DOMAIN’
Slp: Sco: Attempting to check if user account DOMAIN\account exists
Slp: Sco: Attempting to look up AD entry for user DOMAIN\account
Slp: Sco.User.OpenRoot – root DirectoryEntry object already opened for this computer for this object
Slp: Sco.User.LookupADEntry – Attempting to find user account DOMAIN\account
Slp: Sco: Attempting to check if container ‘WinNT://DOMAIN’ of user account exists
Slp: UserSecurity.ValidateCredentials — Exception caught and ignored, exception is Access is denied.
Slp: UserSecurity.ValidateCredentials — user validation failed

I’ve highlighted the problem section. As you can see our account has some permissions on the domain and successfuly gets the SID and various other tasks. However when it comes to the method

Attempting to check if container ‘WinNT://DOMAIN’ of user account exists

it fails…and moreover it then swallows the exception, and then to my amusement actually records that it’s swallowed the exception! To me this is really strange, I guess you could argue that the exception is security related and therefore is swallowed for security protection, but then it records what the error is in the log file, so this seems rather unintuitive to me. The bottom line here is that the account in question doesn’t have the specific privileges on our domain that SQL setup wants here, and so it fails and reports that the service account is invalid. In fact the service account is not invalid, the account used to lookup the service account is invalid. In my mind what should really happen here is that you should get a standard windows AD credentials challenge as the process has caught and handled an access denied error, meaning that it could present this information to the user. But hey, that’s just my opinion.

At the end of the day we changed the setup to use a different account with higher privileges (by running setup with a different logged on user) and everything worked just fine. The key here is that the error is misleading, it’s the interactive account under which you are running setup which has the problem, not the service account you’re trying to add.

SQL Server 2000 cannot start after windows update reboot

Last night we had an incident with one of our customers’  old SQL Server 2000 instances. The machine in question had had it’s WSUS windows update run last night and had been forced to reboot after this had occurred. After this happened the SQL Server service refused to start and just got stuck in a cycle of permanent restarts. When we looked at the error logs we saw the following repeated symptom.

2011-09-12 11:33:00.59 server    Microsoft SQL Server  2000 – 8.00.2039 (Intel X86)
May  3 2005 23:18:38
Copyright (c) 1988-2003 Microsoft Corporation
Standard Edition on Windows NT 5.2 (Build 3790: Service Pack 2)

2011-09-12 11:33:00.59 server    Copyright (C) 1988-2002 Microsoft Corporation.
2011-09-12 11:33:00.59 server    All rights reserved.
2011-09-12 11:33:00.59 server    Server Process ID is 1284.
2011-09-12 11:33:00.59 server    Logging SQL Server messages in file ‘C:\Program Files (x86)\Microsoft SQL Server\MSSQL\log\ERRORLOG’.
2011-09-12 11:33:00.59 server    SQL Server is starting at priority class ‘normal'(4 CPUs detected).
2011-09-12 11:33:00.60 server    SQL Server configured for thread mode processing.
2011-09-12 11:33:00.62 server    Using dynamic lock allocation. [2500] Lock Blocks, [5000] Lock Owner Blocks.
2011-09-12 11:33:00.68 server    Attempting to initialize Distributed Transaction Coordinator.
2011-09-12 11:33:02.84 spid3     Starting up database ‘master’.
2011-09-12 11:33:02.85 server    Using ‘SSNETLIB.DLL’ version ‘8.0.2039’.
2011-09-12 11:33:02.85 spid5     Starting up database ‘model’.
2011-09-12 11:33:02.85 spid3     Server name is ‘XXXXXXXXXX’.
2011-09-12 11:33:02.85 spid8     Starting up database ‘msdb’.
2011-09-12 11:33:02.85 spid9     Starting up database ‘removedforsecurity’.
<SNIP for brevity>
2011-09-12 11:33:02.85 spid23    Starting up database ‘removedforsecurity’.
2011-09-12 11:33:02.85 spid5     Bypassing recovery for database ‘model’ because it is marked IN LOAD.
2011-09-12 11:33:02.85 server    SQL server listening on <removedforsecurity>: xxxx.
2011-09-12 11:33:02.85 server    SQL server listening on <removedforsecurity>: xxxx.
2011-09-12 11:33:02.87 spid20    Starting up database ‘removedforsecurity’.
2011-09-12 11:33:02.89 spid5     Database ‘model’ cannot be opened. It is in the middle of a restore.

The important line here is the last one. Model cannot be opened as it’s marked IN LOAD, this in turn means that tempdb can’t be created which in turn means that the service cannot start.

Quite how model became to be status IN LOAD we’ll never know. I went back through the logs and there was nothing suspicious and no-one has actually attempted to restore it. Circumstantially the evidence points to something being corrupted by windows update, as this is when the problem started, but retrospectively we’ll never be able to say. The MDF and LDF files themselves were intact and seemingly OK, so it was time for a manual attempt to try and get the service back.

The way to recover this (and to troubleshoot it initially) is rather dirty. You have to start the server in single user mode at the console and pass it a couple of trace flags to get you into the system catalogues. You should only ever do this as a last result, but this was a last resort, as it was either do this, or system state restore the entire machine. (I couldn’t restore the SQL Server database as it wouldn’t actually start.) The nature of starting a service in console / single user mode is common enough for SQL machines that won’t start, as you get stuck in a chicken and egg scenario of needing to start it to see why it won’t start! However update the system tables are your peril. In later versions of SQL Server (2005 upwards) the system tables are actually obscured behind views in a separate protected database to make this harder to do, although you can still do it.

The following KB article has a good description of how to do this.

http://support.microsoft.com/kb/822852

The symptoms are the same but the cause was different. That said the solution is also the same in that you have to manually update the status of the database in sysdatabases and then restart the server. There’s no guarantee of success here as it could have been that the MDF file was actually corrupted, but luckily for me it did work and I was saved during a full system restore of the server.

Alltid kul med nytänkande

Med referens till artikeln i IDG http://www.idg.se/2.1085/1.403348/serverrack-ska-radda-jatte-i-kris .

Nöden är väl ändå uppfinningens moder? Trots alla stora och återkommande framsteg som skett genom åren på prestandasidan i form av minne, cpu och format, har väldigt lite hänt på själva plattformssidan. En server anno 2001 är väldigt lik en server idag. Administration och automation av infrastruktur har alltid varit något av en efterkonstruktion vid framtagandet av t.ex. serverprodukter och vi som arbetar med just automation och drift vill gärna uppmuntra fler leverantörer att följa Ciscos exempel.

Att göra en server tillgänglig för produktion innefattar många fler steg än att bara montera servern i racket och sedan slå på strömbrytaren, snarare handlar det om ett 50-tal moment som ska utföras och testas innan man ger tummen upp för produktion. Verktyg som underlättar för oss att kunna systematiskt integrera dom i våra befintliga system och arbetsflöden för att sedan bara kunna klicka på “Kör” stärker både kvaliteten och flexibiliteten för våra kunder.

För att gå tillbaka till framtiden, så vill jag tro att många av oss går och väntar på äkta multitenancy även på hårdvarusidan, där vi kan avbrottsfritt tillföra eller frigöra maskinkraft på låg nivå, men för att då oundvikligen snegla mot stordatorvärlden, så förutsätts då att både hårdvara, OS och applikationer är framtagna och underhålls i harmoni. Och, den moderna termen för detta är väl ändå Platform as A Service (PaaS) och vi får hoppas på att utvecklare hoppar på tåget och går i den riktningen. Än så länge är adaptionen av PaaS väldigt låg för nya produkter och tjänster på nätet och man fastnar i labb-stadiet.

Så, för att återgå i ämnet, är UCS då räddningen för Ciscos framtid? Vem vet, när fler operatörer börjar bygga PaaS plattformar så är Cisco UCS en tilltalande infrastruktur att bygga den kring. Kommer UCS produkten tilltala IT-chefen som köper en server i månaden och har inga automatiserade arbetsflöden eller ambitioner, förmodligen inte. 🙂

/Stefan Månsby

Cannot add new node to windows hyper-v cluster–SCSI disk validation error

We make extensive use of Hyper-V within Basefarm and I recently encountered a strange problem when doing maintenance on a 5 node windows cluster running the hyper-v role. For reasons I won’t bore you with here (pre-production testing basically), I had been evicting and adding nodes to my host level windows cluster, but when trying to add a node back I encountered errors in the validation tests. This was strange as the node had actually previously been in the cluster and nothing had been changed on it whatsoever, so I already knew that it had previously passed the validation test and been successfully running as a cluster node!

The cluster validation reported an error of this format in the storage tests named

Validate SCSI device Vital Product Data (VPD)

cluster_validate_error

It looks like the above picture when you view the validation report.

The actual errors returned were like this:

Failed to get SCSI page 83h VPD descriptors for cluster disk 1 from node <nodename> status 2

(I’ve removed the node name here obviously but it does say specifically which one in the report):

Fortunately before too long I found that this was due to a bug in hyper-v validation for which a hotfix is available here

http://support.microsoft.com/kb/2531907

Downloading and installing this on all the nodes and potential new nodes resolves the error.

This goes to show the value of pre-production testing as the aim of this cluster is to provide a dynamically expanding virtualisation service for one of our largest customers. If it came to a production situation where I needed to add a node quickly, this is not the type of scenario I would be wanting to troubleshoot live!