2011-12-05 20:25
Today I was able to track down the problem. We had a crash last saturday, and today we were near before the same situation.
First of all, we are a "victim" of our own success. We calculated the expected server load based on some statistics gathered during the tests. We have seen around 5 % of all the managers online at the same time, while each user opens around 1.3 windows and therefore server sessions. This sums up to the formula USERCOUNT * 1.3 * 5%. We are around 300 users per today. The result of the formula would be around 20 sessions at the same time....
The server was sized for around 800 users. This would result of 52 sessions at the same time.
What to tell, we made the calculation without involving you!
Our 300 users are very very active. This results in around 30-50 users online at the same time (peak value) - this is around 10-15% of our userbase. Not enough, each of you opens around 2.2 windows. This results in around 60 to 100+ open sessions at the same time.
What happens? The server runs out of memory, the database has to work in the the swap space, which kills the performance and makes the processes slower by factor "several hundred". This results in more users online at the same time, with the simulation processes on top...the devils circle...
How we will solve the situation. I am in contact with the server provider since saturday evening, and we found a solution how to enhance the performance. Basically we will switch to another, bigger machine. The amount of working memory will be 4 times higher than today, the cpu is faster, and even the hard discs are a bit faster than on this machine. If this sounds like delaying the problem until we hit the caps again...this is true. But this gives us the time to work on resource saving in the background.
Until we have the new server available you will have to deal with the situation. Therefore, and only therefore, we say sorry, and thank you very much for your patience.
Special thanks go to our Supporters, without you I wouldn't be able to spend more money for the hardware! Cheers, this makes me proud!
How you can help? Don't open too many windows at the same time ;)
Tags: News