... in action. Amsterdam was great. The city and the convention were really a blast.
I was impressed by the fact how much love, time and effort our players are investing into their hobby. The room has been decorated with so much love, so many presents have been prepared in Tibia style, and most important of all, everybody was talking about the game with so much passion. This really gives you a kick and shows you who you are doing all the work for. A convention is a big boost for your motivation to make the game better and to always have your community in mind, whatever you do. Thanks for the invitation girls.
Tibiacity published some pictures. Check them out, if you're interested.
By the way: I was really lucky with the flights. Most of the flights to and from Amsterdam were canceled, except mine. I am such a lucky bastard.
Monday, August 11, 2008
Thursday, August 7, 2008
Lufthansa ...
... is on strike. Brilliant. Don't they know that I am flying to Amsterdam on Friday and hopefully back to Munich on Sunday. I bet they know and just don't care. From what I found out, the flight just before mine got canceled. Same applies to the Sunday flight. This means, I will have a lot of fun at the check-in counter as a complete airbus-amount of passengers will still be grounded at the airport. Hurray. I am sure they are all very happy, don't care at all about the delay and would never give me that "You idiot, you got your flight huh? Show-off!"-look. And at the same time, Lufthansa is telling me that I am not allowed to carry any self defense weapons with me. Thanks.
Friday, August 1, 2008
I wonder ...
... how long it will take, until you find out. :-)
Edit: 8 hours till now, and still counting.
Edit: 11:23 pm ! Finally. Congratz Matt!
Edit: Well, I am actually surprised how long it took. I thought, our website gets scanned on a regular basis. Obviously not. For all of you, who still don't know, what I am talking about, check this out. Fiction Fighters. That's our new game.
Edit: 8 hours till now, and still counting.
Edit: 11:23 pm ! Finally. Congratz Matt!
Edit: Well, I am actually surprised how long it took. I thought, our website gets scanned on a regular basis. Obviously not. For all of you, who still don't know, what I am talking about, check this out. Fiction Fighters. That's our new game.
Saturday, July 19, 2008
Amsterdam ...
... shall it be. I've just booked my flight to visit the Tibia convention in Amsterdam in 3 weeks. This will actually be the first convention for me and I am really excited to see so many of our Tibia players in RL. I already have many plans on what to take with me as presents and goodies. Gosh, I am really looking forward to the trip. In addition, I will extend my stay for the whole weekend so I get to see the lovely city of Amserdam once again. Last time I've been there must have been 25 years ago, when I travelled through Europe with my parents. I still don't have an hotel yet, but I found a house boat you can book. I definitely want that.
Edit: I talked to my mom. It was actually 29 years ago, that I've been to Amsterdam. Gosh, I am old.
Edit: I talked to my mom. It was actually 29 years ago, that I've been to Amsterdam. Gosh, I am old.
Saturday, May 31, 2008
Making money ...
... is legitimate. I always get confused, when we are accused of making money. Why are people accusing us of being successful? Everybody wants to be successful in what he is doing. When you're in school, you wanna get good grades, when you meet a girl, you like to hear "You are a lovely guy". Well, when you do business, one of the key indicators is "sales". You're having a crappy product, you don't sell it. You listen to your customers and design the product according to their needs, you sell more. Is this kind of behaviour THAT condemnable?
No, we are not a students project anymore. We are a company. This basically means two things:
1. We are professionals
2. We have to earn money
The good comes with the bad. The bad part about it is that we have to think about success, and not only fun anymore. I have to admit that we evaluate projects.
The good part is that we can offer customer support, regular updates and not to forget a certain probability of not having to shut down your favourite game next week due to a shortage of funds.
So, to conclude my Saturday night rant: Whoever posts comments like "You only revamp Rookgard to make it more appealing to new customers and to make more money". Yes. You are right. Don't state the obvious. My advice to you is: Take marketing lessons. You obviously have a talent for it.
No, we are not a students project anymore. We are a company. This basically means two things:
1. We are professionals
2. We have to earn money
The good comes with the bad. The bad part about it is that we have to think about success, and not only fun anymore. I have to admit that we evaluate projects.
The good part is that we can offer customer support, regular updates and not to forget a certain probability of not having to shut down your favourite game next week due to a shortage of funds.
So, to conclude my Saturday night rant: Whoever posts comments like "You only revamp Rookgard to make it more appealing to new customers and to make more money". Yes. You are right. Don't state the obvious. My advice to you is: Take marketing lessons. You obviously have a talent for it.
Friday, April 18, 2008
DDoS - Interview
I know, I haven't postet for a while. Sorry for that. Anyway, I thought I piost an interview, I gave to some of our fansites, concerning DDoS attacks. Maybe you'll find it interesting:
[Mercutio]: First of all, thanks for the invitation and for giving us the opportunity to talk about the recent incidents. I hope this interview helps to clarify some aspects and to answer some of the most interesting questions. Now, let's get started ....
[TibiaCity]: What are DDoS attacks?
[Mercutio]: Technically, DDoS stands for Distributed Denial of Service. Although that sounds a bit rough, it already mentions the key aspects of these kinds of attacks. Many attackers try to force your system into not performing its service. Right, now that was really rough.
In general it works like this: The attacker distributes a small program via the internet to many other computers. This is possible as many people do not really care about or do not really know much about protecting their own PCs from malicious software. After a few weeks this program might have infected hundreds or even thousands of remote computers. So, what does this program do now? Nothing, it just sits on its' host-machine waiting for orders. In the meantime, the attacker gets regular updates on the amount of PCs already infected with his little program. As soon as he thinks the amount of infected PCs is big enough to be successful, he starts his attack. How? He commands all his distributed programs to repeatedly send some data to a specific target at a specific moment.
On our side, we have to examine every incoming data package in order to find out if this is a valid command from one of our players or not. So, the information the attacker sends might be complete nonsense. Unfortunately we have to find out on our side if it really is nonsense or not, as we don’t forward nonsense to our game servers.
The more nonsense the attacker sends, the longer it takes to separate it from valid information. That’s what the normal player experiences in form of lags and kicks. The game servers are running smooth and easy all the time.
That was the technical explanation. For all of you who stopped after the third line, I’ll try to explain it with a real life example: Imagine a big concert which takes place at a football stadium. It is obvious that only ticket holders get access to the concert and are asked to come inside. Now usually people who don’t have a ticket don’t show up at the gates, so the bouncer doesn’t have too much to do.
Unfortunately, the band is so famous that a lot of fake tickets have been sold on the black market. Now the bouncer has to examine every single ticket very carefully. This takes time and delays the entry process even for the fans with valid tickets. Now imagine there are 10,000 times more fake tickets around than valid ones. Now that’s what I call a lag. In the meantime, the show inside the stadium goes on normally.
[TibiaBR]: Why is it so hard to solve this problem?
[Mercutio]: The most important reason is that not everything lies in our hands. Please don’t get me wrong. This isn’t meant to be an excuse and I completely understand that our players don’t care whose responsibility it is – they just want to have a trouble free gaming experience. But sadly enough, that’s the truth. We collaborate with many partners such as service providers, data centres, hardware manufacturers, etc. Some of the issues can only be solved or at least addressed by them.
If I may use the concert example again: We might have control over the bouncers and decide to hire better or even more of them, but we have little to no influence on the amount of entrance gates the stadium has. So it makes no sense to have 1000 bouncers at a single gate. And even if we could convince the owners of the stadium to install more gates, the problem just shifts to an earlier stage: the limited amount of highways leading to the stadium. Can we convince the town to build more of them? Hardly. I think you get my point.
[TibiaHispano]: Are the DDoS attacks a way to get information about the players?
[Mercutio]: Nope. As I’ve explained, DDoS attacks only have the purpose to jam the entry, not the server itself. I am using the concert example again: People with fake tickets don’t get in and therefore don’t get to listen to the music.
[TibiaMx]: In which way do the DDoS attacks interfere with the game server or the website?
[Mercutio]: Once again, usually they don’t have a technical influence on the servers. It’s the access that gets jammed.
[TibiaHispano]: Why is it that DDoS attacks have a bigger effect on companies such as CipSoft in comparison to e.g. Google?
[TibiaNews]: Yes, could you tell us a bit about the technical side of Tibia in general?
[Craban]: To answer this question, I have to explain the main differences of a service like Tibia and e.g. Google. The biggest difference is the server infrastructure. If you log in on a a game world with your character, and your friend also logs in on the same game world, you are actually both on the same server. This is necessary as all actions you perform with your character (moving, talking, fighting, etc) have an immediate and direct influence on not only the character of your friend, but on the game world as a whole. So, we need to control and monitor all of those actions on one server. This does not apply at all to the service Google is offering. If I search for “Tibia” in Google I might most likely end up on a server in Germany. If you look up the expression “Tibia” you might end up somewhere in the U.S., but, even if all Google servers in Europe were down, I can still be redirected to a sever in the U.S. and still get the same result – without even knowing that all European servers are down.
I'll use Mercutio's concert example: I can call 100 friends of mine all over the world to ask them where and when the concert will take place, but to actually listen to it, I surely have to be at a specific place at a given point in time.
The second big difference is the way our game clients communicate with the servers. We are establishing a persistent and synchronised communication between the clients and the server. This means it is absolutely essential that all actions on one game server happen at the same time in a synchronised manner.
Within an online game, interaction is a key feature. Interaction means that whatever I do has an immediate effect on everybody else. Let’s say I am attacking you, then you have to be informed about my attack immediately. During a DDoS attack we don’t get any data from the players, as the connections are jammed. Therefore we do not know what you are all doing at this moment. As the server is still running, your character is still online as well, we just receive no commands regarding what to do with this character. To avoid any unfair advantages of one character over the other, we log-out characters automatically after the player’s client hasn’t answered for a given period of time.
A Google search does not depend on this restriction. When I search for “Tibia”, Google might look this up on 12 different servers on 3 different continents. Whenever Google thinks it has finished searching, it will publish the results. The worst thing that can happen to me is that I get the results a little later than usual. However, “later” is not an option in Tibia.
[TibiaNews]: Are there any servers that aren't being affected by the attacks?
[Mercutio]: Nope. All servers are under attack all the time. It’s just that most of the attacks are so weak that you and sometimes even we don’t notice them.
[TibiaBR]: Seems that the attacks affect mostly US servers... what German servers got different from US ones in terms of security measures?
[Mercutio]: We installed a device called “ocean”. It is a large liquid basin, which acts as a natural firewall. :-) No, seriously. In general, both data centres are alike, however we do have more possibilities to change the configuration of the hardware in Germany - and their staff is reacting faster as well. In addition, German servers are just not getting attacked as much.
[TibiaMX]: Who is responsible for the attacks? Is it possible to track their source?
[Craban]: Yes, it's not easy but it is possible. It requires a coordinated investigation of a few different parties, but up till now, we have been quite successful with these investigations.
[TibiaNews]: Are they Tibia players?
[Mercutio]: Yes. Their main characters are called …..
[Craban]: Stop that! We agreed not to start a manhunt. (And even if we did, I want to be first!)
[TibiaNews]: Can you take proper legal action?
[Craban]: Yes, and we did and will go on doing that. Unfortunately, this is a really time consuming process. We have to consider many national and international laws, which can be completely different to German law. In addition, internet law still isn’t coordinated worldwide, so what might be illegal in one country isn’t in another. However, we are already working together with international partners and are determined to bring the attackers to court. After all they have caused great disappointment to our players.
[TibiaCity]: Why is Tibia under DDoS attacks?
[Mercutio]: Most of the attacks concentrate on a few servers, so we think we are dealing with a personal vendetta, which is used to take revenge over in-game issues. And in some cases I suspect it is just plain boredom. But the attackers shouldn’t forget that what they are doing is highly criminal. Starting a war in game is one thing, attacking servers in reality is something completely different.
[TibiaBR]: What's the size in gigabytes of the attacks your servers are currently suffering?
[Mercutio]: If I answered this question, I could just as well give you a detailed plan on how to bring our service down. Please understand that we cannot share this kind of technical information. So I am really sorry, but I have to use the red “Confidential” stamp on this file.
[TibiaNews]: How much data is coming and going through Tibia on an average day?
[Mercutio]: -prepares the red stamp again but takes a look at Durin first-
[Durin]: All game worlds and the website produce about 1600 GB per day.
[TibiaNews]: At what point in time did Tibia start receiving DDoS attacks?
[Durin]: Right from the start, but it got really nasty during 2007 with a boost in October.
[TibiaNews]: Is there a possibility that they may attack the official fansites as well?
[Mercutio]: Sure, and they already have. -points to [TibiaBR]-
[TibiaBR]: Sometime ago CipSoft mentioned that they were testing some new hardware for the servers, in order to resist better to the attacks. How are these tests going?
[Mercutio]: Over the last weeks, we have thoroughly tested the best machines from all leaders in the industry. It was quite a surprise for us as well, that even the world’s leading manufacturers had problems coming up with hardware that meets our demands. So we finally ended up with the biggest machine we could find out there. Our technicians say it is the “BFG” of internet security. :-)
[Durin]: We plan to have it “mounted and loaded” in mid May.
[TibiaNews]: What are some of your methods of combating the DDoS attacks?
[Mercutio]: Before you can solve a problem, you have to understand it. So the first thing we did was examine the nature, character and origin of the attacks. I have to admit that this phase took us too long. We tried to find the ultimate solution, rather than implementing the first countermeasures right away. After that, we tried out a few things that looked good in books, but turned out to be useless. In the end, we had to learn the hard way. So we’ve really burnt some time back then. Unfortunately our players had to bear the consequences for that.
After this, we decided to go for a double strategy: Upgrades to our hardware and redesign of our technical infrastructure. I’ve already talked about the new hardware. The new infrastructure enables us to better configure all hardware components involved and therefore be better protected against attacks.
In order to stay with the example I could say: We equipped the bouncers with super fast ticket verification devices and placed them at strategic points.
[TibiaBR]: Has CipSoft ever considered the possibility of moving dedicated servers to a better provider?
[Mercutio]: Yes, and we already did. We found a new data centre in the U.S. which is capable of implementing all the security measures I’ve mentioned above. 50% of our American servers have already been moved to the new data centre, the rest will follow sooner or later.
[TibiaMX:] Can you already tell, that the move to a new data centre brought better protection concerning attacks?
[Mercutio]: Yes, we already can see the difference.
[TibiaNews]: Will this have influence on the creation of new servers in the future? (If so, in what way?)
[Craban]: Ahm, well, yes and no. Yes, every new server we may put up in the future will of course be configured and protected with all measures we have at that point of time. No, neither attacks nor countermeasures will influence the decision on whether we put up new servers or non in any way.
[TibiaNews]: We realize that once a problem has been fixed, someone will always try to find a way to break past that solution. Do you ever think that Tibia will be totally DDoS free?
[Mercutio]: Plain and simple: No. We will always be under attack. The real question is: Will the players notice that we are under attack? I think we've gathered a lot of know-how over the past weeks regarding this issue, so we are pretty damn good in defence right now. Still, it’s a constant race….
[TibiaNews]: How long until this current strand of attacks will last?
[Mercutio]: Well, I haven’t spoken to the attackers lately, so you should ask them, not me.
[TibiaNews]: Do you roughly know how long it will take for the DDoS attacks to be fixed?
[Mercutio]: We are very optimistic that the measures we have taken will dramatically reduce the effects of the DDoS attacks on the availability of our service. I’ve explained what we did and when we did it or plan to do it. I also want to stress that we’ve put great effort in not only solving the past problems but also in being better prepared in the future. I am not saying “mission accomplished” but all in all, we’re out of the woods by now.
[TibiaBR]: Brazilian players (and we believe other players too) are very angry because of servers' instability, blaming CipSoft for it. What can you say to them?
[Mercutio]: Although we cannot be held responsible for the attacks themselves, I can understand that many of our players are disappointed by the way we handled the situation. We have to admit that we made some mistakes and spent too much time on analysing and trying out different solutions. Overall it took us too long to implement effective countermeasures.
Besides, we have been criticised for our information policy, more precisely, the lack of information we were giving out to the community. Of course we understand that our players want to get a regular update on the situation, above all what they want to know is if it is safe to go hunting again. On the other hand, every piece of information we share might be useful to the attackers as well. We always had to weigh those two aspects against each other. Unfortunately, the “confidential” side won many times.
To conclude: Yes, we are reading our own boards and we have seen all those posts filled with disappointment, frustration and even anger. Yes, we have a pretty good view on the community’s mood. Yes, we can understand them. So, what can I give as reply?
I would like to apologize to our community. We have been caught quite unprepared and failed to react adequately. Ever since the first strike, we were trying hard to bring all systems back to normal and to ensure a trouble free gaming experience. This didn’t work out as planned all the time, but we actually left no stone unturned to come to a satisfying solution. We finally found it. We are really sorry for the inconvenience this has caused and thank you all for your patience and the constant trust you have put in us.
[Fansites]: Any last words?
[Mercutio]: Once again, thanks for having us here and giving us the opportunity to talk to all of you. Keep up the good work with your fansites.
[Mercutio]: First of all, thanks for the invitation and for giving us the opportunity to talk about the recent incidents. I hope this interview helps to clarify some aspects and to answer some of the most interesting questions. Now, let's get started ....
[TibiaCity]: What are DDoS attacks?
[Mercutio]: Technically, DDoS stands for Distributed Denial of Service. Although that sounds a bit rough, it already mentions the key aspects of these kinds of attacks. Many attackers try to force your system into not performing its service. Right, now that was really rough.
In general it works like this: The attacker distributes a small program via the internet to many other computers. This is possible as many people do not really care about or do not really know much about protecting their own PCs from malicious software. After a few weeks this program might have infected hundreds or even thousands of remote computers. So, what does this program do now? Nothing, it just sits on its' host-machine waiting for orders. In the meantime, the attacker gets regular updates on the amount of PCs already infected with his little program. As soon as he thinks the amount of infected PCs is big enough to be successful, he starts his attack. How? He commands all his distributed programs to repeatedly send some data to a specific target at a specific moment.
On our side, we have to examine every incoming data package in order to find out if this is a valid command from one of our players or not. So, the information the attacker sends might be complete nonsense. Unfortunately we have to find out on our side if it really is nonsense or not, as we don’t forward nonsense to our game servers.
The more nonsense the attacker sends, the longer it takes to separate it from valid information. That’s what the normal player experiences in form of lags and kicks. The game servers are running smooth and easy all the time.
That was the technical explanation. For all of you who stopped after the third line, I’ll try to explain it with a real life example: Imagine a big concert which takes place at a football stadium. It is obvious that only ticket holders get access to the concert and are asked to come inside. Now usually people who don’t have a ticket don’t show up at the gates, so the bouncer doesn’t have too much to do.
Unfortunately, the band is so famous that a lot of fake tickets have been sold on the black market. Now the bouncer has to examine every single ticket very carefully. This takes time and delays the entry process even for the fans with valid tickets. Now imagine there are 10,000 times more fake tickets around than valid ones. Now that’s what I call a lag. In the meantime, the show inside the stadium goes on normally.
[TibiaBR]: Why is it so hard to solve this problem?
[Mercutio]: The most important reason is that not everything lies in our hands. Please don’t get me wrong. This isn’t meant to be an excuse and I completely understand that our players don’t care whose responsibility it is – they just want to have a trouble free gaming experience. But sadly enough, that’s the truth. We collaborate with many partners such as service providers, data centres, hardware manufacturers, etc. Some of the issues can only be solved or at least addressed by them.
If I may use the concert example again: We might have control over the bouncers and decide to hire better or even more of them, but we have little to no influence on the amount of entrance gates the stadium has. So it makes no sense to have 1000 bouncers at a single gate. And even if we could convince the owners of the stadium to install more gates, the problem just shifts to an earlier stage: the limited amount of highways leading to the stadium. Can we convince the town to build more of them? Hardly. I think you get my point.
[TibiaHispano]: Are the DDoS attacks a way to get information about the players?
[Mercutio]: Nope. As I’ve explained, DDoS attacks only have the purpose to jam the entry, not the server itself. I am using the concert example again: People with fake tickets don’t get in and therefore don’t get to listen to the music.
[TibiaMx]: In which way do the DDoS attacks interfere with the game server or the website?
[Mercutio]: Once again, usually they don’t have a technical influence on the servers. It’s the access that gets jammed.
[TibiaHispano]: Why is it that DDoS attacks have a bigger effect on companies such as CipSoft in comparison to e.g. Google?
[TibiaNews]: Yes, could you tell us a bit about the technical side of Tibia in general?
[Craban]: To answer this question, I have to explain the main differences of a service like Tibia and e.g. Google. The biggest difference is the server infrastructure. If you log in on a a game world with your character, and your friend also logs in on the same game world, you are actually both on the same server. This is necessary as all actions you perform with your character (moving, talking, fighting, etc) have an immediate and direct influence on not only the character of your friend, but on the game world as a whole. So, we need to control and monitor all of those actions on one server. This does not apply at all to the service Google is offering. If I search for “Tibia” in Google I might most likely end up on a server in Germany. If you look up the expression “Tibia” you might end up somewhere in the U.S., but, even if all Google servers in Europe were down, I can still be redirected to a sever in the U.S. and still get the same result – without even knowing that all European servers are down.
I'll use Mercutio's concert example: I can call 100 friends of mine all over the world to ask them where and when the concert will take place, but to actually listen to it, I surely have to be at a specific place at a given point in time.
The second big difference is the way our game clients communicate with the servers. We are establishing a persistent and synchronised communication between the clients and the server. This means it is absolutely essential that all actions on one game server happen at the same time in a synchronised manner.
Within an online game, interaction is a key feature. Interaction means that whatever I do has an immediate effect on everybody else. Let’s say I am attacking you, then you have to be informed about my attack immediately. During a DDoS attack we don’t get any data from the players, as the connections are jammed. Therefore we do not know what you are all doing at this moment. As the server is still running, your character is still online as well, we just receive no commands regarding what to do with this character. To avoid any unfair advantages of one character over the other, we log-out characters automatically after the player’s client hasn’t answered for a given period of time.
A Google search does not depend on this restriction. When I search for “Tibia”, Google might look this up on 12 different servers on 3 different continents. Whenever Google thinks it has finished searching, it will publish the results. The worst thing that can happen to me is that I get the results a little later than usual. However, “later” is not an option in Tibia.
[TibiaNews]: Are there any servers that aren't being affected by the attacks?
[Mercutio]: Nope. All servers are under attack all the time. It’s just that most of the attacks are so weak that you and sometimes even we don’t notice them.
[TibiaBR]: Seems that the attacks affect mostly US servers... what German servers got different from US ones in terms of security measures?
[Mercutio]: We installed a device called “ocean”. It is a large liquid basin, which acts as a natural firewall. :-) No, seriously. In general, both data centres are alike, however we do have more possibilities to change the configuration of the hardware in Germany - and their staff is reacting faster as well. In addition, German servers are just not getting attacked as much.
[TibiaMX]: Who is responsible for the attacks? Is it possible to track their source?
[Craban]: Yes, it's not easy but it is possible. It requires a coordinated investigation of a few different parties, but up till now, we have been quite successful with these investigations.
[TibiaNews]: Are they Tibia players?
[Mercutio]: Yes. Their main characters are called …..
[Craban]: Stop that! We agreed not to start a manhunt. (And even if we did, I want to be first!)
[TibiaNews]: Can you take proper legal action?
[Craban]: Yes, and we did and will go on doing that. Unfortunately, this is a really time consuming process. We have to consider many national and international laws, which can be completely different to German law. In addition, internet law still isn’t coordinated worldwide, so what might be illegal in one country isn’t in another. However, we are already working together with international partners and are determined to bring the attackers to court. After all they have caused great disappointment to our players.
[TibiaCity]: Why is Tibia under DDoS attacks?
[Mercutio]: Most of the attacks concentrate on a few servers, so we think we are dealing with a personal vendetta, which is used to take revenge over in-game issues. And in some cases I suspect it is just plain boredom. But the attackers shouldn’t forget that what they are doing is highly criminal. Starting a war in game is one thing, attacking servers in reality is something completely different.
[TibiaBR]: What's the size in gigabytes of the attacks your servers are currently suffering?
[Mercutio]: If I answered this question, I could just as well give you a detailed plan on how to bring our service down. Please understand that we cannot share this kind of technical information. So I am really sorry, but I have to use the red “Confidential” stamp on this file.
[TibiaNews]: How much data is coming and going through Tibia on an average day?
[Mercutio]: -prepares the red stamp again but takes a look at Durin first-
[Durin]: All game worlds and the website produce about 1600 GB per day.
[TibiaNews]: At what point in time did Tibia start receiving DDoS attacks?
[Durin]: Right from the start, but it got really nasty during 2007 with a boost in October.
[TibiaNews]: Is there a possibility that they may attack the official fansites as well?
[Mercutio]: Sure, and they already have. -points to [TibiaBR]-
[TibiaBR]: Sometime ago CipSoft mentioned that they were testing some new hardware for the servers, in order to resist better to the attacks. How are these tests going?
[Mercutio]: Over the last weeks, we have thoroughly tested the best machines from all leaders in the industry. It was quite a surprise for us as well, that even the world’s leading manufacturers had problems coming up with hardware that meets our demands. So we finally ended up with the biggest machine we could find out there. Our technicians say it is the “BFG” of internet security. :-)
[Durin]: We plan to have it “mounted and loaded” in mid May.
[TibiaNews]: What are some of your methods of combating the DDoS attacks?
[Mercutio]: Before you can solve a problem, you have to understand it. So the first thing we did was examine the nature, character and origin of the attacks. I have to admit that this phase took us too long. We tried to find the ultimate solution, rather than implementing the first countermeasures right away. After that, we tried out a few things that looked good in books, but turned out to be useless. In the end, we had to learn the hard way. So we’ve really burnt some time back then. Unfortunately our players had to bear the consequences for that.
After this, we decided to go for a double strategy: Upgrades to our hardware and redesign of our technical infrastructure. I’ve already talked about the new hardware. The new infrastructure enables us to better configure all hardware components involved and therefore be better protected against attacks.
In order to stay with the example I could say: We equipped the bouncers with super fast ticket verification devices and placed them at strategic points.
[TibiaBR]: Has CipSoft ever considered the possibility of moving dedicated servers to a better provider?
[Mercutio]: Yes, and we already did. We found a new data centre in the U.S. which is capable of implementing all the security measures I’ve mentioned above. 50% of our American servers have already been moved to the new data centre, the rest will follow sooner or later.
[TibiaMX:] Can you already tell, that the move to a new data centre brought better protection concerning attacks?
[Mercutio]: Yes, we already can see the difference.
[TibiaNews]: Will this have influence on the creation of new servers in the future? (If so, in what way?)
[Craban]: Ahm, well, yes and no. Yes, every new server we may put up in the future will of course be configured and protected with all measures we have at that point of time. No, neither attacks nor countermeasures will influence the decision on whether we put up new servers or non in any way.
[TibiaNews]: We realize that once a problem has been fixed, someone will always try to find a way to break past that solution. Do you ever think that Tibia will be totally DDoS free?
[Mercutio]: Plain and simple: No. We will always be under attack. The real question is: Will the players notice that we are under attack? I think we've gathered a lot of know-how over the past weeks regarding this issue, so we are pretty damn good in defence right now. Still, it’s a constant race….
[TibiaNews]: How long until this current strand of attacks will last?
[Mercutio]: Well, I haven’t spoken to the attackers lately, so you should ask them, not me.
[TibiaNews]: Do you roughly know how long it will take for the DDoS attacks to be fixed?
[Mercutio]: We are very optimistic that the measures we have taken will dramatically reduce the effects of the DDoS attacks on the availability of our service. I’ve explained what we did and when we did it or plan to do it. I also want to stress that we’ve put great effort in not only solving the past problems but also in being better prepared in the future. I am not saying “mission accomplished” but all in all, we’re out of the woods by now.
[TibiaBR]: Brazilian players (and we believe other players too) are very angry because of servers' instability, blaming CipSoft for it. What can you say to them?
[Mercutio]: Although we cannot be held responsible for the attacks themselves, I can understand that many of our players are disappointed by the way we handled the situation. We have to admit that we made some mistakes and spent too much time on analysing and trying out different solutions. Overall it took us too long to implement effective countermeasures.
Besides, we have been criticised for our information policy, more precisely, the lack of information we were giving out to the community. Of course we understand that our players want to get a regular update on the situation, above all what they want to know is if it is safe to go hunting again. On the other hand, every piece of information we share might be useful to the attackers as well. We always had to weigh those two aspects against each other. Unfortunately, the “confidential” side won many times.
To conclude: Yes, we are reading our own boards and we have seen all those posts filled with disappointment, frustration and even anger. Yes, we have a pretty good view on the community’s mood. Yes, we can understand them. So, what can I give as reply?
I would like to apologize to our community. We have been caught quite unprepared and failed to react adequately. Ever since the first strike, we were trying hard to bring all systems back to normal and to ensure a trouble free gaming experience. This didn’t work out as planned all the time, but we actually left no stone unturned to come to a satisfying solution. We finally found it. We are really sorry for the inconvenience this has caused and thank you all for your patience and the constant trust you have put in us.
[Fansites]: Any last words?
[Mercutio]: Once again, thanks for having us here and giving us the opportunity to talk to all of you. Keep up the good work with your fansites.
Sunday, March 16, 2008
Mercutio meets Mozart
I am off. I'll be on vacation for the next two weeks. Going to Austria to meet a friend and to celebrate his birthday. After that, we'll get in the car and drive to wherever the road takes us. I'll be checking the blog once in a while and maybe keep you updated on the "bottle count". Who knows, I might even upload a picture of my "Mercutio meets Mozart" tour. Take care and happy easter.
Update: Back in Regensburg. The picture has been taken in Bratislava. That's in Slovakia - and yes, I grew a beard.
Subscribe to:
Posts (Atom)