1. Dezember. 2012 | von Christopher Rudolf

On Monday, November 26, 2012, belbo.com was briefly affected by severe server outages at one of our data centers "Server4You". Here are the answers to the most important questions:

What happened?
Due to a cooling failure in the "Server4You" data center, the entire data center was shut down for 12 hours. Due to our backup strategy, we were able to react quickly and brought the system back online at our emergency data center within a few minutes. After checking the data, which was completed successfully, we released the complete system again. At 10 o'clock, all customers were able to work without any problems.
Due to our backup strategy, we had the ability to react quickly and continue operating the system at a secondary data center, however this only happened after checking the appointment data, so the system was unavailable for 120 minutes.

If an entire data center failed, why was zeitfest able to resume so quickly?
zeitfest is basically operated in multiple data centers simultaneously. The switch from the primary data center to a secondary data center was accomplished technologically by changing a DNS entry. A DNS entry provides information about which data center a website is operated on. Normally, this entry is only updated after several hours. However, since we operate our DNS entries on Amazon's "Route 53" high-availability service, this change is completed in just 60 seconds.

What did you learn?
We are very dissatisfied with the duration of the 2-hour outage. Therefore, we will implement an automatic DNS switch that automatically switches to the permanently running emergency system if the primary data center is no longer accessible.
In the end, this means: We are better prepared for a data center failure and your appointments and calendars will remain accessible and viewable next time.

We apologize for the unavailability and the resulting inconvenience to your business.