Backups are Important, Recovery is Essential

On January 31, 2017 Gitlab experienced a massive database outage. They suffered hours of downtime, which they have detailed here. The transparency in that link is breathtaking for a couple reasons. First, any company willing to details their failures and challenges in such public terms gets my kudos. It’s not easy to put yourself on display, warts and all. Second, there are so many self inflicted wounds and unforced errors in the story. This line is particularly noteworthy:

So in other words, out of five backup/replication techniques deployed none are working reliably or set up in the first place.

I’ve worked in IT Operations for ten years and can recall many times in my past where I could have been Gitlab’s Team-member-1. When the pressure is on, it is EXTREMELY easy to make a fatal error. Even when problems arose that could have been a Resume Generating Event, I knew that I had a solid, tested recovery strategy to rely on, powered by Veeam.

Veeam has been the market leader in virtualization backups for over ten years. Many of the standard “must have” recovery features we expect in a backup solution were introduced or perfected by Veeam. They even include the ability to create a “lab” to validate recoverability of backups and to provide a method to test upgrades, updates, or application changes against actual copies of production machines. For all of the great features of Veeam’s virtual backup tools there has always been a gap in the solution. I have yet to work at an organization that had achieved 100% virtualization. Having a solution that rocks for virtual environments is amazing but it’s also frustrating to need to maintain a second system for bare metal machines.

At Tech Field Day 13, Veeam demonstrated an agent based approach to fill this gap. With Veeam Agent for Linux and Veeam Agent for Windows, you can get the same management and recovery options you’re accustomed to for VMs on physical machines. Not only is it designed to work on bare metal servers but workstations and cloud instances can also be protected. A struggle I experienced in my early days using public cloud was a lack of viable backup options. Most cloud workloads are ephemeral but there are some that need to be backed up so this would fit the bill.

Veeam has provided a free Endpoint Backup program for years. In fact, it is their most popular free product. There are some key feature differences between the different license levels of the agent based solution. Clint Wyckoff wrote a fantastic article here that can detail everything much better than I can. I will share their chart to list the differences.

There is one particularly interesting entries on this chart that is worth calling out, “Endpoint protection for mobile users”. This feature is a extremely powerful in an organization with a mobile workforce. If a user’s laptop is being protected with Veeam Agent for Windows but they are away during their scheduled backup time, the backup can still run and cache on their machine. When they return to a network connection that can resolve the Veeam Backup Repository, it will silently sync the backup. This is a great option to make sure your road warriors' data is still protected. Endpoint protection for mobile users pairs perfectly with the feature Backup directly to Veeam Cloud Connect, which allows the backup to go to a Veeam service partner or Cloud Connect archive. With the paid version, a backup could run on the plane and sync up when you arrive at a strong wifi connection.

I’ve chosen to feature just one of the new products Veeam discussed during the Tech Field Day 13 presentation. There are many more management and recovery tools displayed in the session itself. You can watch the full video on the Tech Field Day site if you’re interested in seeing everything they showed. As of this article the Veeam Agent for Windows product is in beta and can be downloaded at their site. I am currently running it and plan to write up those experiences as well for a future article. I’ve explored many of the recovery options and I am extremely impressed.