Server patching is a common task in the IT world. Those of us responding to the WannaCry/Crypt ransomware attack have done a lot of patching over the past few days, drilling the process into our heads.

For the most part, the process is simple. However, small mistakes made during the patching process can spiral out of control into major disasters if made on the wrong servers at the wrong times, done in the wrong ways. To be the best of the best, you want to minimize the small mistakes. Success is often found at the margin, after all.

The following three tips, in no particular order, will minimize your exposure to making mistakes while patching your servers:

• Always communicate with server stakeholders.

When servers are being patched, there are two main impacts: 1) Server performance often slows down, and more importantly 2) the server will need to reboot after the patches are installed, forcibly ejecting all users from the workspace. If users aren’t aware of this ahead of time, things could get ugly as applications and works-in-progress are dumped by the computer in the middle of important business.

When patching a server, I recommend issuing three communications (preferably via e-mail): In advance of patching, immediately before patching, and after patching. The advance notice sent out anywhere from a few hours to a day before patching will help users make plans, the notice of imminent patching will let users know to finish up and save their work before reboot, and the completion notice gives them a timely heads-up that work can continue. Be the courteous IT guy and keep stakeholders in the loop.

• Verify your servers by server name before patching them.

This may not be applicable in every situation, but it’s a great general rule. Personal anecdote: I was once given a list of servers with both the names and IP addresses for patching. Naïve lad that I was, I went ahead and logged into each IP sequentially and began the patching process, rebooting the servers as I went.

Suddenly, I get a frantic phone call: the production servers are rebooting! Why? I quickly realized why: The list of servers I had been given had the wrong IP addresses. I was supposed to be patching test servers, but the IP addresses were for the production servers. Uh oh! Thankfully, the patches were rolled back without much trouble – but it was embarrassing.

If I’d taken the time verify the servers by their server name after logging in, I would’ve quickly noticed that the names were not matching the IP addresses. Take the time to verify your servers by name, and you’ll cut down on the odds of misfortune like mine. And remember that IP addresses can change, while server names do not – providing additional incentive to double-check your server names.

• Ensure that each server comes back up after reboot.

Most of the time, servers will come back online after the reboot with no trouble. But some issue might arise, maybe due to patching or maybe not, that will prevent the server from coming back online. If you just reboot and forget it, then the server might not come back up at all and you won’t know until it’s too late; people are already complaining about it and work is being impacted.

Once again, it’s a mistake that I made early in my career. A few servers weren’t jiving with the new updates for some reason and would not come back online – but I didn’t know, because I just rebooted ‘em and forget ‘em. A supervisor had to come by and ask me why those servers weren’t starting up, which made me feel like a fool when I had to admit I hadn’t double-checked that the patching process had completed successfully. Generally speaking, the patching process ain’t over until the server has come back online and is fully operational, so don’t just tune out mentally after that reboot has been initiated.

That’s it. As long as you keep these three tips in mind, then I guarantee that you will experience far less trouble in the future from your patching experiences. You know what they say about that ounce of prevention – it tastes great with scotch. Or that it’s worth more than a pound of cure… or something. I don’t know, I’m not Confucius. I’m just saying – nip the problems in the bud.

To recap: Communicate with your users before and after patching, verify that you are patching the correct servers by their name, and ensure the servers come back up after reboot. Happy patching!

Categories: Tips

Tags: Servers, Tips, Updates