Automated Patch Management

February 4, 2020

Tweet This:
Share on LinkedIn:

By Taylor Owen, Kovarus Automation Practice Manager

Patch management can quickly become a daunting task as the number of endpoints start to grow. An endpoint can be anything from a switch, VM, hypervisor, storage array, etc. Keeping systems up to date and free of known vulnerabilities is often a full-time job if not a full-time job for multiple people. Why is it a time-intensive activity?

There are many steps associated with patching a system. Assuming we’re patching a system that requires a reboot there are many dependencies that have to be accounted for. Take the example of a simple web server. At a high-level the associate tasks with patching a web server are:

  1. Update the centralized package repository
  2. Check to make sure the web server is healthy
  3. Remove the web server from a load balancing pool (if applicable)
  4. Verify the web server is not receiving traffic
  5. Patch and reboot the server
  6. Verify all applicable services are running and healthy
  7. Add the web server back to the load balancing pool (if applicable)
  8. Ensure that traffic is being routed to the web server

Assuming everything goes well, this would take on average 10–15 minutes. Multiply that by hundreds, if not thousands, of servers, and now you have a lot of person-hours needed for this task. The more systems that are needing to be patched, the greater likelihood that something will be missed or done incorrectly, and cause either security vulnerabilities, production downtime, inconsistencies in the environment, etc.

The good news there are many tools that can help with this process, but the bad news is that there are many tools to explore. Some tools are better at managing patches for Linux or just Windows. Some are great when needing to patch both or adding in the need for application patching. The best patching solution comes down to use case. What are you needing to patch? How often are you wanting to patch? What is the process of validating a patch operationally today? What is the process of identifying the patch needed? How many systems are needing to be patched? How distributed is the infrastructure?

There is not one tool that can solve all of these problems at once, and you want to use the right tool for the right job. There are a few different types of toolsets needed for an automated patch management solution.

Configuration Management

The first type of tool that needs to be selected is what is going to be used for configuration management. This tool will help with defining the long-term state of a system through code. It will typically operate in an asynchronous mode in relation to external, dependent systems. Some tools leveraged in this space would be Red Hat Ansible, Puppet, Chef, Salt, etc. Which one is best? Again, that comes down to use case. All of these tools have strengths and weaknesses.

Orchestration

An orchestration toolset solves the use case where the state of a system needs to be defined for a shorter amount of time and/or involves the management of external, dependent systems. As an example, one might need to stop an application service prior to shutting down a database service to ensure no data loss. Trying to leverage a configuration management tool to do this usually requires workaround or “hacky” solutions that are not easily maintained long-term or do not allow the technology to scale well. Some orchestration tools include Ansible (Red Hat), Bolt (Puppet), etc.

CI/CD Pipelines

While not 100% necessary, pipelines help with the scaling of the configuration management and orchestration platforms. Pipelines can help with the automated testing of Ansible playbooks/roles, Puppet manifest/modules, etc. Pipelines also help with piecing together more complex orchestration task sets. They’re also useful for the automation of promoting patches between environments such as development, test, production, etc. Pipeline tools like Jenkins come with many plugins to help integrate with a plethora of other tools that exist in IT environments today. There are many tools in this space such as Jenkins, CircleCI, TravisCI, etc. Every public cloud provider has some sort of pipeline toolset as well.

With these three types of toolsets in the environment 80% or more of patching tasks can be automated. When should each toolset be leveraged? We’ll go through some scenarios to answer that.

Isolated Updates

I’m defining isolated updates as patches or configurations that need to be updated but will not affect external systems. Some examples of this would be updating OpenSSH or the SSH configuration on a server. More than likely this can be done without affecting services on other systems. For these types of updates, configuration management tools are the best solution. All of the more well-known configuration management tools can do things such as install a package, configure the package, and start the service. Since these dependencies are contained within the system itself, it’s easy to define those dependencies in the configuration management tool.

Non-isolated Updates

Non-isolated updates include activities such as restarting a database server, patching a storage array, updating a web server in a load balancing pool, etc. These types of tasks do not lend themselves well to configuration management tools due to many external dependencies to the system. These types of tasks are done best with orchestration toolsets. The orchestration tasks can describe what tasks need to be done in sequential order between multiple systems. This ensures that dependencies are satisfied to help with the prevention of data loss, environment stability, and overall customer satisfaction. 

Summary

When looking to automate patch management there are many paths and not just one tool to solve all the patching problems. Often times these paths look great when starting, but end up in roadblocks, issues, or technical debt. This is probably not the case for a small number of systems, but when the infrastructure starts to grow, it can end up being painful. Leveraging the skillsets of people who have implemented these types of systems will pay in dividends long-term as they typically help with avoiding known pitfalls and typical process bottlenecks. They will start the project on the right path with best practices in mind to help with ensuring the toolsets mature well in the environment.


Looking to learn more about modernizing and automating IT? We created the Kovarus Proven Solutions Center (KPSC) to let you see what’s possible and learn how we can help you succeed. To learn more about the KPSC go to the KPSC page.

Also, follow Kovarus on LinkedIn for technology updates from our experts along with updates on Kovarus news and events.