Home > Cloud, Microsoft, Opalis, System Center > Automate patch & restart management in the #datacenter using #Microsoft Orchestrator and #wsus #sysctr #automation #mvpbuzz

Automate patch & restart management in the #datacenter using #Microsoft Orchestrator and #wsus #sysctr #automation #mvpbuzz


Introduction:

I have been working on a very interesting task next week for our cloud which is patch management automation.

One of the challenges we face as service provider or cloud provider if you are not a service provider is the patch management within our infrastructure and the cloud.

for years there have been tools and applications that can push updates from vendors to our servers; WSUS and SCCM are great examples of those, but there has been a missing part of the puzzle.

What about the restart management for those Servers/Application, how do we manage the relationship between servers patches, restart and restart order, let us take a deeper look to that.

Suppose that you have a typical infrastructure; this could be based on the cloud or not, This infrastructure consists of the following:

  • 2 Domain Controllers.
  • 1 SQL cluster; 2 Nodes.
  • 2 IIS Front-End Servers running a web application.
  • 2 TMG 2010 servers.

suppose that you use WSUS/SCCM, specified restart schedule and approved the updates, and waiting for servers restart, you have 2 options here:

  • if you had all the servers using single restart option; this means that all servers will reboot in the same time.
  • configure multiple scheduling based on OU/GPO, servers will restart based on schedules for different roles which is fine.

In the first option IIS servers will usually restart faster than SQL cluster; their web application might not start because SQL is not running, IIS serves might restart before the Domain Controllers, and might find the required credentials needed to start the web applications and same for SQL clusters that might reboot before DC and the SQL cluster fails, at the end of the day; who knows?!

the second option is cool, however you will have a larger maintenance window, you don’t know when servers will finish rebooting so you will have to wait and assign 30 minutes for DC reboot for example, then another 30 minutes then SQL servers reboot…etc, but this hurts your SLA and increases your maintenance window.

The Solution:

Somehow, you know your infrastructure requirements, so you know the restart order and priority for your servers, you need to have this relationship mapping first before anything else; as this will be the foundation.

You don’t need a fancy visio diagram or relationship table, all what you need is a simple table saying for example:

Server Name Restart Order
Server1 1
Server2 2

and this is an example,you can go as much complex as you want.

later you can use System Center Orchestrator to automate your patching and restart based on the relationship you defined, this is a very effective way to save your life and time, Orchestrator can interpret your restart order, force servers that needs restart to restart in the order you specified in the schedule you need or you can kick the hall process manually it doesn’t make a difference.

The How:

Disclaimer: use this article at your own risk, the solution described here is not the complete one, you need to do further testing, customization and modification to be enterprise ready, the scripted, files and workflows here are provided AS-IS without any warranty.

Building the blocks: In this section we explore the high-level architecture of the solution and its components and then we proceed with its implementation.

The requirements is very simple, we are using WSUS to deploy updates to servers, we have a restart order as the above table for example we want to restart our servers according to the above restart order.

The Lab Setup: I am running 1 Domain Controller that also hosts my WSUS server, 1 Orchestrator Server running SQL 2008 and Orchestrator, 4 Servers running Windows 2008 (srv1, srv2, srv3,srv4).

The restart order for servers is as following:

Server Name Restart Order
srv1 1
srv2 3
srv3 4
srv4 2

I mapped this restart order in a simple SQL Database configured as the following:

image

The Runbooks Architecture:

The Orchestrator has 3 RBs defined to achieve what we want:

    1. the first RB is the launcher, it queries the the database using the following simple query: (use test select hostname from restartordertbl order by restartorder), it queries the table and retrieve the server names and order them with their restart order.
    2. the RB then writes the servers with their restart priority to a text file, it will be used by a later RB to query server names from that text file (you can write you own script to step that in SQL or csv file, I used text file for simplicity).
    3. the RB sets counters of no. of rows returned, the the incremental counter used in looping and invokes the Core RB.image
    4. the Core RB is the core RB for this environment, it gets the counters, compare them if they are not equal it knows that it needs to loop and then proceeds with reading from the text file.
    5. you need to know that the link between the compare value action and append line action (the link with the purple color ) performs the actual decision it allows the RB to proceed only if the value is false which means the values are not equal and stops if the values are equal which means the loop is completed or there is no servers returned by the query.
    6. it executes the following powershell script to know if the server is pending reboot or not (

$baseKey = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey(“LocalMachine”, “\`d.T.~Ed/{A7DF762F-4857-4114-9AD9-AD7FE15F7148}.LineText\`d.T.~Ed/”)
$key = $baseKey.OpenSubKey(“Software\Microsoft\Windows\CurrentVersion\Component Based Servicing\”)
$subkeys = $key.GetSubKeyNames()
$key.Close()
$baseKey.Close()
If ($subkeys | Where {$_ -eq “RebootPending”})
{
throw “updates”
}
Else
{

})

the scripts queries the pending reboot status of the machine, if the machine is pending reboot then it will break throwing an error, if not it will complete correctly.

  1. The Link between the run powershell action and the restart action (in red color) allows the RB to take the restart path only of the powershell result is failed which is caused by the break event as the server in this case will be pending restart. if not it will take the other path (the green link) which means that server is not pending restart and starts the “Counter Increaser” RB.image
  2. the counter increaser RB is the simplest one, it simply increases the incremental counter and invokes the Core RB looping again.

Things to note:

  • in order to loop in Orchestrator you can’t loop within the RB, you need to use another RB for that this is why I have the Counter Increaser RB.
  • the powershell could restart the machine, but that didn’t work for me so I used the restart action.
  • you can check the link behaviour by selecting a link and click properties.
    Things that needs improvement:

This is a test RBs, we use different RBs in production that meets our specific environment, you will need to modify that above RPs to do:

  • Server checking if the server online or not.
  • the RBs does restart directly, you will need to include sleep time and restart check to make sure that server completed its restart before proceed with the other restart.
  • make the process parallel and maybe restart servers that are not related to others directly.
  • send notification to administrator or customer.
  • run post restart checks to make sure that server completed the reboto and services started successfully.
  • maybe integrate that with SCSM and go with approvals and workflows from there.

you can go epic with this foundation, be dynamic in servers query and database names this can go endless, use this RBs as your foundation and add more and more blocks to meet your infrastructure and customers’ goals, also feel free to comment or ask question I will be glad to do so.

attached below the working RBs they include every thing, make sure to check each step and read description thoroughly, you can download them from https://skydrive.live.com/embed?cid=6B566FD2C47B21C4&resid=6B566FD2C47B21C4%21130&authkey=AB25TJ854Zc4IT0

until later time and happy Eid

Mahmoud

  1. Virimai Mare
    October 27, 2012 at 8:18 am

    the download link is not working. We can i get a copy of the run books

  2. November 20, 2012 at 10:12 am

    Hi,
    some how mediafire deleted my files, all of them #fail, so I will find a suitable file service and upload them to you, hang on there.

  3. November 20, 2012 at 10:21 am

    I have uploaded the RB to Skydrive, please let me know if it doesn’t work : https://skydrive.live.com/embed?cid=6B566FD2C47B21C4&resid=6B566FD2C47B21C4%21130&authkey=AB25TJ854Zc4IT0

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: