Note: This script will work with PowerShell v1.0 and is slow.  If you would like a faster version and you are willing to upgrade your head-node to v2, check this post instead.

I’ve been working in creating a series of demos for a couple of HPC trainings we are delivering this month.  One of the things that I’ve noticed with the heat map, is that when restoring the nodes from a snapshot (they are virtual nodes) the heat map fail to display the info.  This is due to some services on the nodes not starting and failing to communicate with the head node.  The solution is to go through every node and restart this service.  Good luck doing that if you are running a 1,000 node cluster!

You might think “well, why not just use clusrun to stop and start the service?”  Thought of that and then realized that one of the services that need to be restarted is the one that is used for clusrun, so as soon as you shut it down, your clusrun session is over.

Once again, psexec and PowerShell save the day.  If you would like to restore heat map functionality from the head node to all nodes on your cluster, follow these steps:

1. Download and drop a copy of psexec on c:\Windows\System32 on your head node
2. Copy the PowerShell code below and save it in a file, say “C:\FixHeatMap.ps1″

   $HEAD_NODE = "HPC-HN"
   Write-Host "Setting Head Node To:" $HEAD_NODE
   Set-Content Env:CCP_SCHEDULER $HEAD_NODE
   
   $nodes = Get-HpcNode
   $services = "HpcManagement", "HpcNodeManager"
   
   foreach ($node in $nodes)  {
   foreach ($service in $services) {
   
   $serviceStart = "net stop $service"
   $serviceStop = "net start $service"
   $commandBytesStart = [System.Text.Encoding]::Unicode.GetBytes($serviceStart)
   $commandBytesStop = [System.Text.Encoding]::Unicode.GetBytes($serviceStop)
   $encodedStartCommand = [Convert]::ToBase64String($commandBytesStart)
   $encodedStopCommand = [Convert]::ToBase64String($commandBytesStop)
   $commandLine += $encodedCommand
   
   $computer = $node.NetBiosName
   $computer = "\\$computer"
   
   psexec $computer cmd /c "echo . | powershell -EncodedCommand $encodedStartCommand"
   psexec $computer cmd /c "echo . | powershell -EncodedCommand $encodedStopCommand"
   }
   }
   #Start the service that retrieves info from CN to heat-map
   net stop HpcSdm
   net start HpcSdm
   

3. Edit the text of the file so that $HEAD_NODE = “HPC-HN” matches the name of your head node
4. Open an HPC PowerShell command prompt and launch the script, here is the script in action with 4 nodes:

One of the things you will notice is that it can be slow as 3 services are started and stopped for every node via a remote call with psexec.  I know the process can be long but I don’t have time to optimize it.  If there are PowerShell gurus that would like to chip in, please be my guest!

Sources

• http://www.leeholmes.com/blog/UsingPowerShellAndPsExecToInvokeExpressionsOnRemoteComputers.aspx