True Multithreading in PowerShell

Jun 242014

As with all things, the longer you play with something the more you learn about it. It has been nearly 5 years since I wrote my original article on multithreading where I used PowerShell jobs to run multiple items at a time. In the conversations following that post I had a reader submit a fairly nasty note saying that multithreading with jobs was in fact not real multithreading. He also selected some choice words for me making sure I couldn’t post his comments. It hurt my very fragile feelings and, as a result, I started looking at how I could really, truly multithread with PowerShell. The result of that effort is this script. This is a true multithreading script no “ifs”, “ands” or “buts”.

It runs MUCH faster than my other rendition and includes a much more advanced feature set from my other script. The down side is that the script is very difficult to understand if you are new to PowerShell. If you would like to have a script that is more visible and easier to understand, please refer to the version that uses Jobs.

Okay, so the big addition for this script is the ability to either run a script or a cmdlet that’s built in. As well, you can run this within the pipeline! To do that I had to include the begin, process and end blocks. It makes the script a bit more complex, but really pays off when you pipe your custom script into Out-Gridview or pipe your advanced filtering script into a multithreaded one! I have even pulled my SCCM collections via Get-WmiObject and piped them into a multithreaded script! Cool stuff!

Okay, so on to the breakdown.

First we need to get all of our parameters. If this doesn’t make since, please find my post on parameters!

Param($Command = $(Read-Host "Enter the script file"), 
    [Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]$ObjectList,
    $InputParam = $Null,
    $MaxThreads = 20,
    $SleepTimer = 200,
    $MaxResultTime = 120,
    [HashTable]$AddParam = @{},
    [Array]$AddSwitch = @()
)

Param($Command = $(Read-Host "Enter the script file"),

[Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]$ObjectList,

$InputParam = $Null,

$MaxThreads = 20,

$SleepTimer = 200,

$MaxResultTime = 120,

[HashTable]$AddParam = @{},

[Array]$AddSwitch = @()

)

All of these are defined as follows:
Command
This is where you provide the powershell Commandlet / Script file that you want to multithread. You can also choose a built in cmdlet. Keep in mind that your script. This script is read into a scriptblock, so any unforeseen errors are likely caused by the conversion to a script block.

ObjectList
The objectlist represents the arguments that are provided to the child script. This is an open ended argument and can take a single object from the pipeline, an array, a collection, or a file name. The multithreading script does it’s best to find out which you have provided and handle it as such. If you would like to provide a file, then the file is read with one object on each line and will be provided as is to the script you are running as a string. If this is not desired, then use an array.

InputParam
This allows you to specify the parameter for which your input objects are to be evaluated. As an example, if you were to provide a computer name to the Get-Process cmdlet as just an argument, it would attempt to find all processes where the name was the provided computer name and fail. You need to specify that the parameter that you are providing is the “ComputerName”.

AddParam
This allows you to specify additional parameters to the running command. For instance, if you are trying to find the status of the “BITS” service on all servers in your list, you will need to specify the “Name” parameter. This command takes a hash pair formatted as follows:

@{"ParameterName" = "Value"}
@{"ParameterName" = "Value" ; "ParameterTwo" = "Value2"}

1 2	@{"ParameterName" = "Value"} @{"ParameterName" = "Value" ; "ParameterTwo" = "Value2"}

AddSwitch
This allows you to add additional switches to the command you are running. For instance, you may want to include “RequiredServices” to the “Get-Service” cmdlet. This parameter will take a single string, or an aray of strings as follows:

"RequiredServices"
@("RequiredServices", "DependentServices")

1 2	"RequiredServices" @("RequiredServices", "DependentServices")

MaxThreads
This is the maximum number of threads to run at any given time. If resources are too congested try lowering this number. The default value is 20.

SleepTimer
This is the time between cycles of the child process detection cycle. The default value is 200ms. If CPU utilization is high then you can consider increasing this delay. If the child script takes a long time to run, then you might increase this value to around 1000 (or 1 second in the detection cycle).

Now we need to set everything up. This stuff needs to execute outside of the pipeline, so we place it in the “begin” block of the script.

Begin{
    $ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
    $RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)
    $RunspacePool.Open()
        
    If ($(Get-Command | Select-Object Name) -match $Command){
        $Code = $Null
    }Else{
        $OFS = "`r`n"
        $Code = [ScriptBlock]::Create($(Get-Content $Command))
        Remove-Variable OFS
    }
    $Jobs = @()
}

Begin{

$ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()

$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)

$RunspacePool.Open()

If ($(Get-Command | Select-Object Name) -match $Command){

$Code = $Null

}Else{

$OFS = "`r`n"

$Code = [ScriptBlock]::Create($(Get-Content $Command))

Remove-Variable OFS

}

$Jobs = @()

}

Okay, so to break this down, first we need to make our ISS, or initial session state. This is basically the session state to be used when we open our Runspace. Next we create our Runspaces. The RunspacePool is really what’s going to do the multithreading. It will handle starting our threads and continuously start new ones as required. It is the operating environment for our command pipeline. Finally we open the RunSpacePool. Note that “.Open()” opens the RunSpacePool synchronously, creating a Windows PowerShell execution environment.

Now I am running a detection on what the user provided for the $command parameter. First I will look at all of the currently loaded cmdlets. If it is one of those, then we continue. Otherwise we assume it is a script file. If it is a script file then we need to read the file in to a script block that we can pass to our future threads. To do this we need to change the default $OFS (Object Field Separator), for more understanding here, please read my other post!

Okay, so the next step is to start receiving items from the pipeline. We can do this by starting the process block. Note that the process block is executed for each item we find in the pipeline. Meanwhile, if you did not execute in the pipeline it is executed once for the script as a whole. What this means is that we need to assume that $ObjectList will either be a single item or multiple items. The best way to do that is to use a ForEach Loop.

Process{
    Write-Progress -Activity "Preloading threads" -Status "Starting Job $($jobs.count)"
    ForEach ($Object in $ObjectList){
        If ($Code -eq $Null){
            $PowershellThread = [powershell]::Create().AddCommand($Command)
        }Else{
            $PowershellThread = [powershell]::Create().AddScript($Code)
        }
        If ($InputParam -ne $Null){
            $PowershellThread.AddParameter($InputParam, $Object.ToString()) | out-null
        }Else{
            $PowershellThread.AddArgument($Object.ToString()) | out-null
        }
        ForEach($Key in $AddParam.Keys){
            $PowershellThread.AddParameter($Key, $AddParam.$key) | out-null
        }
        ForEach($Switch in $AddSwitch){
            $Switch
            $PowershellThread.AddParameter($Switch) | out-null
        }
        $PowershellThread.RunspacePool = $RunspacePool
        $Handle = $PowershellThread.BeginInvoke()
        $Job = "" | Select-Object Handle, Thread, object
        $Job.Handle = $Handle
        $Job.Thread = $PowershellThread
        $Job.Object = $Object.ToString()
        $Jobs += $Job
    }
        
}

Process{

Write-Progress -Activity "Preloading threads" -Status "Starting Job $($jobs.count)"

ForEach ($Object in $ObjectList){

If ($Code -eq $Null){

$PowershellThread = [powershell]::Create().AddCommand($Command)

}Else{

$PowershellThread = [powershell]::Create().AddScript($Code)

}

If ($InputParam -ne $Null){

$PowershellThread.AddParameter($InputParam, $Object.ToString()) | out-null

}Else{

$PowershellThread.AddArgument($Object.ToString()) | out-null

}

ForEach($Key in $AddParam.Keys){

$PowershellThread.AddParameter($Key, $AddParam.$key) | out-null

}

ForEach($Switch in $AddSwitch){

$Switch

$PowershellThread.AddParameter($Switch) | out-null

}

$PowershellThread.RunspacePool = $RunspacePool

$Handle = $PowershellThread.BeginInvoke()

$Job = "" | Select-Object Handle, Thread, object

$Job.Handle = $Handle

$Job.Thread = $PowershellThread

$Job.Object = $Object.ToString()

$Jobs += $Job

}

So now we have to build the thread that we are going to execute. We do this by adding either the command, or the script. The first IF block is to determine which. A thread either takes an existing PowerShell command, or a Scriptblock. If you remember we built this out in the Begin statement above. Once that is done we need to start giving the user the power to control the item we are calling.

First things first we look at $InputParam. This is what allows the user to execute the child script not just with the argument provided, but also specify the parameter. We see this is useful with the Get-Process cmdlet. Let’s say that you want to see the processes running on 20 different servers. If you just ran Get-Process ServerName you would be looking at your local machine for any processes with the name “ServerName” and you would get no return (probably). Instead you would want to run Get-Process –ComputerName ServerName. The trick here is that when you do this you’ve actually changed things! When an item is just hanging at the end of the statement, it is called an Argument. When you pair the item you are setting with the setting it is called a Parameter. So if the user wants to specify the parameter name, we are actually adding a different item to our thread!

Now we need to see if the user wanted to add some extra parameters. For this I decided that a hash table was perfect. This is because they are built much like a parameter, as they are name / value pairs. The user can provide as many as they want in a single hash table, and we can easily run a ForEach to evaluate this. Again we are going to use the .AddParameter() statement to evaluate.

Finally we need to add any Switches that the user wants to add. A good example of this is the –Force switch on cmdlets like Get-ChildItem. We can do this with an array of string which again allows the user to put in as many various switches as they like. One interesting not here is that I had to use .AddParameter() for this as well. Instead of creating a method called .AddSwitch(), Microsoft simply chose to add an overloaded definition of .AddParameter(). What’s peculiar is that even in their documentation (link below) it does in fact say that providing just a string adds a switch instead of parameter.

Now that we have out thread all set up we simply attach our RunspacePool that we setup in the begin statement, and then tell it to execute our thread!

To avoid having a thread hanging around that we lose track of we need to try to do some tracking here. To do this we first catch the thread by creating the output of the command in a $Handle variable. This will provide the link back to the handle in our “End” block. I then go ahead and create a custom object which I have named $Job to hold all these little gems of knowledge. Then I add my custom object to the array for tracking called $Jobs. This method of creating objects was taught to me by my reader from this post!

At this point all of the code to start the jobs is complete! Now we just have to grab all of the jobs back. That calls for the “End” block which is executed once per script run. Since that is the case, we need one main loop to ensure that it stay running while we need it to. Within the “End” block we will watch for jobs to finish and provide that output as they complete.

End{
    $ResultTimer = Get-Date
    While (@($Jobs | Where-Object {$_.Handle -ne $Null}).count -gt 0)  {
    
        $Remaining = "$($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False}).object)"
        If ($Remaining.Length -gt 60){
            $Remaining = $Remaining.Substring(0,60) + "..."
        }
        Write-Progress `
            -Activity "Waiting for Jobs - $($MaxThreads - $($RunspacePool.GetAvailableRunspaces())) of $MaxThreads threads running" `
            -PercentComplete (($Jobs.count - $($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False}).count)) / $Jobs.Count * 100) `
            -Status "$(@($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False})).count) remaining - $remaining" 

        ForEach ($Job in $($Jobs | Where-Object {$_.Handle.IsCompleted -eq $True})){
            $Job.Thread.EndInvoke($Job.Handle)
            $Job.Thread.Dispose()
            $Job.Thread = $Null
            $Job.Handle = $Null
            $ResultTimer = Get-Date
        }
        If (($(Get-Date) - $ResultTimer).totalseconds -gt $MaxResultTime){
            Write-Error "Child script appears to be frozen, try increasing MaxResultTime"
            Exit
        }
        Start-Sleep -Milliseconds $SleepTimer
        
    } 
    $RunspacePool.Close() | Out-Null
    $RunspacePool.Dispose() | Out-Null
}

End{

$ResultTimer = Get-Date

While (@($Jobs | Where-Object {$_.Handle -ne $Null}).count -gt 0) {

$Remaining = "$($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False}).object)"

If ($Remaining.Length -gt 60){

$Remaining = $Remaining.Substring(0,60) + "..."

}

Write-Progress `

-Activity "Waiting for Jobs - $($MaxThreads - $($RunspacePool.GetAvailableRunspaces())) of $MaxThreads threads running" `

-PercentComplete (($Jobs.count - $($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False}).count)) / $Jobs.Count * 100) `

-Status "$(@($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False})).count) remaining - $remaining"

ForEach ($Job in $($Jobs | Where-Object {$_.Handle.IsCompleted -eq $True})){

$Job.Thread.EndInvoke($Job.Handle)

$Job.Thread.Dispose()

$Job.Thread = $Null

$Job.Handle = $Null

$ResultTimer = Get-Date

}

If (($(Get-Date) - $ResultTimer).totalseconds -gt $MaxResultTime){

Write-Error "Child script appears to be frozen, try increasing MaxResultTime"

Exit

}

Start-Sleep -Milliseconds $SleepTimer

}

$RunspacePool.Close() | Out-Null

$RunspacePool.Dispose() | Out-Null

}

The first part of this is simply to provide some form of pretty output. First, I evaluate what jobs are still running and creating a string (truncated) that names them. I added this so that if the child script is failing on just one or two servers, you will known which they are to fix them. Then a rather complex write-progress statement which I’ll let you look into. For More info on write-progress you can read my blog articles over write-progress or getting the progress of a child job.

Okay, so now we look for jobs that are completed. We can do this by looking into our array of objects where our handle (that we captured above) has .IsCompleted set to $true! We then run a ForEach on each of these to stop it running and dispose of it. Keep in mind that dispose returns all of the output from the thread. What this means is that the script will actually write all of the output as jobs are finished instead of having to wait for all jobs to finish!

Next I added some protection to the script. I had a couple of scripts in my arsenal that would lock up and never finish. When this happened the multithreading script would continue to run and loop forever and ever. To stop this I added a maximum time to wait for additional jobs to finish. To do this I simply look at the system clock for when the last job completed and look at the time gap. If it is greater than out parameter for $MaxResultTime then we’ll throw an error and exit. Note that until PowerShell actually closes those threads will continue to hang around!

As a very last step we clean up our $RunspacePool.

Well that’s all folks! I really hope that this script provide many time saving events for my wonderful readers. I know it has saved me hundreds of man hours!

Following is the full script with the comment block intact for your cutting and pasting pleasure! Note that you can use the advanced controls here to pop out to a new window or show plain code for copy and paste.

#.Synopsis
#    This is a quick and open-ended script multi-threader searcher
#    
#.Description
#    This script will allow any general, external script to be multithreaded by providing a single
#    argument to that script and opening it in a seperate thread.  It works as a filter in the 
#    pipeline, or as a standalone script.  It will read the argument either from the pipeline
#    or from a filename provided.  It will send the results of the child script down the pipeline,
#    so it is best to use a script that returns some sort of object.
#
#    Authored by Ryan Witschger - http://www.Get-Blog.com
#    
#.PARAMETER Command
#    This is where you provide the PowerShell Cmdlet / Script file that you want to multithread.  
#    You can also choose a built in cmdlet.  Keep in mind that your script.  This script is read into 
#    a scriptblock, so any unforeseen errors are likely caused by the conversion to a script block.
#    
#.PARAMETER ObjectList
#    The objectlist represents the arguments that are provided to the child script.  This is an open ended
#    argument and can take a single object from the pipeline, an array, a collection, or a file name.  The 
#    multithreading script does it's best to find out which you have provided and handle it as such.  
#    If you would like to provide a file, then the file is read with one object on each line and will 
#    be provided as is to the script you are running as a string.  If this is not desired, then use an array.
#    
#.PARAMETER InputParam
#    This allows you to specify the parameter for which your input objects are to be evaluated.  As an example, 
#    if you were to provide a computer name to the Get-Process cmdlet as just an argument, it would attempt to 
#    find all processes where the name was the provided computername and fail.  You need to specify that the 
#    parameter that you are providing is the "ComputerName".
#
#.PARAMETER AddParam
#    This allows you to specify additional parameters to the running command.  For instance, if you are trying
#    to find the status of the "BITS" service on all servers in your list, you will need to specify the "Name"
#    parameter.  This command takes a hash pair formatted as follows:  
#
#    @{"ParameterName" = "Value"}
#    @{"ParameterName" = "Value" ; "ParameterTwo" = "Value2"}
#
#.PARAMETER AddSwitch
#    This allows you to add additional switches to the command you are running.  For instance, you may want 
#    to include "RequiredServices" to the "Get-Service" cmdlet.  This parameter will take a single string, or 
#    an aray of strings as follows:
#
#    "RequiredServices"
#    @("RequiredServices", "DependentServices")
#
#.PARAMETER MaxThreads
#    This is the maximum number of threads to run at any given time.  If resources are too congested try lowering
#    this number.  The default value is 20.
#    
#.PARAMETER SleepTimer
#    This is the time between cycles of the child process detection cycle.  The default value is 200ms.  If CPU 
#    utilization is high then you can consider increasing this delay.  If the child script takes a long time to
#    run, then you might increase this value to around 1000 (or 1 second in the detection cycle).
#
#    
#.EXAMPLE
#    Both of these will execute the script named ServerInfo.ps1 and provide each of the server names in AllServers.txt
#    while providing the results to the screen.  The results will be the output of the child script.
#    
#    gc AllServers.txt | .\Run-CommandMultiThreaded.ps1 -Command .\ServerInfo.ps1
#    .\Run-CommandMultiThreaded.ps1 -Command .\ServerInfo.ps1 -ObjectList (gc .\AllServers.txt)
#
#.EXAMPLE
#    The following demonstrates the use of the AddParam statement
#    
#    $ObjectList | .\Run-CommandMultiThreaded.ps1 -Command "Get-Service" -InputParam ComputerName -AddParam @{"Name" = "BITS"}
#    
#.EXAMPLE
#    The following demonstrates the use of the AddSwitch statement
#    
#    $ObjectList | .\Run-CommandMultiThreaded.ps1 -Command "Get-Service" -AddSwitch @("RequiredServices", "DependentServices")
#
#.EXAMPLE
#    The following demonstrates the use of the script in the pipeline
#    
#    $ObjectList | .\Run-CommandMultiThreaded.ps1 -Command "Get-Service" -InputParam ComputerName -AddParam @{"Name" = "BITS"} | Select Status, MachineName
#


Param($Command = $(Read-Host "Enter the script file"), 
    [Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]$ObjectList,
    $InputParam = $Null,
    $MaxThreads = 20,
    $SleepTimer = 200,
    $MaxResultTime = 120,
    [HashTable]$AddParam = @{},
    [Array]$AddSwitch = @()
)

Begin{
    $ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
    $RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)
    $RunspacePool.Open()
        
    If ($(Get-Command | Select-Object Name) -match $Command){
        $Code = $Null
    }Else{
        $OFS = "`r`n"
        $Code = [ScriptBlock]::Create($(Get-Content $Command))
        Remove-Variable OFS
    }
    $Jobs = @()
}

Process{
    Write-Progress -Activity "Preloading threads" -Status "Starting Job $($jobs.count)"
    ForEach ($Object in $ObjectList){
        If ($Code -eq $Null){
            $PowershellThread = [powershell]::Create().AddCommand($Command)
        }Else{
            $PowershellThread = [powershell]::Create().AddScript($Code)
        }
        If ($InputParam -ne $Null){
            $PowershellThread.AddParameter($InputParam, $Object.ToString()) | out-null
        }Else{
            $PowershellThread.AddArgument($Object.ToString()) | out-null
        }
        ForEach($Key in $AddParam.Keys){
            $PowershellThread.AddParameter($Key, $AddParam.$key) | out-null
        }
        ForEach($Switch in $AddSwitch){
            $Switch
            $PowershellThread.AddParameter($Switch) | out-null
        }
        $PowershellThread.RunspacePool = $RunspacePool
        $Handle = $PowershellThread.BeginInvoke()
        $Job = "" | Select-Object Handle, Thread, object
        $Job.Handle = $Handle
        $Job.Thread = $PowershellThread
        $Job.Object = $Object.ToString()
        $Jobs += $Job
    }
        
}

End{
    $ResultTimer = Get-Date
    While (@($Jobs | Where-Object {$_.Handle -ne $Null}).count -gt 0)  {
    
        $Remaining = "$($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False}).object)"
        If ($Remaining.Length -gt 60){
            $Remaining = $Remaining.Substring(0,60) + "..."
        }
        Write-Progress `
            -Activity "Waiting for Jobs - $($MaxThreads - $($RunspacePool.GetAvailableRunspaces())) of $MaxThreads threads running" `
            -PercentComplete (($Jobs.count - $($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False}).count)) / $Jobs.Count * 100) `
            -Status "$(@($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False})).count) remaining - $remaining" 

        ForEach ($Job in $($Jobs | Where-Object {$_.Handle.IsCompleted -eq $True})){
            $Job.Thread.EndInvoke($Job.Handle)
            $Job.Thread.Dispose()
            $Job.Thread = $Null
            $Job.Handle = $Null
            $ResultTimer = Get-Date
        }
        If (($(Get-Date) - $ResultTimer).totalseconds -gt $MaxResultTime){
            Write-Error "Child script appears to be frozen, try increasing MaxResultTime"
            Exit
        }
        Start-Sleep -Milliseconds $SleepTimer
        
    } 
    $RunspacePool.Close() | Out-Null
    $RunspacePool.Dispose() | Out-Null
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

#.Synopsis

# This is a quick and open-ended script multi-threader searcher

#.Description

# This script will allow any general, external script to be multithreaded by providing a single

# argument to that script and opening it in a seperate thread. It works as a filter in the

# pipeline, or as a standalone script. It will read the argument either from the pipeline

# or from a filename provided. It will send the results of the child script down the pipeline,

# so it is best to use a script that returns some sort of object.

# Authored by Ryan Witschger - http://www.Get-Blog.com

#.PARAMETER Command

# This is where you provide the PowerShell Cmdlet / Script file that you want to multithread.

# You can also choose a built in cmdlet. Keep in mind that your script. This script is read into

# a scriptblock, so any unforeseen errors are likely caused by the conversion to a script block.

#.PARAMETER ObjectList

# The objectlist represents the arguments that are provided to the child script. This is an open ended

# argument and can take a single object from the pipeline, an array, a collection, or a file name. The

# multithreading script does it's best to find out which you have provided and handle it as such.

# If you would like to provide a file, then the file is read with one object on each line and will

# be provided as is to the script you are running as a string. If this is not desired, then use an array.

#.PARAMETER InputParam

# This allows you to specify the parameter for which your input objects are to be evaluated. As an example,

# if you were to provide a computer name to the Get-Process cmdlet as just an argument, it would attempt to

# find all processes where the name was the provided computername and fail. You need to specify that the

# parameter that you are providing is the "ComputerName".

#.PARAMETER AddParam

# This allows you to specify additional parameters to the running command. For instance, if you are trying

# to find the status of the "BITS" service on all servers in your list, you will need to specify the "Name"

# parameter. This command takes a hash pair formatted as follows:

# @{"ParameterName" = "Value"}

# @{"ParameterName" = "Value" ; "ParameterTwo" = "Value2"}

#.PARAMETER AddSwitch

# This allows you to add additional switches to the command you are running. For instance, you may want

# to include "RequiredServices" to the "Get-Service" cmdlet. This parameter will take a single string, or

# an aray of strings as follows:

# "RequiredServices"

# @("RequiredServices", "DependentServices")

#.PARAMETER MaxThreads

# This is the maximum number of threads to run at any given time. If resources are too congested try lowering

# this number. The default value is 20.

#.PARAMETER SleepTimer

# This is the time between cycles of the child process detection cycle. The default value is 200ms. If CPU

# utilization is high then you can consider increasing this delay. If the child script takes a long time to

# run, then you might increase this value to around 1000 (or 1 second in the detection cycle).

#.EXAMPLE

# Both of these will execute the script named ServerInfo.ps1 and provide each of the server names in AllServers.txt

# while providing the results to the screen. The results will be the output of the child script.

# gc AllServers.txt | .\Run-CommandMultiThreaded.ps1 -Command .\ServerInfo.ps1

# .\Run-CommandMultiThreaded.ps1 -Command .\ServerInfo.ps1 -ObjectList (gc .\AllServers.txt)

#.EXAMPLE

# The following demonstrates the use of the AddParam statement

# $ObjectList | .\Run-CommandMultiThreaded.ps1 -Command "Get-Service" -InputParam ComputerName -AddParam @{"Name" = "BITS"}

#.EXAMPLE

# The following demonstrates the use of the AddSwitch statement

# $ObjectList | .\Run-CommandMultiThreaded.ps1 -Command "Get-Service" -AddSwitch @("RequiredServices", "DependentServices")

#.EXAMPLE

# The following demonstrates the use of the script in the pipeline

# $ObjectList | .\Run-CommandMultiThreaded.ps1 -Command "Get-Service" -InputParam ComputerName -AddParam @{"Name" = "BITS"} | Select Status, MachineName

Param($Command = $(Read-Host "Enter the script file"),

[Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]$ObjectList,

$InputParam = $Null,

$MaxThreads = 20,

$SleepTimer = 200,

$MaxResultTime = 120,

[HashTable]$AddParam = @{},

[Array]$AddSwitch = @()

)

Begin{

$ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()

$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)

$RunspacePool.Open()

If ($(Get-Command | Select-Object Name) -match $Command){

$Code = $Null

}Else{

$OFS = "`r`n"

$Code = [ScriptBlock]::Create($(Get-Content $Command))

Remove-Variable OFS

}

$Jobs = @()

}

Process{

Write-Progress -Activity "Preloading threads" -Status "Starting Job $($jobs.count)"

ForEach ($Object in $ObjectList){

If ($Code -eq $Null){

$PowershellThread = [powershell]::Create().AddCommand($Command)

}Else{

$PowershellThread = [powershell]::Create().AddScript($Code)

}

If ($InputParam -ne $Null){

$PowershellThread.AddParameter($InputParam, $Object.ToString()) | out-null

}Else{

$PowershellThread.AddArgument($Object.ToString()) | out-null

}

ForEach($Key in $AddParam.Keys){

$PowershellThread.AddParameter($Key, $AddParam.$key) | out-null

}

ForEach($Switch in $AddSwitch){

$Switch

$PowershellThread.AddParameter($Switch) | out-null

}

$PowershellThread.RunspacePool = $RunspacePool

$Handle = $PowershellThread.BeginInvoke()

$Job = "" | Select-Object Handle, Thread, object

$Job.Handle = $Handle

$Job.Thread = $PowershellThread

$Job.Object = $Object.ToString()

$Jobs += $Job

}

End{

$ResultTimer = Get-Date

While (@($Jobs | Where-Object {$_.Handle -ne $Null}).count -gt 0) {

$Remaining = "$($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False}).object)"

If ($Remaining.Length -gt 60){

$Remaining = $Remaining.Substring(0,60) + "..."

}

Write-Progress `

-Activity "Waiting for Jobs - $($MaxThreads - $($RunspacePool.GetAvailableRunspaces())) of $MaxThreads threads running" `

-PercentComplete (($Jobs.count - $($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False}).count)) / $Jobs.Count * 100) `

-Status "$(@($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False})).count) remaining - $remaining"

ForEach ($Job in $($Jobs | Where-Object {$_.Handle.IsCompleted -eq $True})){

$Job.Thread.EndInvoke($Job.Handle)

$Job.Thread.Dispose()

$Job.Thread = $Null

$Job.Handle = $Null

$ResultTimer = Get-Date

}

If (($(Get-Date) - $ResultTimer).totalseconds -gt $MaxResultTime){

Write-Error "Child script appears to be frozen, try increasing MaxResultTime"

Exit

}

Start-Sleep -Milliseconds $SleepTimer

}

$RunspacePool.Close() | Out-Null

$RunspacePool.Dispose() | Out-Null

}

Further Reading:
Initial Session State:
http://msdn.microsoft.com/en-us/library/system.management.automation.runspaces.initialsessionstate%28v=vs.85%29.aspx
Runspaces:
http://msdn.microsoft.com/en-us/library/System.Management.Automation.Runspaces.Runspace(v=vs.85).aspx
PowerShell Class:
http://msdn.microsoft.com/en-us/library/system.management.automation.powershell%28v=vs.85%29.aspx
PowerShell AddParameter Method:
http://msdn.microsoft.com/en-us/library/system.management.automation.powershell.addparameter%28v=vs.85%29.aspx

77 Responses to “True Multithreading in PowerShell”

Vijay says:

August 12, 2023 at 1:03 am

Hello Ryan,

I’m trying to use your script and below is the Error received.

I have tried multiple ways of executing it. Doesn’t work. Please help.

command : .\MultiThreading.ps1 -Command .\Test.ps1 -ObjectList (gc .\Servers1.txt)

Error:
Exception calling “EndInvoke” with “1” argument(s): “The term ‘.\Test.ps1’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling
of the name, or if a path was included, verify that the path is correct and try again.”
At C:\XXXX\MultiThreading.ps1:160 char:13
+ $Job.Thread.EndInvoke($Job.Handle)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : CommandNotFoundException

Reply
- Ryan Witschger says:
  
  August 29, 2023 at 11:26 pm
  
  You don’t need to include the dot notation in the test.ps1 call
  
  Reply
Dbh1985 says:

December 15, 2018 at 12:04 pm

Hello all. I’m new to this forum and a beginner in powershell scripting, but I was able to get it working in my environment.

Goal: run batch,exe,vbs script simultaneously on multiple computers (up to 60) without hanging with admin credentials included
Success:
1) copy above script as mt.ps1 to directory (thanks to Ryan’s hard work on this)
2) create powershell script as test.ps1 to run scripts on multiple computers
A) enter in credential information in this script (mine was 4 lines of code) $username=”admin” $password=”Password” and other 2 lines were powershell credential code
B) invoke expression -computername $computer $credential -scriptblock {“cmd.exe ‘c:\remotefolder\test.bat'”}
3) run below code in powershell or you can save it as .ps1 file
.\mt.ps1 -command .\test.ps1 -objectlist (gc .\IP.txt)

Note* I’m not in front of computer so I will post my test scripting later but this is the general method I used to get it to work. I only have tested this multithreaded script for a few days so I may later change parameters for better use (will keep everyone updated)

When I ran this scripting method it worked like a champ, no hanging even when I put in IP’s to the list that weren’t pingable. I’m used to psexec method which in my experience only worked well running sequentially (line by line). Also, I had to take out logging in the external scripts so I have to get this working from within powershell most likely.

Other things I would like to implement using Ryan’s script (credit again to his hard work) would be a GUI that will possibly show computers in green (successful ping and remote execution) and red (failure pinging and remote execution). I have an autoit script to do this (will post later) but I would like it to be more user friendly.

Let me know if anyone has any questions and I will be happy to answer. Ryan, thanks again, your script is awesome and very flexible to use in any environment I must say.

Reply
James says:

November 1, 2018 at 3:29 pm

Pretty neat… but am having some issues getting output…

I have a script that returns info from a WMI query…. I have set it up so that it returns the data in an array (I think that is a correct description) so that it can be piped elsewhere.

When I run it with this, I do not see any of the output….

If, however I run the script from the command line, it works

Reply
- Ryan Witschger says:
  
  December 13, 2019 at 7:43 am
  
  It sounds like you may be directly writing your output to somewhere other than standard or default out?
  
  Reply
Adam says:

October 30, 2018 at 8:40 am

— Q: The variable “$MaxResultTime” is the max total second that the child script should finish or expectation? While the variable “$SleepTimer” is the check between the child cycle of the job.

If (($(Get-Date) – $ResultTimer).totalseconds -gt $MaxResultTime){
Write-Error “Child script appears to be frozen, try increasing MaxResultTime”
Exit
}

Reply
- Ryan Witschger says:
  
  December 13, 2019 at 7:44 am
  
  That is correct, yes.
  
  Reply
Ravi says:

September 13, 2018 at 12:40 am

WOW… Hat’s off to you Ryan.. I was pulling data from multiple servers and even after using workflow foreach -parallel the job was completing in 30 mins. With your script that 30 min job completed in 30 seconds.. I am amazed to see the results. Thank you very much. And also I noticed the resource utilization on parent server is minimal for both CPU and memory… Once again thank you and great work.

Reply
Randy says:

June 28, 2018 at 3:55 pm

This is amazing. Took me a little while to figure out, then it clicked.

I wrapped it in:
Function Run-CommandMultiThreaded {
[CmdletBinding()]
…
}

so it can be reused.

Also, you need to escape the Regex comparison of $Command with [Regex]::Escape. Otherwise, flawless! Love it!!

Reply
Brian says:

May 5, 2018 at 9:33 am

Hi Ryan, Thanks for your post, this has been invaluable. Below are some optimizations I’ve used in my derivative, some found elsewhere but together make a big difference.

1. Appending to an array creates a new array, so the “$Jobs += $Job” line is expensive, can be optimized (on a script with 50k jobs, this cut run time to a third).
2. out-null is expensive, instead use “$null = foo” rather than “foo | out-null”
3. I renamed a $iss for clarity.
4. progress display slows things down especially invoke-webrequest, it’s fun to watch but I dispensed
5. foreach-object is faster than where-object, and without progress the disposal loop can be refactored.

Param($Command = $(Read-Host "Enter the script file"), [Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]$ObjectList, $InputParam = $Null, $MaxThreads = 20, $SleepTimer = 200, $MaxResultTime = 120, [HashTable]$AddParam = @{}, [Array]$AddSwitch = @() ) Begin{ $SecurityProtocolSaved = [System.Net.ServicePointManager]::SecurityProtocol $ProgressSaved = $global:ProgressPreference [System.Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 $global:progressPreference = 'silentlyContinue' $InitialSessionState = [system.management.automation.runspaces.initialsessionstate]::CreateDefault() $RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $InitialSessionState, $Host) $RunspacePool.Open()
If ($(Get-Command | Select-Object Name) -match $Command){ $Code = $Null }Else{ $OFS = "rn" # OFS special variable Ouptut Field Sperator
$Code = [ScriptBlock]::Create($(Get-Content $Command))
Remove-Variable OFS
}
$Jobs = New-Object System.Collections.Generic.List[System.Object]
}

Process{
ForEach ($Object in $ObjectList){
If ($Code -eq $Null){
$PowershellThread = [powershell]::Create().AddCommand($Command)
}Else{
$PowershellThread = [powershell]::Create().AddScript($Code)
}
If ($InputParam -ne $Null){
$null = $PowershellThread.AddParameter($InputParam, $Object.ToString())
}Else{
$null = $PowershellThread.AddArgument($Object.ToString())
}
ForEach($Key in $AddParam.Keys){
$null = $PowershellThread.AddParameter($Key, $AddParam.$key)
}
ForEach($Switch in $AddSwitch){
$Switch
$null = $PowershellThread.AddParameter($Switch)
}
$PowershellThread.RunspacePool = $RunspacePool
$Handle = $PowershellThread.BeginInvoke()
$Job = "" | Select-Object Handle, Thread, object
$Job.Handle = $Handle
$Job.Thread = $PowershellThread
$Job.Object = $Object.ToString()
$Jobs.Add($Job)
}

}

End{
$running = $true
While ($running) {
$Running = $false
$Jobs | ForEach-object {if ($_.Handle.IsCompleted -eq $True) {
$_.Thread.EndInvoke($_.Handle)
$_.Thread.Dispose()
$_.Thread = $Null
$_.Handle = $Null
} elseif ((-not $Running) -and ($_.Handle -ne $Null)) {
$Running = $true
}
}
Start-Sleep -Milliseconds $SleepTimer
}
$null = $RunspacePool.Close()
$null = $RunspacePool.Dispose()
[System.Net.ServicePointManager]::SecurityProtocol = $SecurityProtocolSaved
$global:progressPreference = $ProgressSaved
}

Reply
John Thayer Jensen says:

March 8, 2018 at 11:52 am

If anyone is reading this, I would love help! Briefly, when a job hangs, the whole thing hangs. The MaxResultTime parameter is not killing the job. I don’t know what to do.

Reply
- Ryan Witschger says:
  
  December 13, 2019 at 7:50 am
  
  In general, my answer would be to first troubleshoot the child script to understand why it’s hanging and repair that. For my own use, the only trouble I do in the multi-threading script is only to help identify that inputs which might have issues, I fix them and then run again.
  
  Reply
John Thayer Jensen says:

February 19, 2018 at 11:02 am

And even that didn’t quite work. It had to be:

if (($Command -is [String]) -and (Test-Path “$Command”)){$CommandName = (Get-Item $Command).Name}

Reply
John Thayer Jensen says:

February 18, 2018 at 2:29 pm

I had to modify the Begin block from:

If ($(Get-Command | Select-Object Name) -match $Command){
$Code = $Null
}Else{
$OFS = “rn”
$Code = [ScriptBlock]::Create($(Get-Content $Command))
Remove-Variable OFS
}

to:

$CommandName = $Command
if (Test-Path $Command) {$CommandName = (Get-Item $Command).Name}
If ($(Get-Command | Select-Object Name) -match $CommandName)
{
$Code = $Null
}
Elseif ($Command -is [ScriptBlock])
{
$Code = $Command
}
Else
{
$OFS = “rn”
$Code = [ScriptBlock]::Create($(Get-Content $Command))
Remove-Variable OFS
}

Part was to pass a scriptblock per @Sam’s suggestion above at http://www.get-blog.com/?p=189#comment-33949

But the other part was that when I passed a file name with a path as the command, the ‘match’ failed, trying to interpret the ‘\’ in the path as an escape character.

jj

Reply
Kevin says:

November 3, 2017 at 12:33 pm

OK I figured it out… arg…. if anyone needs it this worked for me.

‘db01′,’sp01’ | Run-CommandMultiThreaded.ps1 -Command “Get-DiskFree” -InputParam computername -AddSwitch @(“-format”) | ft -GroupBy Name -AutoSize

Reply
Kevin says:

November 3, 2017 at 12:08 pm

Need some help. Here is how I would run my command which is a module.

‘db01′,’sp01’ | Get-DiskFree -Format | ft -GroupBy Name -auto

“-Format | ft -GroupBy Name -auto” — this is the part I can’t get working.

If I can get this working I have other output formats I should be able to figure out but I can’t seem to get this to work no matter how I try it. Really appreciate the script and help.

Reply
Tom Marshall says:

October 12, 2017 at 3:24 pm

This is freaking awesome!!

I have a PowerShell script that I created that will run daily and frequently take upwards of 20,000 folders containing more sub-folders and files and zip them into a matching number of zip files that internally maintain the original file structure.

The script took like 7 hours to process one day’s worth of files.

I took your script, and with a few minor modifications, I can now process the entire bunch in under 30 minutes.

That is so awesome!! I just had to share and say thank you. I don’t know if I would ever have figured this out on my own.

Thank You!!!

Reply
Kevin says:

August 21, 2017 at 1:49 pm

Ryan=Rockstar! Awesome Script.

Reply
Nicholas says:

July 19, 2017 at 6:05 am

Hi there – ive tried to run this script to multithread an EWS script I have which loops through multiple mailboxes and reports on retentiontags applied at the folder level and total count per folder – but i get nothing back? ie. my script generates a CSV file of results at the end? also it keeps prompting me for the same creds again and again that are requested once in the Begin{} block of my script? My script accepts pipeline input and has a begin{}, process{} and end{} block is this somehow messing up running this function?

Reply
Eric says:

May 24, 2017 at 8:42 am

Modified the Process area to allow import-csv. It works so far for me but please test on your own box!

Process { Write-Progress -Activity "Preloading threads" -Status "Starting Job $($jobs.count)" ForEach ($Object in $ObjectList) { If ($Code -eq $Null) { $PowershellThread = [powershell]::Create().AddCommand($Command) } Else { $PowershellThread = [powershell]::Create().AddScript($Code) } If ($InputParam -ne $Null) { If ($Object -is [System.Management.Automation.PSCustomObject]) { $PowershellThread.AddParameter($InputParam, $Object) | out-null } Else { $PowershellThread.AddParameter($InputParam, $Object.ToString()) | out-null } } Else { $PowershellThread.AddArgument($Object.ToString()) | out-null } ForEach ($Key in $AddParam.Keys) { $PowershellThread.AddParameter($Key, $AddParam.$key) | out-null } ForEach ($Switch in $AddSwitch) { $Switch $PowershellThread.AddParameter($Switch) | out-null } $PowershellThread.RunspacePool = $RunspacePool $Handle = $PowershellThread.BeginInvoke() $Job = "" | Select-Object Handle, Thread, object $Job.Handle = $Handle $Job.Thread = $PowershellThread If ($Object -is [System.Management.Automation.PSCustomObject]) { $Job.Object = ($Object.($Object.psobject.properties.name | Select-Object -first 1)) } Else {$Job.Object = $Object.ToString()} $Jobs += $Job }
}

Reply
- Eric says:
  
  May 24, 2017 at 10:55 am
  
  I guess I should add it’s usage. I didn’t see an edit button so..
  
  .\Run-CommandMultiThreaded.ps1 -Command "C:\temp\test.ps1" -ObjectList (Import-Csv "C:\temp\test.csv") -InputParam "computer"
  
  Reply
Erik says:

April 27, 2017 at 9:14 am

I realize this topic is rather old, and no one has responded in a while, but I’ll take a shot and ask my question anyway.

Here is my example:
.\Run-CommandMultiThreaded.ps1 -Command .\copy_file_test.ps1 -ObjectList (gc .\target_computers.txt) -AddParam @{“fileName” = $fileName}

I can’t figure out how to access $fileName inside of copy_file_test.ps1. What am I missing here?

Thank you.

Reply
JohorB says:

March 27, 2017 at 11:59 am

@Ryan,

I have to say that this script is AWESOME!!! I have a customized Log Parser script that reads Windows events and creates PS objects at the end. To do that for the past 3 months events on 200+ servers in the lab took me 7 hours to finish and production has 1000+ servers. You can imagine this kind of performance is simply not acceptable. This script works and cut down my execution time to merely 3 MINUTES! I was “wow” by how good this is. I did notice that the memory kept climbing up while I tested it in the lab (my script deals with big objects by design) so I added a garbage collection action at the end but that was about it. Thank you for your initial hard work and posted this!

Reply
Improv says:

February 6, 2017 at 2:52 am

In case the started thread / cmdlet crashes, you need to escape the endinvoke command:

ForEach ($Job in $($Jobs | Where-Object {$_.Handle.IsCompleted -eq $True})){
try{
$Job.Thread.EndInvoke($Job.Handle)
$Job.Thread.Dispose()
}
catch{}

Reply
Scott says:

January 8, 2017 at 8:35 pm

Hi Ryan,

Thanks for the useful script. I made the below changes which were needed to support a UNC path to my script:

If ($(Get-Command | Select-Object Name) -match [RegEx]::Escape($Command)){ $Code = $Null }Else{ $OFS = "rn"
$Code = [ScriptBlock]::Create($(Get-Content FileSystem::$Command)) # FileSystem:: needed to support UNC paths
Remove-Variable OFS
}

Reply
Sam says:

December 7, 2016 at 7:16 pm

I love this piece of code.

Just a small addition to the if statement in the begin segment
#—-
} elseif ($command -is [ScriptBlock]) {
$Code = $Command
#—-

Now you can use a scriptblock to define the code for multithreading to execute, reducing the need for an extra file.

Reply
Michael says:

December 7, 2016 at 10:35 am

I have gotten this working with massive WMI calls and it’s fantastic!

However I am having one problem. I want to use this code to execute Invoke-Command and run some commands on remote systems in parallel. Unfortunately, Invoke-Command requires a parameter -ScriptBlock which takes, of course, a script block instead of a string… but it seems like this code only allows you to specify additional parameters as strings using the -AddParam parameter. So when I try to execute this command, I receive an error that PowerShell cannot convert the string to a scriptblock.

Any ways to get around this?

My code looks something like this:

$statsUsercount = $selectedServerList | select Server | foreach {$_.Server} | .\Run-CommandMultiThreaded.ps1 -command “invoke-command” -InputParam ComputerName -AddParam @{“ScriptBlock” = “(netstat -an | select-string “1290”).count”}

And this is my error:

Invoke-Command : Cannot bind parameter ‘ScriptBlock’. Cannot convert the “” value of type “System.String” to type “Syst
em.Management.Automation.ScriptBlock”.
+ CategoryInfo : InvalidArgument: (:) [Invoke-Command], ParameterBindingException
+ FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.PowerShell.Commands.InvokeCommandCommand

Reply
Alex says:

October 13, 2016 at 6:24 am

This has proven very valuable and informative. a few snags I hit along the way:

input $Object is being stringified via ToString(). kills more complex scenarios where the full object is actually required.

Switch is being written to output. needed to remove that to clean up the return value

adding a ps1 file as command caused escaping issue. as previously mentioned. I added a if statement to deal with that.

Reply
Alex says:

August 18, 2016 at 7:01 am

This was very helpful to me also. Thanks for posting this.

One thing you didn’t add was how to report errors from the command or script that runs in the runspace. $powershellthread.haderrors reports whether the script encountered errors, and the actual return of the error text is in $powershellthread.streams.error.

Reply
Alban LOPEZ says:

March 15, 2016 at 4:05 am

Very cool code !
I improove it for use ScriptBlock et Load PsSnapin in the spool

$ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault() [void]$ISS.ImportPSSnapIn('Quest.ActiveRoles.ADManagement', [ref]$null) [void]$ISS.ImportPSModule('PSTerminalServices')

Reply
Jorge says:

February 24, 2016 at 2:56 pm

Hi Ryan

Many Belated Thanks for posting this jewel.

It was today that I found this section for comments.

I have been using your script for one year or more, it rocks big time: I have been able to populate sets of 40 databases in several servers, using the script and a CSV file to store parameters: run it doing backup in the sources, run it doing restores and processing (sync logins, etc) in destinations.

Regards from Jorge from Miami Nice

Reply
EM says:

February 9, 2016 at 7:06 pm

This script is great, as long as you don’t need to pass credentials. Can someone show an example where this might be adapted to pass credentials, and even load a partial modules in order to run commands?

Reply
Paul says:

February 4, 2016 at 5:02 pm

Adrian says: “Although this isn’t multithreading, it might be good enough for some things. Issues that I’ve resolved are: ”

My question is that is this multithreading after once the issues are fixed?
Thanks.

Reply
- Ryan Witschger says:
  
  September 1, 2017 at 3:55 pm
  
  I’ve gotten the reply that “this isn’t multi-threading” quite a few times. I’m really not sure why, it is literally spawning a separate thread. I guess it doesn’t match whatever nuanced definition that they are using, but to me it remains true multi-threading.
  
  Reply
ericnils says:

January 27, 2016 at 11:49 am

You’re my hero! I have a number of reporting scripts that poll thousands of machines over high latency connections and have been using single threading or jobs up till now. I can’t wait to convert them. The performance difference between this and the jobs is dramatic and you’ve made it so easy to use. I just ran a quick test performing WMI queries against 5 remote machines and the performance improvement of using this script vs a foreach loop, as measured by Measure-Command, was almost linear with the number of threads (just as you would hope for a multithreaded process). Thanks!

Reply
David says:

January 18, 2016 at 9:36 pm

Nifty stuff! Now that PS v5 is out and supports classes, maybe this can be simplified a lot.

Reply

Adrian says:

December 9, 2015 at 3:39 pm

This isn’t multithreading, this is multitasking. This spawns new powershell processes. I was trying to figure out why I couldn’t execute a function in my local .ps1 file. It’s because it doesn’t have access to the local space. To get around this, I tried to include the script name as the command. This works, but it causes a lot (in my case, too much) of overhead. 🙁

With 20 threads as the default, it started to die so badly. Cutting it back to 4 sort of worked, but it resulted in screwing up the powershell ISE instance making it so it won’t recognize any command, or rather think that any command is a command with no result.

Was actually excited when I saw this, but it is just a dead end.

Adrian says:

December 9, 2015 at 5:19 pm

Although this isn’t multithreading, it might be good enough for some things. Issues that I’ve resolved are:

1. $MaxThreads default value. Changed:

$MaxThreads = 20

1	$MaxThreads = 20

to:

$MaxThreads = ((get-counter "\Processor(*)\% idle time").countersamples).length-1

1	$MaxThreads = ((get-counter "\Processor(*)\% idle time").countersamples).length-1

This prevents overextending the CPU which will actually reduce performance.

2. Command matching. Changed:

If ($(Get-Command | Select-Object Name) -match $Command)

1	If ($(Get-Command \| Select-Object Name) -match $Command)

to:

If ((Get-Command | Select-Object Name | ForEach {$_.name}) -contains $Command)

1	If ((Get-Command \| Select-Object Name \| ForEach {$_.name}) -contains $Command)

which will not match a substring by accident.
3. Added code snippet availability. Changed:

    {
        $Code = $Null
    }Else{
        $OFS = "`r`n"
        $Code = [ScriptBlock]::Create($(Get-Content $Command))
        Remove-Variable OFS
    }

{

$Code = $Null

}Else{

$OFS = "`r`n"

$Code = [ScriptBlock]::Create($(Get-Content $Command))

Remove-Variable OFS

}

to:

    {
        $Code = $Null
    }
    ElseIf (!(Test-Path $Command))
    {
        $Code = $Command
    }    
    Else
    {
        $OFS = "`r`n"
        $Code = [ScriptBlock]::Create($(Get-Content $Command))
        Remove-Variable OFS
    }

{

$Code = $Null

}

ElseIf (!(Test-Path $Command))

{

$Code = $Command

}

Else

{

$OFS = "`r`n"

$Code = [ScriptBlock]::Create($(Get-Content $Command))

Remove-Variable OFS

}

In this way, a piece of code can be placed there. This is useful since you can’t state a function name, you can instead place the function in there. Smaller the code fragment the better the performance.

It is probably the size of the fragment which caused the powershell ISE to go wonky. When I gave it the name of the powershell script, it would dump the entire file into the code snippet (the last else shown in my 3rd fix) which might have caused some issues. It works fine now.

Other than that, this is somewhat useful. Good work! 🙂

Adrian says:

December 9, 2015 at 5:36 pm

A better fix for 2. then I stated above would be:

If ((Get-Command | Select-Object Name | where {$_.Name -eq $Command}))

1

If ((Get-Command | Select-Object Name | where {$_.Name -eq $Command}))

This prevents the conversion of the list to a string list which is better in terms of performance.

A

Reply
John H says:

September 1, 2017 at 12:13 pm

@Adrian

You offer some very nice suggestions, but if I might offer some advice, I would highly recommend checking your ego in these posts. “…might be good enough for some things…” and “…this is somewhat useful…” hardly describes the content of this article. This is a fantastic contribution to the PowerShell community, and Ryan has presented it in a much more professional manner than you have been able to craft in your reply. You might consider tucking your ego in your back pocket and just contributing in future posts. At the very least, try not to sound so insulting in your posts.

@Ryan,

From someone who has been running with the “Jobs” approach for years, thank you for this post! This is fantastic material. I only wish I had paid attention to it sooner.

Cheers,

John H

Reply

Ryan Witschger says:

September 1, 2017 at 3:57 pm

If you are seeing additional processes spawn, I think you are doing something wrong, but I make no guess as to what. You would see this from my older method which uses jobs.

Reply

Ninjigen says:

October 14, 2015 at 9:27 am

Hello,

Thanks for posting this code and explaining every step, it helped me a lot understanding how multi-threading works.

I have, though, encountered a problem when using your script invoking a powershell script existing within the $env:Path variable, since it could not detect it with the Get-Command cmdlet (inside the following if statement) If ($(Get-Command | Select-Object Name) -match $Command)
That’s why I instead instanciate a variable containing the result of (Get-Command $Command), then test if the Path property is null or not. If the Path property is not null, I can try to get the content of the file.

Reply
steve says:

August 21, 2015 at 3:08 pm

As another participant, started – Please forgive my stupidity, but after multiple tests and after reading this thread several times I started to understand that this nice script is not going to solve my problem. Is it helpful ONLY when a certain query should be run against many machines ?

My problem is different. I need to sort info about thousands of mailboxes. So my script queries a single machine (an Exchange server). Since the task takes about 30 minutes I hoped I can cut this time by multithreading. Is it true that this script is not going to help in my case ?

Reply
- Ryan Witschger says:
  
  August 21, 2015 at 8:24 pm
  
  You can run ANY script with ANY input. The script takes any input and run the specified script with the input as an argument.
  
  Reply
  - steve says:
    
    August 24, 2015 at 10:18 am
    
    Well, thank you for the answer, but when I run it with a script of mine (which btw, does not require any arguments) – just nothing happens. Instantly exits back to the console and that’s all. What am I doing wrong ?
    
    Reply
    - steve says:
      
      August 24, 2015 at 10:24 am
      
      Moreover I don’t understand the answer to Andrew (question #4) – you said he gets nothing because he was running “get-services” against nothing. Why nothing, why not getting the services of the current machine ?
      
      Reply
      - Brian McMahon says:
        
        September 28, 2015 at 4:09 pm
        
        The script he has provided requires that you supply -ObjectList (in some manner), which is a list of objects to evaluate the code you are running against. If you look at one of his examples and break it down it’s kind of clear (but I understand why you might get confused):
        
        # gc AllServers.txt | .\Run-CommandMultiThreaded.ps1 -Command .\ServerInfo.ps1
        # .\Run-CommandMultiThreaded.ps1 -Command .\ServerInfo.ps1 -ObjectList (gc .\AllServers.txt)
        
        Lets say AllServers.txt ended up with “computer1, computer2, serverA, serverB” in the txt file. In effect, this will execute:
        ServerInfo.ps1 computer1 … and return the result
        ServerInfo.ps1 computer2 … and return the result
        ServerInfo.ps1 computerA … and return the result
        ServerInfo.ps1 computerB … and return the result
        
        Take a look at his examples closely. ObjectList is absolutely required. If you do not supply ObjectList, the script literally sets itself up to do and return nothing. Look at the “Process” section: “ForEach ($Object in $ObjectList)”. If there is no ObjectList, the script is done.
        
        The author could have made that a tad bit more clear, but if you look closely, the information is there.
      - ap says:
        
        October 6, 2015 at 5:45 pm
        
        It seems like one would need to pipe at least one entry to the command or specify an input filename on $Objectlist. It’s not the same as running Get-Service at the prompt.
        
        $hostname = hostname
        $hostname | .\multithread1.ps1 -Command “Get-Service” -InputParam ComputerName
        
        This ^ ^ produces results. A command against mailboxes using the username should behave the same.
        
        Try with just one —
        $currentuser = “myName”
        $currentuser | .\multithread1.ps1 -Command “mailboxQuery” -InputParam userName
        
        Then if that works, you can use a file with $Objectlist.
    - ap says:
      
      October 6, 2015 at 5:31 pm
      
      I’m not sure how the output is supposed to be handled. I added | out-file filename.txt to the end of my command and got output.
      
      Possible that your input file and input parameter don’t match up for the command you’re using?
      
      Reply
Dennis Coonich says:

July 10, 2015 at 10:00 am

Thanks, many thanks!!!

Reply
Jeremy says:

July 6, 2015 at 12:23 am

This is really great, thank you! The performance improvement is wonderful. I’m all giddy thinking about the uses I have for this. One note, and I admit I didn’t dig into it very much, but I ran into an issue with escaping folder paths when specifying a script file for the Command parameter. Following this example gave me the error below:

.\Run-CommandMultiThreaded.ps1 -Command .\ServerInfo.ps1 -ObjectList (gc .\AllServers.txt)

parsing “X:\Folder 1\Subfolder1\Project-Name\ServerInfo.ps1” – Unrecognized escape sequence \F.
At X:\Folder 1\Subfolder1\Project-Name\Run-CommandMultiThreaded.ps1:93 char:11
+ If ($(Get-Command | Select-Object Name) -match $Command){
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException

To avoid the error I had to remove the .\ prefix from the Command parameter:

.\Run-CommandMultiThreaded.ps1 -Command ServerInfo.ps1 -ObjectList (gc .\AllServers.txt)

Reply
- MuffinsaurusRex says:
  
  August 10, 2015 at 2:13 am
  
  Hi,
  I was also getting a similar ‘Unrecognized escape sequence’.
  
  I ended up fixing this by changing line 96:
  From:
  If ($(Get-Command | Select-Object Name) -match $Command){
  to:
  If ($(Get-Command | Select-Object Name) -match ([regex]::Escape($Command))){
  
  Seems to have solved the problem.
  
  Reply
  - JohorB says:
    
    March 27, 2017 at 11:36 am
    
    In my case, I used \\ (double backslash) to get around that and it worked for me.
    
    Reply
Paul Mannino says:

June 29, 2015 at 1:23 pm

Can’t figure out how to define parameters when multiple parameters are required. This is my parameter definition:

param (
[string] $AWSAccount = $null,
[string] $AWSRegion = $null,
[string] $AWSService = $null
)

Need to run the child script 5*9*17 times.

Reply
- Chase says:
  
  July 23, 2015 at 1:09 pm
  
  -AddParam @{“AWSAccount” = “” ; “AWSRegion” = “” ; “AWSService” = “”}
  
  Reply
- JohorB says:
  
  March 27, 2017 at 11:44 am
  
  I customized the script to pass in a hash table which has various parameters defined and then use AddParameters. It worked for me this way.
  
  Param($Command = $(Read-Host “Enter the script file”),
  [Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]$ObjectList,
  $InputParam = $Null,
  $MaxThreads = 20,
  $SleepTimer = 200,
  $MaxResultTime = 7200,
  [HashTable]$AddParam = @{},
  [Array]$AddSwitch = @(),
  [HashTable]$ParameterList = @{}
  )
  .
  .
  .
  If ($ParameterList -ne $Null){
  $PowershellThread.AddParameters($ParameterList) | out-null
  }
  
  Reply
Chris says:

May 16, 2015 at 3:12 am

Forgive my stupidity but i’m extremely new to programming languages in general and Powershell is my first language… i can’t seem to figure out what i’m doing wrong..

i have a script where i’m querying the functionality of WMI on a bunch of remote machines via psexec (it does a buttload of other stuff as well but this is the bit i want to multithread..), code works fine when run sequentially it runs the command “winmgmt /verifyRepository” command on each machine.. but my GOD is it slow as shit… and so we come to this.. runspacepool type stuff which… is too complex for me to be honest but i was able to get Start-job to work but Start-job might as well be just as bloody slow as doing it sequentially cause of the overhead 🙁 …

so i have the WMI check written as a function called “Verify-WMI”… since i used to run it through a;

foreach ($Item in $Masterlist) {# do some stuff}

it has a variable $computername inside [and a few others] that have values of \\$Item and the like…

so i call it like so:

.\MultiThread\Run-CommandMultiThreaded.ps1 -Command “Verify-WMI” -InputParam computername -ObjectList $Masterlist

I’ve tried this as well but doesn’t work:

.\MultiThread\Run-CommandMultiThreaded.ps1 -Command “Verify-WMI” -InputParam item -ObjectList $Masterlist

somebody pls help 🙁 i have posted my code in pastebin if it helps… here: http://pastebin.com/hvUyFqrg

Line 2460 is where the explosions happen…

if anyone has the time please.. i understand things so much better if i have a working example… i have stolen a LOT of my code from the internet but i’m just lost with this bit…

any assistance MOST welcome.

Cheers,
Chris
[email protected]

Reply
Ajit says:

April 27, 2015 at 8:45 pm

Ryan,
Thank you for making this awesome tool . i have question if i save some command is ps1 file and give the path to the script i get lot of errors what i am doing wrong basically this is the command
i have small question for you first i tired this it didn’t work i put below commandlet in ps1 file
$vms =Get-triVM -Role dfs |select -ExpandProperty name
foreach ($vm in $vms){
$os =Get-WmiObject win32_operatingsystem -ComputerName $vm |select -ExpandProperty caption
[pscustomobject] @{computer =$vm ;os=$os}
}

i get error

xception calling “EndInvoke” with “1” argument(s): “The term ‘.\get-ostype.ps1’ is not recognized as the name of a
mdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify
hat the path is correct and try again.”
t D:\UserData\v-ajmaur\Documents\multithread.ps1:151 char:13
$Job.Thread.EndInvoke($Job.Handle)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : CommandNotFoundException

then i jut tried to put this in ps1 file
Get-WmiObject win32_operatingsystem -ComputerName $_ |select -Property caption,PSComputerName

still get the same error what i amdoing wrong

Reply
- Ryan Witschger says:
  
  December 13, 2019 at 8:36 am
  
  Simply remove the “.\” from “.\get-os.ps1”
  
  Reply
Kaleb says:

March 24, 2015 at 8:02 pm

This is exactly what I was looking for. Very well explained. Thank you fo your post and the time it took you to write this. Now that I see the code and understand the concept I can Implement it into my own scripts.

Reply
Scott says:

March 16, 2015 at 1:26 am

I’ve got a script that’s downloading XML data via curl.exe. So, it’s only running on a single server, but I want to multithread it for better performance.

My invocation is:

E:\Powershell\Scripts\Run-CommandMultiThreaded.ps1 -ObjectList $array
-Command E:\Powershell\Scripts\foo.ps1 -InputParam PageStart

where PageStart is an element from the ObjectList array.

However, I’m getting this error:

parsing “E:\Powershell\Scripts\foo.ps1” – Malformed \p{X} character escape.
At E:\Powershell\Scripts\Run-CommandMultiThreaded.ps1:96 char:11
+ If ($(Get-Command | Select-Object Name) -match $Command){
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException

It’s not liking the full path to the child Powershell script.

However, the workaround is specifying the child Powershell script as either:

E:/Powershell/Scripts/foo.ps1

or E:\\Powershell\\Scripts\\foo.ps1

So, I’ve got my workaround. But, if this can be handled within Run-CommandMultiThreaded.ps1, so much the better.

Thanks for the great work, and I hope this helps someone else who finds this webpage.

Reply
- Ryan Witschger says:
  
  December 13, 2019 at 8:38 am
  
  Good find, I store all my scripts in a single folder so I hadn’t run into this. I’ll post your response as the workaround, and if someone wants to modify to suite that would be great.
  
  Reply
Corey Thomas says:

February 26, 2015 at 7:54 am

This is fantastic code. Well done. The speed is amazing. The only change I would like to see is to wrap this so I can simply import the script and call it like a function or object.

As it is now, I have to use some path to get it which adds a few more steps to my code.

For example, if I am querying SQL, my current path is something like PS SQLSERVER:\> So if I pipe to .\.ps1, it will fail. I have to add more code to exit out of that ps drive path. It’s more of a nuisance than anything else.

Would just be easier as a function I can import. 🙂
Either way, thanks for this awesome script.

Reply
- Adrian says:
  
  December 9, 2015 at 5:35 pm
  
  @Corey,
  
  It is possible to wrap it yourself by putting
  
  function fn-name { # insert all code above here }
  
  1
  2
  3
  
  function fn-name {
  # insert all code above here
  }
  
  However, this can result in a inadvertent match to a user defined function in your .ps1 file while processing the Begin block. What that block does is see if there is a function with that name available. Since this is in a separate file, it will find all of the defaults available. If you put it in your .ps1 file, it will find those in your file.
  
  Reply
Bradley Gray says:

January 12, 2015 at 8:30 am

I’ve noticed that on line 123:

ForEach($Switch in $AddSwitch){ >>> $Switch $PowershellThread.AddParameter($Switch) | out-null }

You output $Switch directly. This was really badly breaking my script and it took forever to figure out why 😉

Great stuff otherwise, I’ve converted it to a cmdlet and we’re using it to parallelise all kinds of tasks, typically slow wmi queries or other remote commands.

Reply
Max says:

November 26, 2014 at 3:15 pm

Hi,
I really like the idea and want to use it but I cant figure out hot to make this run my own script without getting an error about not supporting input from pipeline.
you have any example of a script and how it should look for this script to support running it?

Reply
Laurence says:

November 5, 2014 at 5:20 am

Hi – this work is really impressive. I need to pass a Custom Object as a parameter to the multithreaded operation. Do you have any examples on how to do this? Every time I pass it the object is converted to a string and loses all its properties and methods.

Hope you can enlighten me!

Reply
Andrew Morpeth says:

November 4, 2014 at 2:16 am

Hey, nice work this looks great! When I run a command e.g. .\Run-CommandMultiThreaded.ps1 -Command “Get-Service” I get nothing returned to the console (just a blank line). Is there something obvious I am missing? How do I get the results or check the status of the job?

Much appreciated

Andrew

Reply
- Ryan Witschger says:
  
  July 14, 2015 at 3:46 pm
  
  You aren’t passing any objects for it to run against. You are basically looping through nothing. If you wanted to loop through a bunch of remote systems you could make a text file with the names and provide that to the script.
  
  Reply
objectis says:

September 24, 2014 at 9:23 pm

Thanks, looks like a very promising script. One major problem I ran into though. My existing script uses gc to get the list of servers from a text file, and process some actions serially. Normally, there are 200 servers on it. However, while I tried to use your script with mine, I found that the commands were constantly looping through the same first 10 servers. For example, I am also doing gc Input.txt, which is saying there are 200 jobs, and my internal script is also using this file to perform some actions on it. Any thoughts will be greatly appreciated. I am assuming I am doing something wrong based on what others have posted on your comments section.

Many thanks once again.

Reply
- Ryan Witschger says:
  
  December 13, 2019 at 8:45 am
  
  It sounds like your child script is doing the “gc” command while this needs to be done either by the multi-threading script or in the pipeline and not by the child. Your child script should be set to configure only one item at a time.
  
  Reply
Brian says:

July 1, 2014 at 7:31 am

Wow. Insane performance increase in massive WMI calls. I’m talking 4000 servers with 10 calls each in 2 minutes!

Reply
- Ryan Witschger says:
  
  July 3, 2014 at 4:51 pm
  
  Yea, I am really glad I got this polished enough to release. As an admin, this type of speed in multithreading definitely makes your life easier!
  
  Reply
Justin says:

June 24, 2014 at 9:09 pm

I did something similar http://gallery.technet.microsoft.com/scriptcenter/Invoke-Async-Allows-you-to-83b0c9f0

Reply
- Jana Sattainathan says:
  
  August 31, 2016 at 1:35 pm
  
  Nice article. I have not yet tried the function in this post. However, I use Invoke-Async. To say that I use Invoke-Async “a lot” is really an understatement. I love it and cannot live without it. Very stable and clean.
  
  Here is a small blog post on your function:
  https://sqljana.wordpress.com/2015/08/18/invoke-async-asynchronous-parallel-processing-for-any-workload-in-powershell/
  
  Reply
  - Ryan Witschger says:
    
    September 1, 2017 at 3:49 pm
    
    Thanks, my readers will enjoy your work.
    
    Reply
Boe Prox says:

June 24, 2014 at 7:44 pm

Neat stuff! You should post this out on the Technet Script Repository (http://gallery.technet.microsoft.com/scriptcenter). One thing that you could look at doing is instead of enumerating through the $AddParam hash table and using AddParameter(), just put the entire hash table in using AddParameters() as it takes a hash table. On a side note (as well as plugging my own thing :)), I did a talk on runspaces at the NorCal PowerShell User Group recently (http://learn-powershell.net/2014/06/11/norcal-powershell-user-group-presentation-on-runspaces-is-available/), the audio isn’t the best but I have a bunch of code demos as well as the slide deck posted. Keep up the awesome work!

Reply

Get-Blog : Ryan’s PowerShell Blog

True Multithreading in PowerShell

77 Responses to “True Multithreading in PowerShell”

Leave a Reply Cancel reply