Class.MTTaskList.AKA-MTAnything

<#
#######################################   QUICK COMMAND EXAMPLES   #######################################

[MTTaskList]::new($ParameterList, $ScriptBlock, 20).Finalise().Results()
    Creates 20 process threads and runs the script using the list of parameters, waits, closes, returns results.

$Work = [MTTaskList]::new($ParameterList, $ScriptBlock, 20)
    Creates 20 process threads and runs the script using the list of parameters. Leaves runnning in the background and
     open to adding more tasks. No variables necessary. By default the number of threads is based on your CPU cores.
     New Parameter(s) can be add with AddTask or AddTaskOverride. A scriptblock here is used as the default to run,
     BUT if the scriptblock is not provided, all new tasks MUST be added with AddTaskOverride.
$Work.AddTask($NewParameters)
    Add new task using the earlier scriptblock and new parameter(s)
    Parameters can be a string like "WSxxxxx020", a hashtable like @{Name="WSxxxxx020"}, or an array/list of those.
$Work.AddTaskOverride($NewParameters, $NewScriptBlock)
    Adds a new task, with a new scriptblock. Typically useful if this TaskList is purely for running many random tasks quickly
    Or for applying modified scriptblocks instead of using parameters.
$Work.Wait().CollectResults()
    Waits for current tasks to finish, collects output for the tasks that are complete and haven't been collected previously.
    Outputs as ThreadTask objects with these properties:
        .Parameters: contain the Parameters of the task
        .Scriptblock: the scriptblock used
        .Result: result output from the scriptblock. only applied when CollectReslts or Finalise are run.
        .Instance: Each tasks has an Instance number starting from 1, to easily find tasks in the order they were assigned.
    Leaves TaskList open to adding more work.
$Work.Close()
    Gets results of completed work so far and closes the TaskList.
$Work.Dispose()
    Force closes the TaskList.
$Work.Finalise()
    Waits for tasks to complete, with a progress bar, then retrieves and stores all results.
    Closes the TaskList so it can't take any new tasks.
$Work.Results()
    Outputs the .Result value of all tasks, whether they've been collected or not. Should only be used when all work
     is complete. Outputs as array of objects, 1 per task, of output from the task's scriptblock.
     Typically that object will be an array of strings, unless your scriptblock outputs another object type.
$Work.Result(1)
    Outputs the .Result value of a specific task based on it's Instance index. (the order the task was added)
    Used for quick reference of work once it's completed and collected.

TaskList can bloat resources if not closed properly.
Close, Dispose, or Finalise will properly close the TaskList processes.
Closing your terminal window will also flush any TaskList you may have left running.

#############################################   HOW TO USE   #############################################

####################   Main Guide   ####################

#############  Including the Class  #############

# Copy the Class info below into your console session, or the top of your scriptfile.
# - When using in a script file: MUST be after Params() and before you use it.
# - To declutter your script, you can include this file in the same folder and use the following line.
    . "$PSScriptRoot\Class.MTTaskList.ps1"
# or
    . "Literal:Path\to\File\Class.MTTaskList.ps1"


#############  Preparing your ScriptBlock and Parameters  #############

# Create a ScriptBlock to be reused. Include parameters that you will provide values for later.
# If Parameters are new to you, it's very easy to make a simple set. Parameters are variables set in a "param()" block
#   at the start of a script or scriptblock.
# They can get very advanced, with type validation, positions, sets, Required properties, and more. Research them if you're interested.
# For a basic setup, see below.
$Scriptblock = {     #Scriptblocks are actually [string] objects, basically plain text, but must be wrapped in curly braces: {}
param($var)          #This sets parameter variables that will receive any values to change between runs. Comma separated.
Start-sleep $var     # Do a thing with the given value.
Write-output "Slept for $var seconds"     # This output will be collected later.
}

# Generate a list of hashtables that contain parameters for each task.
$worklist = For ($i = 0; $i -lt 100; $i++){
@{var=$i}   # This is a hashtable, with the name on the left being the name of the parameter it will be assigned to in the ScriptBlock
            # Parameter names MUST match. The value to provide for processing is on the right. Multiple entries are separated by semicolon: ;
}

# You can now use a simple array of parameters.
#   For example: a Ping script, and the WorkList is an array of targets. @("Computer1","Computer2")
# An array of Arrays is NOT SUPPORTED at this time.

# TEST your script and hashtable design by running it like below. Use only 1 hashtable of parameters to test.
# This method is called "Splatting" and is the basis of easily applying data to a reuseable script.
$hashParameters = @{varA="fu";varB="bar"}
.$Scriptblock @hashParameters


#############  Starting work  #############

# The TaskList goes through 5 steps: Initialisation, Adding Work, Complete Work, Collect Results, Close TaskList.
# Here we will Initialise and Add Work.
# Use the [MTTaskList] class to create a tasklist that will process your tasks.
$workload = [MTTaskList]::new($Scriptblock, 5)
# Provide the Scriptblock and the MaxThreads value (Maximum amount of processes you want to run at a time).

$workload = [MTTaskList]::new($worklist, $Scriptblock, 5)
# You can provide the list of task variables and they will be added as tasks when creating the list.
# You can leave out the MaxThreads and it will use the number of processors instead.

# You can add more tasks later as long as the list hasn't been closed. The original scriptblock is the default script used.
$workload.AddTask($worklist)

# You can also add tasks with a new scriptblock override. This is not encouraged, but allows for a general processing pool.
$workload.AddTaskOverride($worklist, $ScriptBlock)

# If working in a console session, you can track progress by running this:
$workload.Progress()


#############  Retrieving Results  #############

# This is the second half of the process, but a big half. How to retrieve and handle the output of your scriptblock.
# How you handle it is up to you, but let's start with retrieving it.

$workload.Tasks
# Returns all Task objects. These objects contain 4 useful properties:
#  "Instance"   : Unique task ID, in the order it was added, starting at 0. Usable reference when correlating results with submission order.
#  "Parameters" : The hashtable or collection of parameters originally supplied.
#  "Results"    : The output of the script. This is empty until work is complete and ".CollectResults()" or ".Wait()" are run.
#  "Scriptblock": Hidden value - The script used. Useful when checking what was run, especially when AddTaskOverride is used.

$workload.IsCompleted()
# Returns true/false if all tasks have completed running.

$workload.IsCollected()
# Returns true/false if all tasks have collected output into their Result property.
# IMPORTANT! Result Collection is not automatic. It is triggered during ".Finalise()" or by ".CollectResults()".

$workload.CollectResults()
# Collects results for any completed tasks that haven't already been collected, copying the Output of each task to it's
#  respective .Tasks.Result property. Each task can only be collected once.
# This will output Task objects from "$workload.Tasks" but filtered for just the current collection round.
# It is best to capture this in a variable or pipe it straight into your output handler.

$workload.Wait()
# Halts the terminal and waits for all supplied tasks to complete. Shows progress bar.
# IMPORTANT! When running this in script, please note: THIS CANNOT BE CANCELLED

$workload.ResultByIndex($Number)
# This returns the Result property of the Task at the provided Instance number.
# This is an alternative to "$workload.Tasks[$Number].Result", this should be more convenient and easier to understand.
# NOTE: Instances start at "0" to match the Array[Row] format so you can ForEach through your own array and easily reference the same object.
# NOTE: Result value is an empty placeholder until .CollectResults(), .Close(), or .Finalise() is run.

$workload.Results()
# Is similar to "$workload.Tasks.Result", returning the Result property of all tasks.
# BUT each array item is a completely separate result, and can be processed per item.
# $workload.Tasks.Result will merge all results into one array. This will keep them separate items.
# Mainly used to conveniently get output after .Finalise() is run on terminal session tasks.
# In all other cases, you should access results by:
#   $workload.Tasks[$Number].Result
#   $workload.ResultByIndex($Number)
#   $workload.CollectResults() | ForEach-Object {$_.Result}
# NOTE: Result value is an empty placeholder until .CollectResults(), .Close(), or .Finalise() is run.


# To handle output data as it is completed, make a loop similar to the one below.
# This is good for logging progress or saving changes in case of script interruption.
# When Multithreading, there should still be a management process that handles all tasks, and that is your script.
While ($workload.IsCollected() -eq $false) {
    $workload.CollectResults() |  Foreach-object {
        #Do something with $_.Result
        #Check $_.Parameters.varA for value, etc.
    }
    start-sleep 1
}


#############  Cleanup  #############

# Always close the TaskList when you are finished with it. This doesn't erase the list, just closes the background processing of tasks.
# No new tasks can be added. Do this when you are definitely not going to add any more work.

$workload.Close()
$ Collects results for tasks currently completed, and closes the TaskList.
# This will collect processed results before closing the list as a matter of cleanup.
# It WILL NOT wait for the tasks to finish and will forcibly close the list.

$workload.Dispose()
# Force closes the processing pool without waiting or collecting results.

$workload.Finalise()
# This will wait until all work is complete, copies the task results, and closes the TaskList.
# This is good for simple, or single phase scripts where you want to use multithreading for better performance,
#  or for terminal use to quickly run a batch of tasks and wait for the output.
# IMPORTANT! When running this in script, please note: THIS CANNOT BE CANCELLED


####################   Basic Example 1   ####################

# Include the Class block
Class MTTaskList {}

# Creat ScriptBlock
$Scriptblock = {...}

# Setup the required Task pool
$workload = [MTTaskList]::new($Scriptblock, 6)

# Add the work. This will involve collecting data and organising into Parameter Sets for the script
. blah blah blah get data
$worklist = $AListOfComputerOrOtherStuff | ForEach-Object {
    @{Parameter1="Hello";Computer="$_";Parameter3="World"}
}
$workload.AddTask($worklist)

# Create a main loop that will handle post-processing of work.
While ($workload.IsCollected() -eq $false) {
    $workload.CollectResults() |  Foreach-object {
        # Do Some stuff
        $_.Result | Where {# Something you need}
        # This is the place where you decide what to do with what comes out of the scriptblock.
        # Was it successful? Do you need to remove it from a ToDo list?
    }

    # It's good to wait a bit before looping to process more work, or this loop will use 100% of the CPU.
    start-sleep 1
}

# Close the TaskList when you're finished with it
$workload.Close()

# Do any other end-of-script work you may need
........



####################   Advanced example 2   ####################

# You can setup as many Task pools as you want. But keep in mind they all share the same computer resources.
# The main reason I do this, and one of the big benefits to a multiphase script, is when you separate the task into
#   different Scriptblocks that have different loads that you can plan for.
# I have a network file copy SB where 10 threads is about the best performance before bottlenecking the network interface,
#   and then remotely executing that file is just waiting for the other end to finish. Due to install time and other factors,
#   it is efficient ro run the install commands separately instead of waiting for a filecopy thread to be free.
#   And since that can take a long time, but little processing, I give it more threads.

# Include the Class block
Class MTTaskList {}

# Creat ScriptBlocks
$SB1 = {...}
$SB2 = {...}

# Setup the required Task pools
$workload1 = [MTTaskList]::new($SB1, 6)
$workload2 = [MTTaskList]::new($SB2, 12)

# Add the work. This will involve collecting data and organising into Parameter Sets for the scripts
. blah blah blah get data
$worklist = $AListOfComputerOrOtherStuff | ForEach-Object {
    @{Parameter1="Hello";Computer="$_";Parameter3="World"}
}
$workload1.AddTask($worklist)


# Create a main loop that will handle task migration from the first phase to the second phase, and
#   also if you want to log notes or save progress, etc, you'll want that post-processing to happen here.
While (($workload1.IsCollected() -eq $false) -or ($workload2.IsCollected() -eq $false)) {
    $workload1.CollectResults() |  Foreach-object {
        # Do Some stuff
        # Find returned values that the next phase depends on.
        $_.Result | Where {# Something you need}
        # Construct the hashtable to pass to phase 2
        $NewParameters = $_.Parameters
        $NewParameters.NewParameter = # whatever
        # If, for whatever reason, you want to carry the whole output, you can do like this, just make sure SB2 has a parameter for it:
        $NewParameters.SB1Output = $_.Result
        # Add as a new task to the Phase 2 task pool
        $workload2.AddTask($NewParameters)
    }
    $workload2.CollectResults() |  Foreach-object {
        # Do Some stuff
        $_.Result | Where {# Something you need}
        # This is the place where you decide what to do with what comes out of the second scriptblock.
        # Was it successful? Do you need to remove it from a ToDo list?
    }

    # It's good to wait a bit before looping to process more work, or this loop will run 100% of the CPU.
    start-sleep 1
}


# Close the TaskLists when you're finished with them
$workload1.Close()
$workload2.Close()

# Do any other end-of-script work you may need
........

#>








Class MTTaskList {
    [System.Collections.Generic.List[psobject]]                 $Tasks
    [string]                                                    $ScriptBlock
    [int32]                                                     $MaxThreads
    hidden [System.Management.Automation.Runspaces.RunspacePool]$RunspacePool

    # Constructors ---------------------------------------------------------------------
    MTTaskList() {
        $this.MaxThreads = $env:number_of_processors
        $this.Init()
    }

    MTTaskList([int32]$MaxThreads) {
        $this.MaxThreads = $MaxThreads
        $this.Init()
    }

    MTTaskList([string]$ScriptBlock) {
        $this.MaxThreads = $env:number_of_processors
        $this.ScriptBlock = $ScriptBlock
        $this.Init()
    }

    MTTaskList([string]$ScriptBlock, [int32]$MaxThreads) {
        $this.MaxThreads = $MaxThreads
        $this.ScriptBlock = $ScriptBlock
        $this.Init()
    }

    MTTaskList([hashtable[]]$ParameterSets, [string]$ScriptBlock) {
        $this.MaxThreads = $env:number_of_processors
        $this.ScriptBlock = $ScriptBlock
        $this.Init()
        $this.AddTask($ParameterSets)
    }

    MTTaskList([hashtable[]]$ParameterSets, [string]$ScriptBlock, [int32]$MaxThreads) {
        $this.MaxThreads = $MaxThreads
        $this.ScriptBlock = $ScriptBlock
        $this.Init()
        $this.AddTask($ParameterSets)
    }

    MTTaskList([string[]]$ParameterSets, [string]$ScriptBlock) {
        $this.MaxThreads = $env:number_of_processors
        $this.ScriptBlock = $ScriptBlock
        $this.Init()
        $this.AddTask($ParameterSets)
    }

    MTTaskList([string[]]$ParameterSets, [string]$ScriptBlock, [int32]$MaxThreads) {
        $this.MaxThreads = $MaxThreads
        $this.ScriptBlock = $ScriptBlock
        $this.Init()
        $this.AddTask($ParameterSets)
    }

    # Methods --------------------------------------------------------------------------
    # *** Shared Initialiser Method ***
    hidden Init() {
        # Initialiser: creates a RunspacePool to be used and creates empty List for Tasks
        # Generating the list later causes problems.
        $this.RunspacePool = [RunspaceFactory]::CreateRunspacePool(1, $this.MaxThreads)
        $this.RunspacePool.Open()
        $this.Tasks = [System.Collections.Generic.List[psobject]]::new()
    }

    # Method for adding new ThreadTask objects with the default scriptblock
    [MTTaskList] AddTask($ParameterSets) {
        If ($this.ScriptBlock) {
            $this.AddTaskOverride($ParameterSets, $this.ScriptBlock)
        } Else {Throw "No default scriptblock, and none provided"}
        return $this
    }

    # Method for adding a new Task with a specific ScriptBlock
    [MTTaskList] AddTaskOverride([string]$ScriptBlock) {
        $this.AddTaskOverride(@{}, $ScriptBlock)
        return $this
    }

    # Method for adding a new Task with a ScriptBlock override, and String parameters
    [MTTaskList] AddTaskOverride([System.Collections.Generic.List[string]]$ParameterSets, [string]$ScriptBlock) {
        ForEach ($parameterSet in $ParameterSets) {
            # Creates a nested scriptblock so that plain string parameters can be applied with "-ArgumentList"
            $actioning = $this.AddTaskOverride(@{}, "Invoke-Command -Scriptblock {$($ScriptBlock)} -ArgumentList $parameterSet")
            # Modifies the Parameters record
            $actioning.Tasks[-1].Parameters = $parameterSet
        }
        return $this
    }

    # AddTask method with a ScriptBlock Override, used as the primary Adding method, all others are Aliases
    [MTTaskList] AddTaskOverride([System.Collections.Hashtable[]]$ParameterSets, [string]$ScriptBlock) {
        If ($this.RunspacePool.IsDisposed -eq $false) {
            ForEach ($parameterSet in $ParameterSets) {
                $instanceNumber = If ($this.Tasks){$this.Tasks.Count}Else{0}
                ###  This section generates an object like a class without requiring a separate class  ###
                # First run only: Adds 'ThreadTask' Class info to TypeData for current session/scope only
                # This way a separate class isn't needed. Specifically needed to hide some properties.
                If (-not (get-typedata ThreadTask)) {
                    $typeData = @{
                        TypeName = 'ThreadTask'
                        DefaultDisplayPropertySet = 'Instance','Parameters','Result'
                    }
                    Update-TypeData @TypeData
                }
                $newTask = [PSCustomObject]@{
                    PSTypeName = 'ThreadTask'
                    Instance = $instanceNumber
                    Parameters = $parameterSet
                    ScriptBlock = $ScriptBlock
                    Result = @()
                    Shell = [PowerShell]::Create().AddScript($Scriptblock).AddParameters($parameterSet)
                }
                $newTask.Shell.RunspacePool = $this.RunspacePool
                $newTask | Add-Member -MemberType NoteProperty -Name Status -Value $newTask.Shell.BeginInvoke()
                ###  End of Object creation  ###
                $this.Tasks.Add($newTask)
            }
        } Else {throw "This TaskList has been closed and cannot accept any more tasks."}
        return $this
    }

    # Alias Method for collecting results. Used as primary collector method.
    [array] CollectResults() {
        If (-not $this.Iscollected()) {
            If (-not $this.IsCompleted()) {
                return $this.CollectNewResults()
            } Else {
                return $this.CollectAllResults()
            }
        } Else {return @()}
    }

    [array] CollectNewResults() {
        If ($this.RunspacePool.IsDisposed -eq $false) {
            Try {
                $completedTasks = $this.Tasks | Where-Object { $_.Status.IsCompleted -eq $true }
                foreach ($completedTask in $completedTasks) {
                    $completedTask.Result = $completedTask.Shell.EndInvoke($completedTask.Status)
                    $completedTask.Status = $null
                    $completedTask.Shell.Dispose()
                }
                return $completedTasks
            }
            Catch {$this.Dispose();throw "An error was received while retrieving your results. Your MTTaskList has been closed.`nPlease review your parameters and scriptblock, and try again.`n$($error[0])"}
        } Else {throw "This TaskList has been closed. Cannot collect results."}
    }

    [array] CollectAllResults() {
        $totalCount = ($this.Tasks).count
        # Collection loop, needed because some work gets stuck in completed and can take a while to collect.
        $collectedResults = @()
        while (-not $this.IsCollected()) { #Loading bar to show collection progress
            $collectedCount = ($this.Tasks | Where-Object Status -eq $null).count
            [int]$taskProgress = $collectedCount / $totalCount * 100
            Write-Progress -Activity "Collecting results for all Tasks" -Status "$taskProgress% Complete:" -PercentComplete $taskProgress
            $collectedResults += $this.CollectNewResults()
            Start-Sleep -Milliseconds 25
        }
        return $collectedResults
    }

    # Method to Wait until all new results are complete.
    [MTTaskList] Wait() {
        $totalCount = ($this.Tasks).count
        $finishedCount = 0
        while (-not $this.IsCompleted()) { #Loading bar to show processing progress
            $finishedCount = $totalCount - ($this.Tasks.Status | Where-Object IsCompleted -eq $false).count
            [int]$taskProgress = $finishedCount / $totalCount * 100
            Write-Progress -Activity "Waiting for all Tasks to complete" -Status "$taskProgress% Complete:" -PercentComplete $taskProgress
            Start-Sleep -Milliseconds 25
        }
        return $this
    }

    # Method to collect results and close the Runspace Pool
    [void] Close() {
        [void] $this.CollectResults()
        $this.Dispose()
    }

    # Method to forcibly close the Runspace Pool
    [void] Dispose() {
        $this.RunspacePool.Close()
        $this.RunspacePool.Dispose()
        $this.Tasks | ForEach-Object {$_.Shell.Dispose()}
    }

    # Aggregate Method to wait until all work is complete, collect all results, and dispose the processing pool.
    [MTTaskList] Finalise() {
        [void] $this.Wait()
        [void] $this.CollectAllResults()
        $this.Dispose()
        return $this
    }

    # Method to report if all work is completed or not.
    [bool] IsCompleted() {
        If ($this.Tasks.Status.IsCompleted -eq $false) {return $false} Else {return $true}
    }

    # Method to report if all work is collected or not. '.IsCompleted' is a sub-property that is removed when work is collected.
    [bool] IsCollected() {
        If ($this.Tasks.Status -ne $null) {return $false} Else {return $true}
    }

    # Method to give a progress report
    [array] Progress() {
        $totalCount = ($this.Tasks).count
        If ($totalCount -gt 0) {
            $unfinishedCount = ($this.Tasks.Status | Where-Object IsCompleted -eq $false).count
            $queuedCount = $unfinishedCount - $this.MaxThreads
            $collectedCount = ($this.Tasks | Where-Object Status -eq $null).count
            $output = @()
            $output += "Queued:    " + $(If ($queuedCount -gt 0) {$queuedCount} Else {"0"})
            $output += "Running:   " + $(If ($unfinishedCount -gt $this.MaxThreads) {$this.MaxThreads} Else {$unfinishedCount})
            $output += "Completed: " + ($totalCount - $unfinishedCount)
            $output += "Progress:  " + (($totalCount - $unfinishedCount)/$totalCount).tostring("P")
            $output += "Collected: " + $collectedCount
            return $output
        } Else {
            return "No work has been assigned yet"
        }
    }

    # Collection method to return all results
    [array] Results() {
        $collatedResults = [System.Collections.Generic.List[psobject]]::new()
        $this.Tasks | ForEach-Object {$collatedResults.Add($_.Result)}
        return $collatedResults
    }

    # Collection method to return session Result based on Instance Index
    [array] Result([int32]$Number) {
        return ($this.Tasks | Where-Object Instance -eq $Number).Result
    }
}