PowerShell V4 – PipelineVariable Common Parameter

The big new feature in Windows PowerShell 4.0 is Desired State Configuration however there are a number of other minor features one of which is a new common parameter called PipelineVariable.  This parameter allows you to store the current pipeline object into the variable specified by the name you provide.  This capability comes in handy when, in the process of executing commands in the pipeline, information is lost due to transformations on the objects flowing down the pipeline.  For example, take the following case:

PS> Get-ChildItem *.ps1 | Select-String function | Get-Member


   TypeName: Microsoft.PowerShell.Commands.MatchInfo

Name         MemberType Definition
----         ---------- ----------
Equals       Method     bool Equals(System.Object obj)
GetHashCode  Method     int GetHashCode()
GetType      Method     type GetType()
RelativePath Method     string RelativePath(string directory)
ToString     Method     string ToString(), string ToString(string directory)
Context      Property   Microsoft.PowerShell.Commands.MatchInfoContext Context {get;set;}
Filename     Property   string Filename {get;}
IgnoreCase   Property   bool IgnoreCase {get;set;}
Line         Property   string Line {get;set;}
LineNumber   Property   int LineNumber {get;set;}
Matches      Property   System.Text.RegularExpressions.Match[] Matches {get;set;}
Path         Property   string Path {get;set;}
Pattern      Property   string Pattern {get;set;}

Note in this output that the FileInfo objects output by Get-ChildItem got replaced with MatchInfo objects in the pipeline.  Most of time this is perfectly fine except when you’d like to access some information that was available on a pipeline object in a preceding stage of the pipeline. 

An example of this kind of scenario comes from a question on StackOverflow, the OP wanted to know how to print out the filename and just the immediate parent directory’s name e.g. output “Scripts\foo.ps1” instead of “C:\Users\Keith\Scripts\foo.ps1”.  This can be accomplished with Split-Path and Join-Path manipulation of the MatchInfo’s Path property e.g.:

Get-ChildItem *.ps1 | Select-String function | 
    Foreach {"$(Split-Path (Split-Path $_ -Parent) -Leaf)\$($_.Filename)"}

 

This works just fine but being a clever PowerShell user you happen to know that the FileInfo objects output by Get-ChildItem have a convenient property called “Directory.BaseName” that would be perfect in this scenario.  Unfortunately you quickly realize that after Select-String you no longer have access to that FileInfo object.  Select-String outputs a new type of object called Microsoft.PowerShell.Commands.MatchInfo. You could make the FileInfo object available via another variable ($fi) using the Foreach command as shown below:

Get-ChildItem *.ps1 | Foreach {$fi = $_; $_} |
    Select-String function | 
    Foreach {"$($fi.Directory.BaseName)\$($_.Filename)"}

 

The new PipelineVariable allows you to eliminate the extra Foreach command.  The example below effectively does the same thing as the previous example:

Get-ChildItem *.ps1 -PipelineVariable fi | Select-String function | 
    Foreach {"$($fi.Directory.BaseName)\$($_.Filename)"}

 

There are a few things to note about the PipelineVariable parameter.  First, it has a conveniently short alias “pv” just like the other common parameters tend to have e.g. “ea” for ErrorAction, “ov” for OutputVariable, “wv” for WarningVariable.  Second, you do not specify a “variable” as the value for a PipelineVariable parameter. Instead you specify the name of a variable.  In other words, do not do this: “-PipelineVariable $fi” unless you really want to programmatically provide the name of another variable in which to store the pipeline object.  Most of the time you will not specify the “$” but just the name of a variable e.g. “-PipelineVariable fi”.  Third, using the PipelineVariable parameter works great as long as all the commands in the pipeline between the storage and the use of the PipelineVariable are streaming commands.  If you use Sort-Object or Group-Object in between, then the object your pipeline variable refers to will not be correct because the way these commands work.  Commands like Sort-Object and Group-Object aggregate all pipeline input, process it and then start a new flow of objects down the pipeline. 

Here’s a simple example that demonstrates the problem. We have the following filenames and sizes:

PS> ls *.txt
…
Mode           LastWriteTime       Length Name
----           -------------       ------ ----
-a---     7/19/2013 12:16 AM     20000006 Large.txt
-a---     7/19/2013 12:16 AM       200006 Medium.txt
-a---     7/19/2013 12:16 AM         2006 Small.txt
-a---     7/19/2013 12:16 AM           26 VerySmall.txt

 

Let’s try to sort the files based on length while using the PipelineVariable parameter.  Yes I know, completely contrived but it demonstrates the point.

PS> Get-ChildItem *.txt -pv fi | Sort-Object Length |
>>     Foreach {"$($_.Name) size is $($fi.Length)"}
>>
VerySmall.txt size is 26
Small.txt size is 26
Medium.txt size is 26
Large.txt size is 26

 

This is obviously not correct.  What happens is that the Foreach scriptblock doesn’t execute until Get-ChildItem has output *ALL* its objects.  That’s because the Sort-Object command buffers up all its input before it sends any output down the pipeline.  So by the time the Foreach begins to execute, the pipeline variable $fi will always contain the last object output by Get-ChildItem.  If you think about it this makes sense.  Sort-Object can’t send any “sorted” output down the pipeline until it sees all its input.  The issue happens with any non-streaming command which also includes Group-Object and Measure-Object.

Finally, one last example where this feature comes in handy.  This has to do with using Select –ExpandProperty which is a great way to walk down collections of collections.  The issue I’ve run into doing this is sometimes I want to preserve a property from the collection’s parent object but can’t because there is an identically named property on the items in the collection.  Here’s an example:

PS> Get-Command -Type Cmdlet | Select Name -ExpandProperty ParameterSets
Select : Property cannot be processed because property "Name" already exists.

 

In other words, I can’t include the Name from the CmdletInfo object because it would conflict with the ParameterSet.Name property. This means you have to use the Foreach trick I showed earlier:

Get-Command -Type Cmdlet | Foreach {$cmd=$_;$_} | Select -ExpandProperty ParameterSets

 

But if you then go on to expand the Parameters collection in each ParameterSet you run into the same problem again since each Parameter object also has a Name property.  Rather than injecting another Foreach command into the pipeline, this can be simplified using the PipelineVariable parameter in PowerShell 4.0 like so:

Get-Command -Type Cmdlet -pv cmd | 
    Select -Exp ParameterSets -pv pset | 
    Select -Exp Parameters | 
    Format-Table @{n='Cmdlet';e={$cmd.Name}}, @{n='ParameterSet';e={$pset.Name}}, Name

 

BTW you may be tempted to use a feature of Select-Object where you can add synthetic properties to objects in the collection as PowerShell expands the collection like so:

Get-Command -Type Cmdlet | Select @{n='Cmdlet';e={$_.Name}} -ExpandProperty ParameterSets

 

This works great the first time you execute it.  But upon subsequent executions, you get an error indicating that the property Cmdlet already exists.  I believe this happens because in PowerShell 3.0 this approach modifies the actual objects via the Dynamic Language Runtime. This is fine except when those objects are cached i.e. not regenerated every time they are asked for.  When that happens, you get an error because you are trying to add the same property again to the same objects you previously added the property to. 

That wraps up this discussion on the new PipelineVariable parameter in PowerShell 4.0.  I expect this parameter will save you some typing in scenarios like the ones described above.  Just watch out for aggregating commands like Group-Object and Sort-Object.